> Whether through string literals, conversion from other types (e.g., numeric to string), restrictions on identifiers, specification for file formats, company-policy on language for source files, conversion from strings with an ASCII charset, etc., you very often can be certain about the contents of that string.
With the exception of conversion from numbers (which has its own optimizations that are likely equally applicable in UTF8 since Arabic numbers are just ASCII anyway), I’d say all of your examples sound like bugs waiting to happen.
Why shouldn’t string literals be allowed to contain complex emoji? Why should identifiers disallow them? Why should there be a company policy around putting complex emoji places?
Just saying “let’s just declare things such that strings aren’t allowed to have multi-code point grapheme clusters” sounds great until you accidentally let that assumption leak into a place where a user wants to use an emoji and can’t make it match their skin tone.
I’d also say that such restrictions are putting the cart before the horse; the typical reasons for restricting the allowed character set, are precisely because you want to make lazy assumptions about things like string offsets. Saying that such assumptions are a good thing because you have these restrictions in place, seems like circular logic to me.
> Why shouldn’t string literals be allowed to contain complex emoji?
I didn't say they shouldn't, just that many do not and you know that at parse time.
> Why should identifiers disallow them?
I don't write the language specs. Many languages don't allow classes, methods, variables, etc. to have complex grapheme clusters in them.
> Why should there be a company policy around putting complex emoji places?
Performance. Code sanity. Indexing. Ease of typing. Again, I'm not the one writing the policies. But, they exist.
> Just saying “let’s just declare things such that strings aren’t allowed to have multi-code point grapheme clusters” sounds great until you accidentally let that assumption leak into a place where a user wants to use an emoji and can’t make it match their skin tone.
I'm making a clear distinction between situations where you have user-supplied data and data under control of the language runtime, developer-created files, or those just adhering well-defined file formats. These are all strings and commonly consist of simple codepoints; indeed, many times they're just ASCII characters.
I addressed user-supplied values when I wrote "And you can manage boundaries where strings with known properties are joined with strings with unknown properties (e.g., variable interpolation in a template file)." TruffleRuby, for example, uses ropes as its underlying structure, so if you have a template written using all ASCII characters (rather common) and interpolate a user-supplied value, you can put the user string in one rope, the template in others and link them all together into a tree with ConcatRopes. The template ropes still know they only have simple codepoints and operations on those parts can be fast. The user variable only knows it's a generic UTF-8 string and operations on that string, if any, can go down the slower path. Oftentimes, there are no operations to perform on that user string other than to display it. Its mere presence doesn't need to adversely affect the rest of the template.
> I’d also say that such restrictions are putting the cart before the horse; the typical reasons for restricting the allowed character set, are precisely because you want to make lazy assumptions about things like string offsets. Saying that such assumptions are a good thing because you have these restrictions in place, seems like circular logic to me.
I'm not making any assumptions. I've spent an awful lot of time optimizing string performance in the context of a Ruby runtime and your initial claim of constant-time subscripting being a myth doesn't match my experience. Ruby allows as complex of a string as you want, but the reality is there are many situations where strings, by either by restrictions or de facto, will not have multi-codepoint grapheme clusters. In many situations you'll have strings with all the codepoints in the ASCII range. If the only information the runtime records when parsing a string is that "this is a UTF-8" string and then operates on all UTF-8 strings uniformly, you leave a lot of performance on the table. The best performing situation is when you don't have to deal with variable-width codepoints in a UTF-8 string. UTF-16 and UTF-32 aren't terribly common in Ruby, but they exist as valid encodings (well, UTF-16BE/UTF-16LE and UTF-32BE/UTF-32LE) and have simpler execution paths than UTF-8 for many use cases.
With the exception of conversion from numbers (which has its own optimizations that are likely equally applicable in UTF8 since Arabic numbers are just ASCII anyway), I’d say all of your examples sound like bugs waiting to happen.
Why shouldn’t string literals be allowed to contain complex emoji? Why should identifiers disallow them? Why should there be a company policy around putting complex emoji places?
Just saying “let’s just declare things such that strings aren’t allowed to have multi-code point grapheme clusters” sounds great until you accidentally let that assumption leak into a place where a user wants to use an emoji and can’t make it match their skin tone.
I’d also say that such restrictions are putting the cart before the horse; the typical reasons for restricting the allowed character set, are precisely because you want to make lazy assumptions about things like string offsets. Saying that such assumptions are a good thing because you have these restrictions in place, seems like circular logic to me.