2
3
Conversation
Seems like preliminary panic. AssemblyScript can move to UTF-8 like everyone else
1
4
UTF8 doesn't solve security / validation problems:
onlineutf8tools.com/validate-utf8
websec.github.io/unicode-securi
1
It's the standard string encoding in 2021 & good enough. Other encodings should be used for legacy purposes only
1
2
Dart, C#, Java, JavaScript internally uses UTF16. And They won't be able to change it to UTF8 even in the future. Otherwise it would require a complete re-do of the String API and loose compatibility. Also UTF8 does not solve the security problems like I said
2
1
This isn't accurate. JavaScript doesn't have UTF-16 strings. If it did, every JavaScript string could be represented with standard UTF-8.
JavaScript implements strings as arrays of UCS2 characters which is not standard Unicode or UTF-16. That's why an extended UTF-8 is required.
1
3
JavaScript has mixed UCS2 and UTF16 api. Some methods like codePointAt, fromCodePoint, toUpperCase/toLowerCase, for..of str, Array.from(str), normalize, regex's test are unicode aware. The rest are USC2 include substr and length. But it doesn't really make much difference...
2
The super duper fun gotcha of .codePointAt() is the pos parameter is still UCS-2.
'💩💩'.codePointAt(0) //128169
'💩💩'.codePointAt(1) //56489
Imagine all the devs swapping .charCodeAt(i) with .codePointAt(i) in their loops and thinking their code is now astral-safe. 😱
2
2
A nicer API would use opaque cursor objects for nearly all cases not covered by iterators. Internally indexing by UTF-8 or UTF-16 code units is fine. Indexing by code point is still extremely arbitrary and doesn't make much sense outside of cursor/iterator use cases.
2
JavaScript doesn't have UTF-16 strings and isn't indexing by UTF-16 code units but rather is UCS2 with broken Unicode support bolted onto it. If it was using UTF-16 code units it would throw an exception for that kind of stuff as you're pointing out.
If you have UCS2 arrays and bolt on broken APIs for using them as Unicode strings, they're still UCS2 arrays. Even if the APIs worked properly... they're still UCS2 strings. They could fix it, but they lack the will to switch to working Unicode and stop supporting broken stuff.