HN thread news.ycombinator.com/item?id=279464
Conversation
Seems like preliminary panic. AssemblyScript can move to UTF-8 like everyone else
1
4
UTF8 doesn't solve security / validation problems:
onlineutf8tools.com/validate-utf8
websec.github.io/unicode-securi
1
It's the standard string encoding in 2021 & good enough. Other encodings should be used for legacy purposes only
1
2
Dart, C#, Java, JavaScript internally uses UTF16. And They won't be able to change it to UTF8 even in the future. Otherwise it would require a complete re-do of the String API and loose compatibility. Also UTF8 does not solve the security problems like I said
2
1
This isn't accurate. JavaScript doesn't have UTF-16 strings. If it did, every JavaScript string could be represented with standard UTF-8.
JavaScript implements strings as arrays of UCS2 characters which is not standard Unicode or UTF-16. That's why an extended UTF-8 is required.
1
3
JavaScript has mixed UCS2 and UTF16 api. Some methods like codePointAt, fromCodePoint, toUpperCase/toLowerCase, for..of str, Array.from(str), normalize, regex's test are unicode aware. The rest are USC2 include substr and length. But it doesn't really make much difference...
2
The strings are inherently not UTF-16 because they can contain invalid Unicode. UTF-16 doesn't support any string that isn't also supported by UTF-8. There is no lossy conversion between UTF-8 and UTF-16. They're different encodings of the same thing able to represent it all.
1
C++ strings also can contain and produce invalid UTF8. Does it mean C++ not support UTF8? =)
2
std::string is just a dynamic array of bytes able to manage a terminating NUL for C string compatibility. It doesn't have any specific encoding. Being able to stick UTF-8 in std::string doesn't mean it provides any form of UTF-8 support or that it's a UTF-8 string type.
You still haven't answered, is it possible to conclude that C++ does not support UTF8? Yes or no?
1
C strings and std::string are not Unicode string types and while they can hold UTF-8 they are not UTF-8 string types and do not provide any actual UTF-8 support. They're arrays of bytes. C++ std::string is implemented as a library and other libraries can implement string types.
1
Show replies