Conversation

"you don't need to know about the implementation" twitter.com/richardstartin

This Tweet is unavailable.

2

1

28

Replying to

Seems this was a regression caused by the complexity of Compact Strings which has been fixed for a while now. bugs.openjdk.java.net/browse/JDK-817 Maybe one day Java will get proper strings instead of pretending that their UCS2 strings are UTF-16 and that UTF-16 is actually a good idea.

1

5

Replying to

and

Java just ossified around all the worst ideas of the 90s so I wouldn't count on it.

1

1

Replying to

and

I'm optimistic JavaScript will get proper strings because WebAssembly is UTF-8 focused and they're making good decisions about how interoperability works with JavaScript despite it upsetting people invested in UCS2. Python switched from UCS2 to UCS4 which was a dumb decision...

1

2

Replying to

and

Python took that same multiple string representation approach and browsers did the same for JavaScript. The strings are always fixed size units in Java / JavaScript / Python and they bloat themselves up to a larger unit if you add a single character from the range requiring it.

2

Marc-André Moreau

Replying to

and

I think the most reasonable approach today would be for most languages to use UTF-8 internally, yet natively support UTF-16 and UTF-32 as well, with easy conversion. Trying to settle on *one* isn't going to work because that boat has already sailed ⛵

2

Replying to

and

Using UTF-8 everywhere (strings, input/output) and having iterators/cursors for code points and grapheme clusters works fine. It's perfectly efficient to get back to the same place in the string as before. There's no real world use case for jumping to the nth code point quickly.

2

2

Marc-André Moreau

Replying to

and

The funny thing is I think I've never actually used code that truly needs to know that character index in a string truly is the beginning of a real character. strcmp doesn't care. strlen doesn't care. strdup doesn't care. What cares, aside maybe from font rendering code?

1

Marc-André Moreau

Replying to

A lot of the common functions treat the string as nothing more than an array of elements (uint8_t, uint16_t, uint32_t) and doesn't really need to know what they correspond to individually. Maybe one exception would be uppercasing/lowercasing because it needs the real "character"

1

Replying to

and

Unicode code points are only low-level semantic characters. They form higher-level grapheme clusters composed out of an arbitrary number of code units. In a terminal, the cells are generally grapheme clusters, but free flowing text has higher level groups due to ligatures, etc.

1

Replying to

There are useful operations which can be performed on code points but indexing code points isn't one of them. Monospace text editor or terminal is presumably storing a string of code points (grapheme clusters) for each cell. You may input code points but you edit them in groups.

1:54 PM · Jan 13, 2022Twitter Web App

Replying to

It's not like that outside of those cell-based cases because the fonts can define ligatures, etc. and combine together multiple grapheme clusters when they have a fancier way of rendering them together, etc. Java style UCS2 strings would still be misguided if they used UCS4.

1

Replying to

Trying to define strings based on grapheme clusters would also be horribly wrong because they change the rules over time such as adding combining forms of emojis and it's not actually useful. It's not actually what a terminal or text editor wants. It's more flexible than that.