Nostr Web Client

i have not looked deeply into it but the way Go does it is convert the variable length symbols into an array of 32 bit values (there is ~4bln points in the field) and you have to compute the byte length in UTF-8 encoding by the values of each element of the array...

i don't know about avoiding breaking up words, word boundaries are another thing again, as you know in some languages there isn't a notion of a "space" character but rather each symbol is an atomic unit, and as i'm fairly sure that they are maximum 4 bytes long encoded in most cases it's either you are splitting on a space or a symbol boundary, spaces are not considered as symbols except as far as the bytes they take up, which is 1, i think, just a hex 0x20

uniseg looks pretty much like a good choice. i should just point out that one of the Go Authors was one of the co-designers of the Unicode spec, Rob Pike, so Go's handling is especially good. It might help you find a better alternative by searching for his recommendation for C/C++ projects needing to do this

most of it his handled already in javascript also, because internationalisation was very important to web systems from very early on, but server and desktop app languages have lived in a bubble of mostly english for a long time.

as regards to the truncation length decision, i think that visual distance should be what you are aiming for here, and for the most part, most distinctive language scripts have a fairly uniform relative width for the same symbols, so i think the right place to evaluate that question has to do with the typography side of the encodings, the byte length is probably not relevant at all

21823843... 1y ago 💬 1

> It might help you find a better alternative by searching for his recommendation for C/C++ projects needing to do this

Thanks, I will look into this. Go has much better Unicode support than C++, which basically doesn't have it at all, or rather, you need to pull in a library to do even basic things. This is why I'm doing the hack I mentioned above: I don't want to add a dependency on a library like ICU (also it is very efficient).

OTOH, Perl has outstanding Unicode support. If you don't care about byte length, then you can simply pick out the first 100 grapheme clusters with a regexp like this: /^\X{0,100}/. This will handle the segmentation according the the TR-29 rules.

Although you're probably right for this use-case, in my experience byte-length limits are quite common and you have to deal with them somehow, ideally without causing weird artifacts. For example in nostr you have byte-size limits on note length, tag values, etc. Another tricky aspect is that theoretically grapheme clusters are unbounded in length. So a single "character" could take up gigabytes of encoded space -- worth keeping in mind due to the DoS risk.

Reply to this note

Please Login to reply.

Discussion

ᴛʜᴇ ᴅᴇᴀᴛʜ ᴏꜰ ᴍʟᴇᴋᴜ 1y ago

yeah, i think if you call a "character" meaning a UTF-8 rune, ie, unpacks to what is effectively a 32 bit value, and call each 32 bit value, which Go calls "runes" then you have a pretty solid scheme, though it's not strictly fixed in the sense of bytes it's fixed in the sense of runes