this is why Go's standard library has a UTF-8 rune converter, UTF-8 is a variable length encoding
Truncating text is complicated.
Today I spent some time fixing some bugs on oddbean.com that I've been putting off for a while. Most just involved some uninteresting grunt work, but there's one that is a huge rabbit hole and, if you've never thought about it before you may be surprised at how deep it goes.
On Oddbean, we only show the first ~100 characters of a nostr note and then cut it off ("truncate" it). This is all well and good, except some titles got an unexpected weird character at the end:
Nostr Advent Calendar 2024 の 11 日目の記事を書きました。 2024年のNostrリレ�…
Now, I'm no expert on Japanese script but I'm pretty sure that diamond question mark character is not supposed to be there. What gives?
The answer is that almost all text on the web is encoded in UTF-8, which is a multi-byte Unicode encoding. That means that these Japanese characters actually take up 3 bytes, unlike Latin letters which take up 1. Oddbean was taking the first 100 bytes and cutting it off there. Unfortunately, that left an incomplete UTF-8 encoded code point which the browser replaces with a special replacement character (U+FFFD, the diamond question mark).
OK, easy fix right? Just do substr() on the code-points (not the UTF-8 encoding). Sure, but that is quite inefficient, requiring a pass over the data. Fortunately there is a more efficient way to fix this that relies on the fact that UTF-8 is a self-synchronising code, meaning you can always find nearest code point boundaries no matter where in the string you jump to. So that is what I did:
Problem solved right? Well, that depends on your definition of "solved". Notice above I've been referring to "code points" instead of characters? In many languages such as English we can pretty much get away with considering these the same. However in other scripts this is not the case.
Sometimes what we think of as a character can actually require multiple code-points. For example, the character 'â' can be represented as 'a' followed by a special ' ̂' combining character. Most common characters such as â *also* have dedicated code-points, and which representation is used depends on the Unicode Normal Form. You may also have seen country flags represented by two composite characters, or emoji alterations such as skin tone -- it's the same principle. Cutting in between such characters will cause truncation artifacts.
So rather than "character" (which is an imprecise notion), Unicode refers to Extended Grapheme Clusters, which correspond as closely as possible with what we think of as individual atoms of text. You can read more than you ever wanted to know about this here: https://www.unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries
Note that many langauges need special consideration when cutting on graphemes (or indeed words, lines etc). Especially Korean Hangul script is interesting, having been designed rather than evolved like most writing systems -- in fact it's quite elegant!
So my hack for Oddbean doesn't do all this fancy grapheme truncation, and that's because I know if I tried I would end up in a seriously deep rabbit hole. I know because I have and I did! 10 years ago I published the following Perl module: https://metacpan.org/pod/Unicode::Truncate
I'm pretty proud of this yak shave, because of the implementation. I was able to adapt the regular expressions from Unicode TR29, compose them with a UTF-8 regular expression, and compile it all with the Ragel state machine compiler ( https://www.colm.net/open-source/ragel/ ). As a result, it can both validate UTF-8 and (correctly!) truncate in a single-pass.
If you want (a lot) more Unicode trivia, I also made a presentation on this topic: https://hoytech.github.io/truncate-presentation/
Discussion
Is there an easy way to truncate to a max byte-length using the Go standard library? This is the best answer I could find (after only 1 minute of searching though):
Note that as the answer says, this doesn't understand grapheme clusters, meaning that café can become cafe (depending on normalisation form). Also it involves another pass over the data which, if already known to be UTF-8, is redundant.
This 3rd party package looks to be the rough equivalent of my Perl module: https://github.com/rivo/uniseg
i have not looked deeply into it but the way Go does it is convert the variable length symbols into an array of 32 bit values (there is ~4bln points in the field) and you have to compute the byte length in UTF-8 encoding by the values of each element of the array...
i don't know about avoiding breaking up words, word boundaries are another thing again, as you know in some languages there isn't a notion of a "space" character but rather each symbol is an atomic unit, and as i'm fairly sure that they are maximum 4 bytes long encoded in most cases it's either you are splitting on a space or a symbol boundary, spaces are not considered as symbols except as far as the bytes they take up, which is 1, i think, just a hex 0x20
uniseg looks pretty much like a good choice. i should just point out that one of the Go Authors was one of the co-designers of the Unicode spec, Rob Pike, so Go's handling is especially good. It might help you find a better alternative by searching for his recommendation for C/C++ projects needing to do this
most of it his handled already in javascript also, because internationalisation was very important to web systems from very early on, but server and desktop app languages have lived in a bubble of mostly english for a long time.
as regards to the truncation length decision, i think that visual distance should be what you are aiming for here, and for the most part, most distinctive language scripts have a fairly uniform relative width for the same symbols, so i think the right place to evaluate that question has to do with the typography side of the encodings, the byte length is probably not relevant at all
> It might help you find a better alternative by searching for his recommendation for C/C++ projects needing to do this
Thanks, I will look into this. Go has much better Unicode support than C++, which basically doesn't have it at all, or rather, you need to pull in a library to do even basic things. This is why I'm doing the hack I mentioned above: I don't want to add a dependency on a library like ICU (also it is very efficient).
OTOH, Perl has outstanding Unicode support. If you don't care about byte length, then you can simply pick out the first 100 grapheme clusters with a regexp like this: /^\X{0,100}/. This will handle the segmentation according the the TR-29 rules.
Although you're probably right for this use-case, in my experience byte-length limits are quite common and you have to deal with them somehow, ideally without causing weird artifacts. For example in nostr you have byte-size limits on note length, tag values, etc. Another tricky aspect is that theoretically grapheme clusters are unbounded in length. So a single "character" could take up gigabytes of encoded space -- worth keeping in mind due to the DoS risk.
yeah, i think if you call a "character" meaning a UTF-8 rune, ie, unpacks to what is effectively a 32 bit value, and call each 32 bit value, which Go calls "runes" then you have a pretty solid scheme, though it's not strictly fixed in the sense of bytes it's fixed in the sense of runes