Post by @elilla@transmom.love

most people who are into writing systems are not into computers and vice-versa. so Unicode drama was a very particular kind of hobby drama. most of you have no idea how funny the current situation is.

see, it gets pretty tricky to define what's a "character" across every writing system in the world. for 20 years, Unicode had very strict rules about what gets into the standard: no "rich text" or "graphical variations", only "plain text abstract characters" are defined. you might think that "A" is a big pointy wedge with an horizontal stroke but from the point of view of Unicode, "A" is a number (65) and a name ("LATIN CAPITAL LETTER A") and some properties ("Letter", "Uppercase", "Narrow", "Left-to-right"). the facts that "A" is usually this triangley thing, but in some styles A looks more like a round oval with a tail, and in blacktext A looks more like an U shape—all that mean nothing to Unicode. all of those are merely graphical variations ("glyphs") of 65, LATIN CAPITAL LETTER A.

this gets of course fuzzy at the edges. The character 骨, "bone", can be written with the top component facing left, or facing right. This is unarguably a "graphic variation"—it's still the same "abstract character"—so Unicode leaves that for the font to decide. Unfortunately it has become custom in the modern era (but not in historical texts) that mainland China draws it to the left, whereas Japan and Taiwan and the others do it to the right. This means that when people changed their systems from legacy encodings to Unicode, suddenly their characters would be drawn in the hated Chinese way / the hated Japanese way, and since Unicode is a "foreign imposition" they would blame foreigner ignorance of their cultural nuances. In fact the Unicode hànzì team was made of specialists from East Asia and the reasoning for this bug was perfectly logical for nerds: if your system is set to Japanese it should be using Japanese fonts with the glyph to the right, it's not our fault if Microsoft/Linux/etc. is getting the wrong fonts. but in your legacy system it would "just work" and look right; upgrade to Unicode, problems happen. the dorks at Unicode would have endless discussions over this and reject the idea of adding "Chinese 骨" and "Japanese 骨" as different characters because that's technically Wrong, these are the same abstract character.

multiply that by a thousand. imagine 20 years of strict gatekeeping like this, with people constantly proposing new characters from this and that source and the guardians of Unicode rejecting them because they can be understood as merely graphical variations of existing characters, or as too obscure to be worthy encoding, etc. (it took so long to get hentaigana into Unicode, like ~8 years between proposal and finally having the characters encoded! you can't represent any premodern manuscripts without it!)

then Japanese people start making cellphones with Internet, before everybody else.

and they put "picture" (e) "characters" (moji) because they're cute. each company makes their own emoji set, incompatible on purpose so you have to buy into their special, incompatible little internets.

see, the only situation where Unicode would make exceptions to their strict gatekeeping was "round-trip compatibility". a character like ª is just an especially typeset 'a' and thus should not count as an abstract character. but some previous text encodings like ISO-8859-1 already had an 'ª' separated from 'a'. so if you converted a text file from ISO-8559-1 and Unicode had no 'ª', it would become 'a', and now you can't distinguish it from regular 'a', so there's no way to convert it back to ISO-8859-1. you cannot "round-trip" between encodings. so in this case and in only ☝️ this case, the Consortium would allow breaking the usual gatekeeping rules, carefully and case-by-case.

then a guy from Google asked to add all the Japanese emoji from every vendor. after all, without that you couldn't represent Japanese meeru messages in Unicode and convert them back. the Consortium obviously refused all those silly pictures as out of scope. so Google teamed up with Apple and kinda highkey threw their weight, financially and otherwise, transparently because they wanted this feature in the nascent smartphone consumer culture, and powerful interests completely overrode the cabal of writing hobbyists who had hitherto kept Unicode so pure. after years of elaborate technical criteria refusing tons of characters that people proposed for serious use, Unicode 6 (2010) included the first set of emoji.

fast forward to now, and all of those Unix greybeards and linguistics associates just nod sadly in a corner from the purely ceremonial Consortium chairs at whatever Big Tech mandarins came up with this year, like, "yeah sure that's what we need, U+1FAE9 FACE WITH BAGS UNDER EYES 🫩, whatever. U+1FAC8 HAIRY CREATURE 🫈, why not at this point."

9 Replies

15 Boosts

David Chisnall (*Now with 50% more sarcasm!*)

@david_chisnall@infosec.exchange 3 days ago

elilla&: back to Néojaponisme: 改

@elilla@transmom.love 3 days ago

@david_chisnall I don't understand your proposal? like the characters already are names. like 🫣 is not a little picture, it is a number (129763) and a string (FACE WITH PEEKING EYE). this is defined on NamesList.txt on unidata, and readily queryable on any standard library, or you can just get the TXT directly etc. how you display that name if you don't have a font is up to you. for example, if you find character 129763 in a text and you want to know what's the name in Python:

unicodedata.name(chr(129763))
'FACE WITH PEEKING EYE'
# or grep from https://www.unicode.org/Public/UNIDATA/NamesList.txt

and they you could display a generic face cos it says "FACE" in it, or maybe a little footnote "face with peeking eye", etc. this is true of any unicode character. for some complex types like Han or emoji there's even more data provided by the standard; the CLDR for example includes for 129763 the annotations "captivated | embarrass | eye | face | hide | hiding | peek | peeking | peep | scared | shy | stare", in various languages, and a text-to-speech equivalent (which is usually the character name but not always; see https://github.com/unicode-org/cldr/blob/main/common/annotations/en.xml ). you can query that directly or using libraries such as libicu. and you could fallback for example by using any other emoji you have that matches one or more of the annotations, or use it as hints to inflect a TTS voice, or display all of them as a tooltip, etc. etc.

and yeah you could round-trip by defining a set of names embedded within the text, like if you were putting the character databases redundantly on every single text file in the world, plus their code point mapping in other stands, either also embedded in the text files or, more likely, in something like the unidata, since it's a one-to-many mapping. but then you have massive data duplication, and you have to process a lot of long strings and keep track of state, which is undesirable when parsing text (experience has shown all sorts of problems when the parser state gets mixed up due to data loss and so on; it's a bug generator). the logical optimisation that make things much simpler and more robust is to build a sort of int-based index for the strings. and if you make the ints the same across documents and platforms you can efficiently do data interchange, too. that is in fact what Unicode is; a big bag of ints, each with a name associated to it, and a bunch of various other standartised, text-based properties.

David Chisnall (*Now with 50% more sarcasm!*)

@david_chisnall@infosec.exchange 3 days ago

@elilla The difference is that you need an up-to-date copy of the Unicode specification and adding a new emoji requires coordination across the Unicode standard, vendors who ship the standard, and font designers. And, given the unbounded number of possible pictures that people might want, you don't need to allocate them from a fairly scarce resource. Unicode has space for a little over a million characters, which is quite tight for fitting all human character sets in (and it still doesn't contain all, including some quite large omissions), before you add an unbounded demand for encoding little pictures that are not characters used by any language.

MSN Messenger used to do inline images as :picture name:. This was slightly annoying if you had text that contained colons (e.g. IPv6 addresses), but that's mostly because the colons are also useful characters. If they'd encoded it as {start picture encoding codepoint}picture{end picture encoding codepoint}, then the parsing is simple: when you see {start picture encoding codepoint}, read ahead to {end picture encoding codepoint}, which is something that's trivial to vectorise. Then you have a slow-path hash lookup of the string, but string-indexed hash tables are a well-understood field and are easy to make fast.

After that, anything that has pictures for the latest version of the Unicode standard will be able to find an image, but that image doesn't have to be something that you can encode in a font, you can ship them as multiple resolution PNGs or whatever else your platform provides. And if you have an older version of the standard, you can fall back to 'picture of a cat' or whatever the name is, rather than tofu. If vendors want to extend it, they are free to do so without the Unicode consortium needing to approve.

Mastodon has a notion of a custom emoji, which are explicitly character-like pictures that can be embedded in text but which aren't part of the Unicode standard. There is an additional layer on top for looking up the picture and that allows it to be rendered consistently if desired. But this can't be represented directly in Unicode. With my proposal, it would be: you'd embed these directly as picture names with some prefix, and then have an ActivityPub protocol for looking up the ones an instance doesn't know about and rendering them as text.

Most text-rendering things already have support for this. For example, the NeXT / Apple NSTextView reserves a code point in the private use area to mean 'picture goes here' and has out-of-band metadata for storing what the picture is, so all of the rendering machinery for handling this already existed before Unicode added emoji.

elilla&: back to Néojaponisme: 改

@elilla@transmom.love 3 days ago

@david_chisnall not going to argue your every point as I think we simply see things differently—I'd much rather have an int index on each file and look it up on a regularly updated system-wide .txt than duplicate that information everywhere and risk the incompatibility + accessibility hell mastodon custom emoji have become, for example—but I would just like to point out that the Unicode codespace is *vastly* oversized to encode every writing system of the world. the total number of Chinese characters isn't defined but it certainly doesn't reach 200.000, most of which are used only to represent archaic orthographies—a typical value of a modern dictionary will be 60k; in modern texts the most verbosely literate reach maybe 10k. there's only 3 other systems of comparable complexity: Mayan, Egyptian and Sumerian, two of them already encoded in Unicode, Mayan currently under work. and this is representing these complex systems maximally, already fully expanded. Chinese for example is made of a bit over a thousand primitives recombined in various layouts, and you can already represent that in Unicode via IDS. it would be conceivable to represent them by combining in the first place.

most writing systems in the world have a few dozen characters. only few of them reach the thousands. only 4 reach O(100.000). Unicode has over 4 *million* codepoints. before the advent of emoji, most users including Chinese readers would never even *see* a character outside of the BMP (the first 65k plane), which is plenty enough to include practical Chinese/Japanese plus every major modern script of the world plus all sorts of obscure symbols that only nerds like me would bother using. which is why all other planes are nicknamed "astral planes".

you can criticise emoji being part of Unicode on various grounds, but there is absolutely no argument that codepoints are a scarce resource they're eating up. put all emoji current and future on the SMP, like atm 3,953 characters sandwitched between Ottoman Siyaq numbers, mahjong tiles and alchemical tiles doesn't even *begin* to risk exhausting *one* of the astral planes. the map below is the map of one of the planes, not of the entire Unicode:

A diagram of the Supplementary Multilingual Plane, one of the 16 planes in Unicode. It is presented in 16 rows of 16 columns. "Supplementary symbols" takes a bit over one row near the bottom. A crude label for "All the emoji" marks five blocks; that is, one third of one sixteenth of one sixteenth.

1+ more replies

elilla&: back to Néojaponisme: 改

@elilla@transmom.love 3 days ago

most writing systems in the world have a few dozen characters. only few of them reach the thousands. only 4 reach O(100.000). Unicode has over 4 *million* codepoints. before the advent of emoji, most users including Chinese readers would never even *see* a character outside of the BMP (the first 65k plane), including all sorts of obscure symbols that only nerds like me would bother using. which is why all other planes were nicknamed "astral planes".

elilla&: back to Néojaponisme: 改

@elilla@transmom.love 3 days ago

unicodedata.name(chr(129763))
'FACE WITH PEEKING EYE'
# or grep from https://www.unicode.org/Public/UNIDATA/NamesList.txt

and yeah you could round-trip by defining a set of names and their code points, but then you have to process a lot of long strings and keep track of state, which is undesirable when processing text. the logical optimisation that make things much simpler is to build a sort of int-based index. and if you make the ints the same across documents and platforms you can efficiently do data interchange, too. that is what Unicode is.

OCTADE

@octade@soc.octade.net 4 days ago

A few years ago I wrote a unicode parser for a hash verification program. That was interesting. I miss plain old ASCII.

CC: @elilla@transmom.love

elilla&: back to Néojaponisme: 改

@elilla@transmom.love 3 days ago

@octade Unicode is a wonderful standard and ASCII is extremely Euro- and Anglocentric.

you generally don't have to parse unicode yourself because the algorithms are already given (in the technical annexes / UAX) and reference implementations are already provided (e.g. libicu).

Dag Ågren ↙︎↙︎↙︎

@WAHa_06x36@mastodon.social 3 days ago

@elilla @octade Also, you can’t even write proper ENGLISH in ASCII. It is one of those things that are simple, elegant, and wrong.