spent an excessive number of hours learning about just enough information theory to make sense of why jarek duda's tANS (table-based asymmetric numeral system) actually works because unlike yann collet at fb he seems to be trying very hard to write papers that explain things. there's a clear improvement over time commensurate with a language deficit and it's why i decided to dig in more because to me this is very obviously trying to do things right:
The modern data compression is mainly based on two approaches to entropy coding: Huffman (HC) and arithmetic/range coding (AC). The former is much faster, but approximates probabilities with powers of 2, usually leading to relatively low compression rates. The latter uses nearly exact probabilities - easily approaching theoretical compression rate limit (Shannon entropy), but at cost of much larger computational cost.
higher compression ratio is of course what everyone wants from compression and the way facebook zstd uses duda's work is literally just to compress the prefix trees computed from the standard "huffman" encoding. that does not conform to any sort of theory--but duda's work does.
Sci-Hub: are you are robot?