compression: Implement @ciscorn's dictionary approach

[jepler]

Massive savings. Thanks so much @ciscorn for providing the initial code for choosing the dictionary at #3370 (comment)

Now, a "dictionary" is chosen based on the frequency of strings of various lengths within the input corpus, according to a somewhat magic weight function. Some huffman codes, starting at 128, are set aside to indicate indices into this dictionary. The dictionary itself is up to 256 code points. The big advance (compared to bigrams) is that the dictionary items are arbitrary length. For instance, on Trinket M0 in English, here are some of the dictionary items: [' must be ', ' argument', ' specifi', 'attribut', 'support', ...]. " must be " occurs 21 times and gets a Huffman coding of 11001101 which is very economical (fewer bits than the single code point "%", for instance).

This adds a bit of time to the build, in order to compute the dictionary.

I also deleted a line to produce the "estimated total memory size" -- at least on the unix build with TRANSLATION=ja, this led to a build time KeyError trying to compute the codebook size for all the strings. I think this occurs because some single unicode code point ('ァ') is no longer present as itself in the compressed strings, due to always being replaced by a word. Because this line was not accounting for ngrams before or words now, it was inaccurate anyway.

As promised, this seems to save hundreds of bytes in the German translation on the trinket m0.

Testing performed:

• built trinket_m0 (en, de_DE)
• built and ran unix port in several languages (en, de_DE, ja) and ran simple error-producing codes like ./micropython -c '1/0', verifying there were no display errors

[ciscorn]

@jepler Thank you for implementing my idea!

I've created a pull request on your local repo (jepler#3) to do further optimizations.

[tannewt]

Could you explain briefly how this works? It's not clear to me from the code what units it now compresses.

[tannewt]

Also, how does it compare to MicroPython's approach? micropython#5861

[jepler]

I improved the initial comment on this PR and also added an explanation in the comments of translate.h:

+// - code points starting at 128 (word_start) and potentially extending
+//   to 255 (word_end) (but never interfering with the target
+//   language's used code points) stand for dictionary entries in a
+//   dictionary with size up to 256 code points.  The dictionary entries
+//   are computed with a heuristic based on frequent substrings of 2 to
+//   9 code points.  These are called "words" but are not, grammatically
+//   speaking, words.  They're just spans of code points that frequently
+//   occur together.
+//
+// - dictionary entries are non-overlapping, and the _ending_ index of each
+//   entry is stored in an array.  Since the index given is the ending
+//   index, the array is called "wends".

Micropython's approach is very English-centric. It literally looks for the most frequent "words", where "words" are surrounded by whitespace on either side. It has hard-coded assumptions that all code points are below 128, so it uses huffman codes below 128 for the characters themselves and above 128 for the words, and does Huffman compression on those values. The table itself is not limited to 256 bytes and does not have an explicit table of lengths or ends, but only because it can re-use the high bit of each byte in the table to indicate "is last letter in word".

To contrast, @ciscorn's scheme (like our existing scheme) works for arbitrary Unicode encodings, and is not limited by a definition of a "word". This is especially important for the Japanese translation, as whitespace doesn't play the same role in setting word boundaries as it does in English and other languages you might be familiar with:

msgid "%q must be >= 0"
msgstr "%qは0以上でなければなりません"

If we went back to the "word length" method of computing the offsets into the dictionary table, we could move beyond 256 bytes of table. Otherwise, I think this accomplishes everything Micropython does and more.

[tannewt]

Ok, thank you for the comments! I think it's a good balance between a fully greedy solution and one that won't take forever. :-)