i spent an hour working on what i might call a "generalized zip format" and one result that somehow works is that you can read it backwards or forwards. as in, you can make it go backwards from the end of a seekable file, or you can make it go forwards from a non-seekable stream, and the data stored in every record is unchanged
Discussion
Loading...
@hipsterelectron is it like a double linked list where each node represents a file and has compressed data of the file
i bet someone else has done basically all of this before but i doubt it's remotely as cute. this shit is cute
@hipsterelectron palindromic compression? Sounds like the kind of delightfully silly thing I wish we could expect more of in computing.
@emily_s there is actually no compression yet! the schema is recursive and supports compression at each level
@emily_s this began because the distinction between data and metadata seems to be reasonably application-defined. so for example instead of requiring that every local record (e.g. stored file data) necessarily include a timestamp and permissions (which in many cases are immaterial), you can have a schema identifier and then data length characterize the entire record. the schema indicates how to interpret the variable-sized data
@emily_s in particular i got very upset about the (imo awful) cpython zip implementation earlier as it deploys some dreadful internal state that breaks a standard optimization i developed for copying entries without decompression from a separate zip. this works and is supported in the rust zip crate, but in general i was looking to avoid the use of "offsets" in favor of lengths alone. it turns out that you can extend this approach arbitrarily
@emily_s sketch of format is here: https://codeberg.org/cosmicexplorer/dice/src/branch/main/TODO.org
unfortunately it is not quite palindromic, but rather can be laid out at either forward or reverse polarity. the recursive nature is quite interesting however
2+ more replies (not shown)