The UNIX system has been in wide use for over 20 years, and has helped to define many areas of computing.
Post
Replies:
15
Boosts:
16
For this reason, there needs to be caution and care put into any decision that affects an entire ecosystem.
usually we do PEPs for that. but pypi doesn't do PEPs
oh my god forgot they had another blog post https://blog.pypi.org/posts/2025-08-07-wheel-archive-confusion-attacks/
especially notice how our boy seth calls out the wheel format PEP for being vague:
The most detail provided is:
and then does not propose a PEP change. he does have another accusation though:
However, most Python installers today do not do this check and extract the contents of the ZIP archive similar to unzip and then amend the installed RECORD within the virtual environment so that uninstalling the package works as expected.
our boy seth is claiming that "most python installers" will:
- recompute checksums for the files it just extracted
- open the
RECORDfile and write the new value
and then implies this is some sort of cover-up for the uninstall operation
the author of the wheel PEP seth calls out was one of the few people who reached out to help me with my http zip PR that pip still refuses to merge https://github.com/pypa/pip/pull/12208
it's so fucking batshit to make a whole blog post about "popular installers" failing to check the RECORD file without mentioning or contacting any of them to fix it
ok i'm more insane now but i'm also much more certain that seth larson writing active falsehoods about zips and checksums is because:
- plaintext metadata and redundant offsets make it very difficult to retroactively modify data without breaking a human-checkable invariant
- storing both compressed and uncompressed size means the decompression process cannot be used to inject extra data through length extension
- zip files having a magic number and unambiguous offsets at the end foils possibly foils length extension
- when negative range requests suddenly broke on pypi, that was likely about length extension
METADATA on the other hand is not plaintext, because its semantics depend upon the version of cpython/pip, and are not machine-checkable or human-visible
google's dirhash is so fucking funny though
problem is to make a good proof of concept for length extension you need to have a ton of parallel compute, close to a whole data center, to figure out how to go from:
(A) length N, checksum C [good]
(B) length N + M, checksum C [bad]
since SHA-256 is cryptographic, you need to essentially mine a bitcoin
it's a little harder than a bitcoin aiui, since you need to ensure the result N + M seems valid at first, but actually injects some evil data
https://en.wikipedia.org/wiki/Length_extension_attack
this page claims SHA-256/512 and SHA3 are not susceptible, which i'm pretty sure is false? it does admit:
A secret suffix MAC, which is calculated as Hash(message ‖ secret), isn't vulnerable to a length extension attack, but is vulnerable to another attack based on a hash collision.[7]
ah, so i was mistaken! length extension seems to require that the valid prefix is not known to the attacker
anyway, you know who operates pypi and has a lot of spare GPUs? obviously that's the very definition of google scale
that's why the RECORD file is weirdly redundant with the required zip metadata, EXCEPT: standard zips only provide a crc32 checksum for each entry, which is not cryptographic and much easier to collide with
(i don't blame phil katz, he was only seeking to capture actual network errors, and computers were much slower back then.
i do blame zstd for their several layers of deeply odd functionality, including the optional non-cryptographic xxhash and multiple variants of "frames" and "blocks" which would decompress to 0-length outputs and are completely and obviously redundant in the protocol.)
the reason i'm mentioning this is of course: how can we make the fractal zip even more powerful than its progenitor?
i continue to find it ridiculous that no one has identified the properties of tree hashing (especially blake3) to be ideal for a filesystem. but then again, most filesystems generally have no idea what's going on most of the time
https://github.com/BLAKE3-team/BLAKE3/tree/master/c
Hashing large files with this function usually requires memory-mapping, since reading a file into memory in a single-threaded loop takes longer than hashing the resulting buffer. Note that hashing a memory-mapped file with this function produces a "random" pattern of disk reads, which can be slow on spinning disks. Again it's important to benchmark your specific use case.
this is mostly unhelpful, but we will forgive them
