I'm sure this won't cause anyone any headaches:
"Since DEFLATE is the underlying compression used in archive/zip, compress/gzip, compress/zlib, and image/png, the outputs from those packages may also have changed."
18
Replies
4
Boosts
@andrewnez
I hope no software forge uses go to generate git-archive tarballs on the fly, that are later used by distros to build packages,and checked against a known-good hash...
@wolf480pl @andrewnez archive tarballs should be among the releases' files
@SRAZKVT
some upstreams (eg. bfdd) don't have releases, so some distro's (eg. OpenWrt) packages use the "generate tarball from commit" endpoint as a source download url
@andrewnez
@wolf480pl @SRAZKVT @andrewnez
Huh, they could use git clone --depth 1 and diff the contents instead of relying on tarball integrity.
This keeps coming up again and again.
Commit hashes ought to remain stable.
@CyReVolt @wolf480pl @andrewnez a git clone and a digest of the tree should suffice, but i don't know of a message digest for filesystem trees
maybe @hipsterelectron ?
@SRAZKVT @CyReVolt @wolf480pl @andrewnez this is a fascinating topic and i believe deserves a new checksum algorithm to minimize recomputation upon change. in particular all the SHAs are actively designed against this since they are used in cryptography to hide information and cannot be computed in parallel (hence requiring hacks like sorting directory entries). hash bucketing entries by name is a good start
@hipsterelectron
like, I think git already does this, in a way where recomputation after a single-file update is O(updated file size + diectory depth)
If that method doesn't solve your problem then I don't understand what problem you're trying to solve.
@SRAZKVT @CyReVolt @andrewnez
@wolf480pl that's cool i wasn't asking for help though
@hipsterelectron
But but like, let's bring it back on topic of the discussion we were having.
If directory hashing as done by git is not sufficient for what you and @ SRAZKVT expect for verifying an upstream source archive, I'm curious what other properties you'd need that git tree hashes don't have, and which I'm not seeing.
@wolf480pl @hipsterelectron my issue with git is it uses sha1 (yes, i know git technically supports sha256 but basically no one is using that)
@SRAZKVT
Ok but you can just reimplement the same scheme with sha256, Keccak or Blake2 instead and it will retain all of its nice properties.
Yeah, it's not something available out of the box, and not something that will match existing hash values people are already using, but it doesn't require new cryptography.
@hipsterelectron
@wolf480pl @SRAZKVT all of the things you mentioned here are examples of new cryptography. they are specifically intended for use as cryptographic message digests which means indifferentiability. this means they can be used for signatures without leaking information about the input. in the case of a checksum mismatch for source code, we don't want that at all: we want the forensic capacity to determine what changed. this is also something we want for filesystem integrity
@hipsterelectron
they're existing cryptography that has widespread Implementations and could be used as a drop-in replecement in higher-level schemes, such as git's directory hashing scheme.
Forensic capacity to determine what's changed sounds interesting, but is it possible information-theoretically, without having the plaintext of both archives?
@wolf480pl @SRAZKVT BLAKE3 does not have widespread implementations and its C implementation is very poorly done and they unbelievably have decided to use an intel-only threading library which is absolutely the wrong parallelism model. i've proposed implementing it to AMD https://github.com/amd/aocl-crypto/issues/7
the drop-in replacement scenario is part of the problem. the interface supported is purely digesting a byte stream of indefinite length, which is (along with indifferentiability) a huge reason why it can be used to obscure changes. a filesystem can view a file object as a series of fixed-size chunks, and a directory tree as a series of buckets (there may be a way to avoid lexicographic sorting of names within buckets with some sort of commutative modular exponentation).
the construction that we can take advantage of is in the preservation of the hash tree at all these levels. byte stream digests used for the standard "merkle tree" are incapable of retaining a tree of per-file chunks or bucketed directory entries. minimizing recomputation of the hash tree allows it to be shared across versions more stably. a "merkle tree" hash of a directory cannot be recomputed without access to the entire plaintext directory entries, whereas this allows for diffs across versions to be represented by a branch of the hash tree that can be patched against the prior version
@hipsterelectron @SRAZKVT
hmm so for directories, instead of having a list of filenames and hashes, then removing one entry, adding a new one, and hashing the whole list
you'd want it to be some kind of commutative group, so that
H_old * old_dentry^-1 * new_dentry == H_new
?
and then diffing would be just
new_dentry * old_dentry*-1 = H_new * H_old^-1
although what do you do with such product, if you don't know which file got modified?
@wolf480pl @SRAZKVT you'd definitely need to know which file got modified. the reason this came about was because of the very obnoxious "reproducible builds" project negging maintainers to make very subtle changes to the build process everyone else uses because they are for some reason incapable of writing their own build scripts and/or patching the source to make their desired changes. that's an issue in itself, but part of it is because their reliance upon purely the top-level checksum makes their own process excessively prone to backdoors because it only records the final state of the build process and not intermediate ones
@hipsterelectron @SRAZKVT
wait, so if I already need to know what file was modified, what kinda forensic info does your ideal directory hashing scheme give me over the existing ones?
