of course the toml crate has serde on it
Replies:
15
the way packages can and do work around this is to simply write to a known filesystem path as a primitive form of IPC. this is highly problematic
pip's case is interesting too:
(1) it has a specific fetch-parse-cache process it needs to perform before it can do any resolve work
(2) it needs to be portable, so it can't add its own native code
(3) it has builds and fetches that can occur in parallel
(4) it wants to avoid being a build system like cargo, but it ends up propagating build provenance through its outputs
the pants project used to have a set of binary bootstrap scripts for things like gcc (that's the one i maintained for twitter's c++ code)
the reason this arose was:
(1) i need to build zstd if not available
(2) i need to build curl if not available
(3) i need to build make if not available
there are distinct contexts separating all of these! a bootstrap make doesn't need to have guile extensions. a bootstrap curl doesn't need to have zstd built in. a bootstrap zstd doesn't need multithreading
a task system for recursive bootstrapping (generating shared resources like interpreters and libraries) that also fetches and caches...........
pants had this separation between file/directory content (stored persistently in a merkle tree in a k/v db by checksum key) and in-memory build products. one problem with it was that it stored absolutely every single intermediate file and directory used over the course of a bootstrapping process. really wasteful and necessitated the obnoxious LMDB store. this was the passion project of the bazel engineer who convinced twitter to switch to bazel
but a persistent file store that stores important nodes is a really sick idea. i just earlier built a terrible "resource" fetching abstraction for cargo build scripts and realized wait this is kind of like the pants file store? and then recalled that spack has one of these too, but it stores specific intermediate directory outputs
i think for complex tasks that python is the right way to achieve extensibility. no glorified yaml like starlark with its own mysterious semantics. it's gonna be a real programming language
but the thing autoconf does achieve is not requiring you to have a python interpreter beforehand, which is why it can be used to build the python interpreter
one other curious thought from the pip case: there may be information that should be globally available, but is highly inefficient in distinct file outputs--it needs to be synchronized into some shared state (like a sqlite db). this is how the available versions for a python package can be made queryable
that's notable because it represents a different kind of state and a different type of dependency than other tasks
in the case of querying python indices, it represents the world state at that time
(importantly, the "world state" paradigm can also be applied to other forms of global mutable state like the filesystem. this is one problem with spack externals -- they don't have any concept of invalidation)
so where does it end?
if we want to use this bootstrap zstd, we have:
(1) use pkg-config to find one matching the desired version spec
(2) bootstrap a known version from source
spack is a massive piece of machinery to resolve a dependency graph with versions of c and c++ and python etc. it's not worth anyone's time to recreate from scratch. but spack itself has bootstrap needs (it needs python)
