Had some great ideas while sleeping, but... now I have even more questions. Feels like I'm overcomplicating the first steps again.
0 Boosts
0 Likes
One of the questions I have is: what information should a derivation contain? What do I need to build it? And how do I serialize that in a sane way?
Previously, when I was doing KDL config + Fennel extensions, it was all nicely KDL, and I could conveniently encode the builder + its "arguments" as a KdlDocument.
That's probably not the right approach when using Fennel for the config language.
I think I'm overengineering this again, because I keep thinking about bootstrapping, and reproducibility from ~zero. Those are nice, but... probably not important right now.
All I need is to have a derivation, and some builders that can produce the desired output. For simplicity's sake, I'd even implement a fuck-provided builder that requires no external tools, that can copy some declared files to the store. But how do I encode that in the derivation?
Maybe I'd continue using KDL. I do need a serialization format for derivations, and Fennel's not it: I don't want a programming language in the derivations! It's 100% data, and KDL feels like a better fit than a Fennel table.
In which case, the internal builder could also be parameterized via KDL. But that means all other builders would receive their parameters that way too...
On second thought, that's not a bad thing.
Now, since I decided to use Fennel for the configuration too, that meant that I do not need to register handlers for the various KDL nodes, as there won't be any. I just point fuck to a configuration, tell it which derivation to work with, it picks out that function, runs it, and gets a derivation it then decides what to do with (build it, run it, display it, that kind of stuff).
No registration whatsoever necessary!
However, I need fetchers and builders too, and most of them should be on the Fennel side. How do I work with those?
In other words, the question in my head is this: can I turn fetchers and builders into derivations too?
...and that's pretty much how I ended up thinking about bootstrapping.
Lets consider an imaginary hello package, a single shell script that prints "Hello World", and the derivation is a directory with bin/hello as the main entry point.
derivation hello-0 {
source type=declarative-tree {
file path="bin/hello" mode=0o0555 "..."
}
build-system identity
system arch=x86-64 os=linux
output out {
path "..."
hash algo=blake3 "..."
}
}
This should be enough to build the hello package.
How do we get from here to something more useful with a fetcher?
derivation gnu.hello-2.12.3 {
source type=tarball {
fetch {
url "..."
}
path "..."
hash algo=blake3 "..."
}
build-system autotools
system arch=x86-64 os=linux
output out { ... }
}
Here, the tarball source type, the fetcher, and the build-system would all be derivations this depends on. And then we have a bootstrap problem.
So perhaps, to cheat, and to fix this chicken-and-egg problem, I should provide a few built-in source types, fetchers and build-systems that don't resolve to derivations, and assume that the tools are simply... present.
What if I namespaced these things?
derivation gnu.hello-2.12.3 {
source type=bootstrap/tarball { ... }
build-system bootstrap/autotools
...
}
That'd let me get off the ground.
Still, though: how do I implement source unpackers, fetchers and builders in Fennel? How do I tell which one's a built-in, and which one need another derivation evaluated?
Meh. Not enough spoons. Gonna do something else.
Distro building is hard. Building a declarative package manager even more so.
But it's also very fun and educational, and that alone is worth it, even if I end up abandoning the project1.
Not likely, unless someone else comes up with a declarative distribution that can use NetBSD, and is neither Nix nor Guix-based. I need something like that, and if noone else builds it, I will. ↩︎
...what if I cheat, and have a built-in exec builder & fetcher, that executes random stuff outside of the managed part of the system?
Something like...
derivation gnu.hello-2.12.3 {
source type=exec {
exec "/usr/bin/tar" "-xz" "--strip-components=1" "-f" "/some/path"
fetch type=exec {
exec "/usr/bin/curl" "-L" "-o" "/some/path" "https://..."
}
hash algo=blake3 "..."
}
build-system shell #"""
./configure --prefix=/some/store/path
make
make install
"""#
}
Hrm, will have to bake some built-in environment variables in in this case, so that I can download to and make install DESTDIR= to a temporary directory that gets atomically renamed to the final store path.
There's obviously no sandboxing at play here, yet.
Oooh, and the Fennel library! I REMEMBERED! I actually had it figured out in my dreams, but forgot. Oops.
fuck will have a configurable module path, which it will add to the fennel path, and calling a Fennel-side function will be as simple as picking it from the global table.
So something like fetch type=git would live in /bla/bla/fuck-modules/fetcher/git.fnl or something. It doesn't need to be part of the configuration. Even if I use the same language for both, they're still separate.
That's the key part I forgot.
With that said, although separate, the configuration and the helper evaluators will share the same module path.
So the library that implements, say, (package ...), will also be able to supply the git fetcher.
But first, lets try building from the ground up again - trying to build top-down ended with sadness.
So next mini-milestone: a Derivation struct that can serialize & load itself. Then a shell builder. Then a way to actually store the derivation & build in the store. Then shell fetcher.
Unpacking is part of the build, so... I'll just shove the tar call into the builder.
And once I have those, then I can start handling dependencies, and then I have most of the derivation handling covered, I think.
One remaining question, however, is how to record dependencies. Nix records the drv paths, which would work for me, too, as long as the paths exist. I don't have an easy way to map the drv path back to the derivation.
I'd have to, like, iterate over all derivations in a configuration to find it. Not sure if that's a good idea... it should be reasonably cheap, though, as that doesn't do anything destructive or computationally expensive - and can be done in parallel too.
I'll think about this when I get there.
Ah. Nix has an SQLite database for mapping stuff from paths to derivations and the like. I guess I should do the same.
I'm not a fan of SQLite, but Postgres would be an overkill of a dependency here, and my gut feeling is that I want a relational database.
Questions about laziness, macros, namespacing, parameters, and suchlike. Yes, very vague, but the questions themselves are more of a feeling in my head than questions I can actually describe or ask.
It's complicated.