The UNIX system has been in wide use for over 20 years, and has helped to define many areas of computing.
the day win wang and i changed the world together at twitter inc was when we realized the infrastructure we'd constructed couldn't do context-specific locality. win's rsc scala compiler was DAG-scheduled and needed to share memory in a persistent local jvm. once it generated an outline to compile against, we could farm out embarrassingly parallel AOT-compiled scalac jobs (using the scoot RPC system from drew gassaway).
pants couldn't do that—yet. but we did it together. immediate 2x improvement before any further optimization. that's why compiler and build tool devs need to work together!!!
good thing too, cause it was like 3 days before this talk https://youtube.com/watch?v=87K4_v2IvBg it's not me giving it but there's a point where he shows the zipkin traces we constructed and you can just see the parallelism explode like a dubstep drop
This algorithm is localized in the code and may change as experience with replicated files grows.
also of course this project treats inodes like a real thing that has meaning
The storage site allocates an inode number from a pool which is local to that physical container of the filegroup. That is, to facilitate inode allocation and allow operation when not all sites are accessible, the entire inode space of a filegroup is partitioned so that each physical container for the filegroup has a collection of inode numbers that it can
allocate.
ted ts'o screaming crying throwing up rn
When all the storage sites have seen the delete, the inode can be reallocated by the site which has control of that inode (i.e. the storage site of the original create).
so this is legitimately the reason you want inodes like internally right? to do localized resource indexing! and like ok, but:
- if that's the purpose of the inode, then don't fucking expose it to userspace? if i use an inode that way, it's not gonna be in my dirents!
- the user still deserves an external inode! generate it completely differently! recycle it in its own way!
Solutions to the number representation and byte ordering problems have not yet been implemented.
LMAO
holy shit they made a virtual chroot for subprocess execution. THESE MOTHERFUCKERS MADE DOCKER!
Basically the scheme consists of four parts:
I'M READY I'M FUCKING READY
so like not only does clearly hint at some of the work we would do like 40 years later with spack to codify shared library ABI, you could even argue for the direct thread from this to my proposal to systemd to force upfront declaration of their dlopen() deps!!
![Basically the scheme consists of four parts:
a. Make the globally unique name of the object
in question refer to a special kind of directory (hereafter referred to as a hidden directory] instead of the object itself.
b. Inside this directory put the different versions of the file, naming them based on the
context with which they are associated. For
example, have the command /bin/who be a
hidden directory with the file entries ~5 and
vaz that are the respective load modules.
c. Keep a per-process inherited context for
these hidden directories. If a hidden directory is found during pathname searching (see
section 4.4 for pathname searching), it is examined for a match with the process's context rather than the next component of the
pathnames passed to the system.
d. Give users and programs an escape mechanism to make hidden directories visible so
they can be examined and specific entries
manipulated.](https://asset.circumstances.run/media_attachments/files/116/592/086/818/798/340/original/0a1353421e8be56e.png)
a special kind of directory (hereafter referred to as a hidden directory)
nobody talks about this
Keep a per-process inherited context for these hidden directories.
this is specifically for ABI translation. this is pretty obviously the future
Give users and programs an escape mechanism to make hidden directories visible so they can be examined and specific entries manipulated.
this is literally something pouzin has literally said
not only does this naming scheme allow us to store and name load modules for different sites, but allows us to transparently run a requested command on the site for which a load module exists.
multiplatform binary!!!!! fat pex files!!!!!!
the pants guy and i worked on a groundbreaking fucking project together right as i left twitter very similar to this local module resolution work (it involved the network too) and we both found each other working on very similar followup projects several years later
nerd sniping john sirois is one of my proudest feats
