my emacs is completely fucked
Post
Replies:
15
The operating system groups files into directories. To specify a file,
you must specify the directory and the file's name within that
directory.
cannot express how delighted reading this sort of documentation makes me. not just "here's an API" but "here's why there's a whole section on this topic and we have some functions for parts of it"
maybe that excerpt sounds really overly basic but this is exactly the level you have to be thinking at to have a path abstraction that works robustly across platforms
the docs just called me out for my use of the term "path" right as i said that above!!!! bc the stuff emacs works on is largely concrete files as opposed to search paths (and in fact i totally agree with their distinction here)
this distinction is actually like the most interesting part of my directory traversal work: using file descriptors and inodes to confidently read and write and have a clear concept of progress and completion, instead of path strings which are fundamentally racy and cannot be deduplicated
@hipsterelectron so wait what is different in practice when you do that
@kirakira great question! there's a short answer and a long answer. the long answer is likely still beyond my ability to do it justice but it's kinda why i've been so focused on filesystems for so long.
so essentially the filesystem is where fucking everything happens all at once. the NASDAQ trading floor. people biking to work. kids walking to school in their outfits. it's new york, baby! it all happens here!
@kirakira it's been really fascinating to learn with build tools how that differs from the challenges of network and web browser engineering like servo. the biggest distinction there is that most async models are wildly inappropriate for local file i/o. there are larger discussions to have about why, but the close technical distinction here is that networking and servers are largely optimizing:
(1) interactions with many other entities they don't control
(2) with incredibly nondeterministic latency for every request
(1) interactions with many other entities they don't control
(2) with incredibly nondeterministic latency for every request
@kirakira so that's the reason why async works really well because it lets you timeshare your cycles when these unpredictable latencies finally occur. but the filesystem is different. you have some degree of control over it (but not much). but importantly, instead of web/http standards for things like caching artifacts (as i'm taking advantage of with pip), you generally have to handle a massive variety of complexity that has grown directly from all the different ideas people had about UNIX from bell labs. it's the C ABI baby!!!
but most importantly, you're often writing code that runs on your users' laptops and touches their files. that's a different degree of closeness and responsibility that you have to bear on top of figuring out all these situations that don't have protocols to provide easy answers.
@kirakira so the reason i started investigating this in the first place was because i thought it would be relatively plug and play. we used the rust
ignore crate in pants for almost two years before we amicably parted ways because pants needed more control over that traversal. this was supposed to be something i could contribute to @ethersync which like pants has to concern itself not just with a single filesystem crawl but also correctly maintaining state over time. this mutable state is where things get really dicey.
@kirakira so then i almost immediately found a deadlock in ripgrep https://github.com/BurntSushi/ripgrep/pull/3010 because it turns out recursive directory traversal is almost uniquely difficult to map to standard parallelism models.
the literally fractal complexity that results from every directory potentially having an entirely new set of entries is genuinely just very difficult to solve.
contrast to a problem space that can be solved: https://github.com/zip-rs/zip2/pull/236. zip files tell you up front what contents they have and you can plan and schedule your work around this. this ability to schedule "offline" is what the filesystem gives you, and it's something i haven't been able to translate to the async executor model vs one with explicit threading.
@kirakira in particular, the reason ripgrep had this occur (andrew gallant is great and i admire him a lot to be clear) is because when you traverse a directory recursively, you legitimately might want to return early, either from a subbranch (e.g. if it doesn't match your regex patterns, like the
ignore crate) or from the entire process. differentiating that from erroneous behavior is extremely difficult and i spent probably several days thinking about that PR before trying to solve it on my own.
@kirakira the reason i focused on (1) file descriptors and (2) inodes is because they (at least fds) represent some of the very few rigid contracts you can actually request from the filesystem (something @zwol has been pondering recently). i believe for example when working on the parallel zip implementation (which required expanding the rust libc wrapper) that POSIX does not guarantee the behavior of parallel write operations on the same file handle, but it does guarantee that parallel reads with
pread() will succeed.so while path strings are a really tempting way to reduce the problem to an abstraction, they can be and often are a complete lie. just yesterday i started getting errors in my shell because i'd unlinked the directory i was in, and even though i added the directory back later (so my shell was still displaying a valid path), the filesystem does not take IOUs like that >=[
@hipsterelectron @kirakira it should be said that _some_ of the problems people have with filesystems as they are today _would_ be fixed by more pervasive async support for local file access -- because "local" can be a lie. sometimes open() does have to go talk to the network, or to a FUSE server process. the key missing feature is a fd (more precisely an "open file description") that isn't attached to any file: compare how socket+bind split up the labor vs how open does it all at once.
@hipsterelectron @kirakira (plus of course O_NONBLOCK on a regular file should actually make read() and write() return EWOULDBLOCK if you ask for data that isn't in the page cache, and select(), FIONREAD, etc, should tell you the truth about regular files, but that doesn't involve adding any new concepts)
@zwol @kirakira yes! one very interesting thing about directory traversal is that you undergo a similar process to database designers where you realize e.g. the page cache (or singular readdir()) is perhaps an acceptable heuristic but not really a protocol. but the page cache and block layer is certainly far more hashed out than the VFS at this point (it was incredibly fortunate that posix_getdents() dropped right as i started this)
