my emacs is completely fucked
Post
Replies:
15
@hipsterelectron so wait what is different in practice when you do that
@kirakira great question! there's a short answer and a long answer. the long answer is likely still beyond my ability to do it justice but it's kinda why i've been so focused on filesystems for so long.
so essentially the filesystem is where fucking everything happens all at once. the NASDAQ trading floor. people biking to work. kids walking to school in their outfits. it's new york, baby! it all happens here!
@kirakira it's been really fascinating to learn with build tools how that differs from the challenges of network and web browser engineering like servo. the biggest distinction there is that most async models are wildly inappropriate for local file i/o. there are larger discussions to have about why, but the close technical distinction here is that networking and servers are largely optimizing:
(1) interactions with many other entities they don't control
(2) with incredibly nondeterministic latency for every request
(1) interactions with many other entities they don't control
(2) with incredibly nondeterministic latency for every request
@kirakira so that's the reason why async works really well because it lets you timeshare your cycles when these unpredictable latencies finally occur. but the filesystem is different. you have some degree of control over it (but not much). but importantly, instead of web/http standards for things like caching artifacts (as i'm taking advantage of with pip), you generally have to handle a massive variety of complexity that has grown directly from all the different ideas people had about UNIX from bell labs. it's the C ABI baby!!!
but most importantly, you're often writing code that runs on your users' laptops and touches their files. that's a different degree of closeness and responsibility that you have to bear on top of figuring out all these situations that don't have protocols to provide easy answers.
@kirakira so the reason i started investigating this in the first place was because i thought it would be relatively plug and play. we used the rust
ignore crate in pants for almost two years before we amicably parted ways because pants needed more control over that traversal. this was supposed to be something i could contribute to @ethersync which like pants has to concern itself not just with a single filesystem crawl but also correctly maintaining state over time. this mutable state is where things get really dicey.
@kirakira so then i almost immediately found a deadlock in ripgrep https://github.com/BurntSushi/ripgrep/pull/3010 because it turns out recursive directory traversal is almost uniquely difficult to map to standard parallelism models.
the literally fractal complexity that results from every directory potentially having an entirely new set of entries is genuinely just very difficult to solve.
contrast to a problem space that can be solved: https://github.com/zip-rs/zip2/pull/236. zip files tell you up front what contents they have and you can plan and schedule your work around this. this ability to schedule "offline" is what the filesystem gives you, and it's something i haven't been able to translate to the async executor model vs one with explicit threading.
@kirakira in particular, the reason ripgrep had this occur (andrew gallant is great and i admire him a lot to be clear) is because when you traverse a directory recursively, you legitimately might want to return early, either from a subbranch (e.g. if it doesn't match your regex patterns, like the
ignore crate) or from the entire process. differentiating that from erroneous behavior is extremely difficult and i spent probably several days thinking about that PR before trying to solve it on my own.
@kirakira the reason i focused on (1) file descriptors and (2) inodes is because they (at least fds) represent some of the very few rigid contracts you can actually request from the filesystem (something @zwol has been pondering recently). i believe for example when working on the parallel zip implementation (which required expanding the rust libc wrapper) that POSIX does not guarantee the behavior of parallel write operations on the same file handle, but it does guarantee that parallel reads with
pread() will succeed.so while path strings are a really tempting way to reduce the problem to an abstraction, they can be and often are a complete lie. just yesterday i started getting errors in my shell because i'd unlinked the directory i was in, and even though i added the directory back later (so my shell was still displaying a valid path), the filesystem does not take IOUs like that >=[
@kirakira i think inodes are potentially a little overrated but i'm not sure about that yet and would need to talk to an expert more thoroughly. but one basic thing the
ignore crate (and ripgrep more generally) does that is clearly pessimal is not differentiating symlinks from other types of paths, so they traverse the same output twice. i suspect ripgrep keeps track of more persistent identifiers like inodes when it actually reads the files, but i think it's the wrong abstraction that leads to more than just deadlocking if you're relying on it for correctness like pants
@kirakira the specific thing i investigated to solve the deadlock from this recursive problem was (1) deduplicating posix paths (hence the path library which is production quality) (2) codifying directory streams as a resource you're blocking on--instead of kinda glossing that over with the work-stealing approach of ripgrep/
ignore.so this should still be plausible on windows too, but POSIX literally just standardized last year the posix_getdents() libc function call https://pubs.opengroup.org/onlinepubs/9799919799/functions/posix_getdents.html. and this is really fucking sick for a variety of reasons but mostly it codifies what everyone already agreed on and that is POSIX's job
@hipsterelectron @kirakira about damn time that got standardized! might actually dust off my glibc contributor hat to get that implemented, if no one's beaten me to it
@zwol @kirakira i would personally appreciate that since it would make it much easier to push for acceptance in the rust libc crate lmao https://github.com/rust-lang/libc/pull/4522 i don't know what i expected but it wasn't "this posix standard sounds fake" lmao. i also investigated and found impls for free/open/dragonfly/net/bsd, but strangely not for macos (i mentioned this to them during an interview)
cc @leo most powerful non sequiter on a PR i have literally ever received
@hipsterelectron ?? it's not like they cared about the multiple bizarre Nintendo homebrew abis that two projects use
@leo i think it would be funny to wait a few months for glibc to implement it so i can come back and say yeah rust is officially slower than glibc
