cool. sweet. nice. sold on this whole rust business.
2 media
![[codspeed benchmarking comment saying that one PR that rewrites some of the core logic in rust will increase performance across our benchmarks by 37%. each of the tests are a processing graph run in different ways - a synchronous runner just calls the graph... synchronously.. in order, and a ZMQ runner is a multiprocess runner that runs each node concurrently, swapping events over zeromq. i figure the details are probably not important to anyone but me, but if you are curious about the actual numbers in the table, HTML is probably easier for screenreaders and tables than i can render here, so that's over here: https://github.com/miniscope/noob/pull/243#issuecomment-4977387758 ]](https://media.neuromatch.social/media_attachments/files/116/922/550/387/514/873/original/2169463cba808fda.png)
![[benchmarks that i ran locally on just the part that was swapped out - regular graphs are scheduled 4-5x faster than before, and large graphs are more like 10x]](https://media.neuromatch.social/media_attachments/files/116/922/569/501/061/294/original/30b72150f1a6bbf6.png)
@jonny if you are trying to pass things around without having to make snapshots or take locks, may I introduce you to our lord and savior, the im crate
re: self-congratulatory, benchmarking, not important
@ricci
wat is dat lemme see
@jonny here's what you get to do: taking a new reference to an im data structure, like passing it through a channel, basically gives you a whole ass atomic snapshot, for "free" (okay not free but surprisingly cheap). You can keep working on it in one thread, the other sees a consistent view, when you're ready to update it, just pass another reference through the channel
re: self-congratulatory, benchmarking, not important
@ricci
That's awesome. We have sort of gone the other direction of "design away shared objects and declare ordering and dependencies directly" but I want to see this thing anyway
re: self-congratulatory, benchmarking, not important
so our graph processing lib has something like 30 microseconds of overhead per call, about a quarter of that can be trivially trimmed, and another quarter can be written out. and that's without using any of the yno graph parts whose whole purpose is to make it easy to run graphs of operations p close to optimally concurrent. So its looking sorta sweet I think
re: self-congratulatory, benchmarking, not important
Gonna see if I can move the event store to rust so I can do post-run copying out of the GIL so we can do full on time travel too
