(Also, the cache will not be there on launch. I have enough moving parts already.)
Discussion
Loading...
@algernon have you spent any time trying to tune postgres?
bc if it's largely a bunch of reads I think it should perform pretty well for that
also, in the real world will you generally have a ton of scrapers scraping the same urls in a reasonably short period of time or will they generally be scraping entirely different urls to the point where caching isn't worth it?
@solonovamax I did not tune postgres at all. I'm also running it in Docker, which will slow it down, I imagine. All the postgres-performance tuning I did is to make sure that all relevant queries that need to be fast can use the index, and the index isn't unreasonably huge either.
As for scrapers: scrapers will never hit the cache. The only requests that will are the ones iocaine will let through.
Caching the garbage responses is pointless, due to volume alone. The garbage generator is also one of the fastest parts of the system, it wouldn't be worth caching it even if the volume wasn't unbounded.
I mean, iocaine will eat considerably more than it does on Eru, because there will be more instances. But counting with ~100MiB / instance, I can still fit a dozen within a GiB or so, which will leave like... 3GiB free for other things. Some of that will be the OS, but I suppose about 1.5GiB might remain free for cach-y usage?
And that'll grow as I upscale the server if needed.
Also: in this context, it would reduce memory use a lot if the training data used by iocaine could be shared between the instances.
Something to explore later on.
(Also, the cache will not be there on launch. I have enough moving parts already.)
The current plan is to finish the static server: I need to teach it to handle conditional requests (if-modified-since, if-match and the like) and compression, and then it's pretty much good to go, I think.
It can already handle range requests, and can serve partial content, and I'm pretty darn proud of that part of the code1. I've yet to test it with anything that can support resuming, mind you.... but I'll get there.
Once the static server is done, there's a whole bunch of proxy work left: like, actually spinning up configured iocaine instances, and updating the configuration live. That'll be fun.
Then, some HTML UI to write, a whole lot of docs, and then I "only" need to figure out how to make it easier for people to upload a static site. S3 is... more difficult than I anticipated. So I might try webdav or sftp, or "drop a zip, and I'll take care of it, yo!" (since I'm going to use an invite system, I trust my beta users enough to not try and abuse the system).
Because I didn't cheap out by
select * from blobs-ing the whole file, but only retrieve the parts actually requested. It also supports multiple ranges, not just one, so parallel downloads will work too. ↩︎
That all sounds like a lot, and February 17th is almost here. But... it's not that much. I think the largest hurdles are behind me, I know how the system will work, and I know what's missing. A lot of the questions I had a month ago, have been resolved.
Mind you, there's still some risk. And I might have to compromise and call the February 17th launch an early alpha.
In fact, I should probably do that anyway. A lot of things will be missing that I'll be adding later. But the deadline is Important. To me.
I'd rather launch with something bare bones that at most a handful of people can even try to use, but launch on time, rather than postpone it 'till it's friendlier. I kinda would like to have some people poke at it sooner rather than later, so I'll know which parts to focus on.
1+ more replies (not shown)