@mwl after this change: https://github.com/freebsd/pkg/commit/83243c7894afd319f009cc9135cc461767deea35 and thanks to #valgrind's massif tool, in a reference jail I used with LOTS of packages (including many package with plenty of files) the maximum memory peak for a pkg upgrade went from 1.4GB down to 199MB!
@mwl after this change: https://github.com/freebsd/pkg/commit/83243c7894afd319f009cc9135cc461767deea35 and thanks to #valgrind's massif tool, in a reference jail I used with LOTS of packages (including many package with plenty of files) the maximum memory peak for a pkg upgrade went from 1.4GB down to 199MB!
I just stress-tested the current dev state of #swad on #Linux. The first attempt failed miserably, got a lot of errors accepting a connection. Well, this lead to another little improvement, I added another static method to my logging interface that mimics #perror: Also print the description of the system errno. With that in place, I could see the issue was "too many open files". Checking #ulimit -n gave me 1024. Seriously? 🤯 On my #FreeBSD machine, as a regular user, it's 226755. Ok, bumped that up to 8192 and then the stress test ran through without issues.
On a side note, this also made creating new timers (using #timerfd on Linux) fail, which ultimately made swad crash. I have to redesign my timer interface so that creating a timer may explicitly fail and I can react on that, aborting whatever would need that timer.
Anyways, the same test gave somewhat acceptable results: throughput of roughly 3000 req/s, response times around 500ms. Not great, but okayish, and not directly comparable because this test ran in a #bhyve vm and the requests had to pass the virtual networking.
One major issue is still the #RAM consumption. The test left swad with a resident set of > 540 MiB. I have no idea what to do about that. 😞 The code makes heavy use of "allocated objects" (every connection object with metadata and buffers, every event handler registered, every timer, and so on), so, uses the #heap a lot, but according to #valgrind, correctly frees everything. Still the resident set just keeps growing. I guess it's the classic #fragmentation issue...
Nice, #threadpool overhaul done. Removed two locks ( #mutex) and two condition variables, replaced by a single lock and a single #semaphore. 😎 Simplifies the overall structure a lot, and it's probably safe to assume slightly better performance in contended situations as well. And so far, #valgrind's helgrind tool doesn't find anything to complain about. 🙃
Looking at the screenshot, I should probably make #swad default to two threads per CPU and expose the setting in the configuration file. When some thread jobs are expected to block, having more threads than CPUs is probably better.
https://github.com/Zirias/poser/commit/995c27352615a65723fbd1833b2d36781cbeff4d
