Finally getting somewhere working on the next evolution step for #swad. I have a first version that (normally 馃檲) doesn't crash quickly (so, no release yet, but it's available on the master branch).
The good news: It's indeed an improvement to have multiple parallel #reactor (event-loop) threads. It now handles 3000 requests per second on the same hardware, with overall good response times and without any errors. I uploaded the results of the stress test here:
https://zirias.github.io/swad/stress/
The bad news ... well, there are multiple.
1. It got even more memory hungry. The new stress test still simulates 1000 distinct clients (trying to do more fails on my machine as #jmeter can't create new threads any more...), but with delays reduced to 1/3 and doing 100 iterations each. This now leaves it with a resident set of almost 270 MiB ... tuning #jemalloc on #FreeBSD to return memory more promptly reduces this to 187 MiB (which is still a lot) and reduces performance a bit (some requests run into 429, overall response times are worse). I have no idea yet where to start trying to improve this.
2. It requires tuning to manage that load without errors, mainly using more threads for the thread pool, although these threads stay almost idle ... which probably means I have to find ways to make putting work on and off these threads more efficient. At least I have some ideas.
3. I've seen a crash which only happened once so far, no idea as of now how to reproduce. sigh. Massively parallel code in C really is a PITA.
Seems the more I improve here, the more I find that should also be improved. 馃お