Analyzed and THIS is the core part of the fix:

https://github.com/Zirias/poser/commit/b1aa9784955db4712d695d217344cb65bbff6ed6

I'll have to revisit that later, it's probably wasteful as it is right now, I should come up with some better idea to implement async logging later. But it DOES fix the bug I finally identified that could make #swad crash.

Now, just running it, observing it, stress-testing it from time to time ... wish me luck that THIS was indeed the only reason left for crashing in "normal operation".

I already know there's more work to do regarding reload (SIGHUP) with the multi-threaded approach, it's currently unsafe unfortunately, must be fixed before the next release.

Oh boy, I have a lead! And it's NOT related to #TLS. I finally noticed another pattern: #swad only #crashed when running as a #daemon. The daemonizing wasn't the problem, but the default logging configuration attached to it: "fake async", by letting a #threadpool job do the logging.

Forcing THAT even when running in foreground, I can finally reproduce a crash. And I wouldn't be surprised if that was actually the reason for crashing "pretty quickly" with #LibreSSL (and only rarely with #OpenSSL), I mean, something going rogue in your address space can have the weirdest effects.

For two days straight, I just can't reproduce #swad #crashing with *anything* in place (#clang #sanitizer instrumentation, attached #debugger like #lldb) that could give me the slightest hint what's going wrong. 😡

But it *does* crash when "unobserved". And it looks like this is happening a lot sooner (or, more often?) when using #LibreSSL ... but I also suspect this could be a red herring in the end.

Situation reminds me of my physics teacher back at school, who used to say something in german I just can't ever forget:

"Wer misst, misst Mist."

Feeble attempt in english would be "the one who measures measures crap", it was his humorous way to bring one consequence of #Heisenberg's indeterminacy principle to the point. And indeed, #debugging computer programs always suffers from similar problems...

I need help. First the question: On #FreeBSD, with all ports built with #LibreSSL, can I somehow use the #clang #thread #sanitizer on a binary actually using LibreSSL and get sane output?

What I now observe debugging #swad:

- A version built with #OpenSSL (from base) doesn't crash. At least I tried very hard, really stressing it with #jmeter, to no avail. Built with LibreSSL, it does crash.
- Less relevant: the OpenSSL version also performs slightly better, but needs almost twice the RAM
- The thread sanitizer finds nothing to complain when built with OpenSSL
- It complains a lot with LibreSSL, but the reports look "fishy", e.g. it seems to intercept some OpenSSL API functions (like SHA384_Final)
- It even complains when running with a single-thread event loop.
- I use a single SSL_CTX per listening socket, creating SSL objects from it per connection ... also with multithreading; according to a few sources, this should be supported and safe.
- I can't imagine doing that on a *single* thread could break with LibreSSL, I mean, this would make SSL_CTX pretty much pointless
- I *could* imagine sharing the SSL_CTX with multiple threads to create their SSL objects from *might* not be safe with LibreSSL, but no idea how to verify as long as the thread sanitizer gives me "delusional" output 😳

I just stress-tested the current dev state of #swad on #Linux. The first attempt failed miserably, got a lot of errors accepting a connection. Well, this lead to another little improvement, I added another static method to my logging interface that mimics #perror: Also print the description of the system errno. With that in place, I could see the issue was "too many open files". Checking #ulimit -n gave me 1024. Seriously? 🤯 On my #FreeBSD machine, as a regular user, it's 226755. Ok, bumped that up to 8192 and then the stress test ran through without issues.

On a side note, this also made creating new timers (using #timerfd on Linux) fail, which ultimately made swad crash. I have to redesign my timer interface so that creating a timer may explicitly fail and I can react on that, aborting whatever would need that timer.

Anyways, the same test gave somewhat acceptable results: throughput of roughly 3000 req/s, response times around 500ms. Not great, but okayish, and not directly comparable because this test ran in a #bhyve vm and the requests had to pass the virtual networking.

One major issue is still the #RAM consumption. The test left swad with a resident set of > 540 MiB. I have no idea what to do about that. 😞 The code makes heavy use of "allocated objects" (every connection object with metadata and buffers, every event handler registered, every timer, and so on), so, uses the #heap a lot, but according to #valgrind, correctly frees everything. Still the resident set just keeps growing. I guess it's the classic #fragmentation issue...

More interesting progress trying to make #swad suitable for very busy sites!

I realized that #TLS (both with #OpenSSL and #LibreSSL) is a major bottleneck. With TLS enabled, I couldn't cross 3000 requests per second, with somewhat acceptable response times (most below 500ms). Disabling TLS, I could really see the impact of a #lockfree queue as opposed to one protected by a #mutex. With the mutex, up to around 8000 req/s could be reached on the same hardware. And with a lockfree design, that quickly went beyond 10k req/s, but crashed. 😆

So I read some scientific papers 🙈 ... and redesigned a lot (). And now it finally seems to work. My latest test reached a throughput of almost 25k req/s, with response times below 10ms for most requests! I really didn't expect to see this happen. 🤩 Maybe it could do even more, didn't try yet.

Open issue: Can I do something about TLS? There must be some way to make it perform at least a bit better...

() edit: Here's the design I finally used, with a much simplified "dequeue" because the queues in question are guaranteed to have only a single consumer: https://dl.acm.org/doi/10.1145/248052.248106

I now added a #lockfree version of that MPMC job queue which is picked when the system headers claim that pointers are lockfree. Doesn't give any measurable performance gain 😞. Of course the #semaphore needs to stay, the pool threads need something to wait on. But I think the reason I can't get more than 3000 requests per second with my #jmeter stress test for #swad is that the machine's CPU is now completely busy 🙈.

Need to look into actually saving CPU cycles for further optimizations I guess...

Finally getting somewhere working on the next evolution step for #swad. I have a first version that (normally 🙈) doesn't crash quickly (so, no release yet, but it's available on the master branch).

The good news: It's indeed an improvement to have multiple parallel #reactor (event-loop) threads. It now handles 3000 requests per second on the same hardware, with overall good response times and without any errors. I uploaded the results of the stress test here:

https://zirias.github.io/swad/stress/

The bad news ... well, there are multiple.

1. It got even more memory hungry. The new stress test still simulates 1000 distinct clients (trying to do more fails on my machine as #jmeter can't create new threads any more...), but with delays reduced to 1/3 and doing 100 iterations each. This now leaves it with a resident set of almost 270 MiB ... tuning #jemalloc on #FreeBSD to return memory more promptly reduces this to 187 MiB (which is still a lot) and reduces performance a bit (some requests run into 429, overall response times are worse). I have no idea yet where to start trying to improve this.

2. It requires tuning to manage that load without errors, mainly using more threads for the thread pool, although these threads stay almost idle ... which probably means I have to find ways to make putting work on and off these threads more efficient. At least I have some ideas.

3. I've seen a crash which only happened once so far, no idea as of now how to reproduce. sigh. Massively parallel code in C really is a PITA.

Seems the more I improve here, the more I find that should also be improved. 🤪

#C #coding #performance

I just fixed a "horrible" bug in #swad:

https://github.com/Zirias/poser/commit/fcd8f4eb44d9676dde2546042b5fe3165aecc52c

In case you don't understand C: This potentially dereferenced "wild" and null pointers before the (copy-and-pasted 🙈) typo was fixed, which means it's "undefined behavior", so might do surprising things, but more likely crash.

It affects the #epoll (on #Linux) and #eventports (on #Solaris / #illumos) backends. A quick smoke test on these platforms was done in swad 0.11 and didn't show any unexpected behavior. Only after preparing for the next release (that hopefully has multiple parallel event loops) by moving some static service data to thread-local storage, it suddenly failed on illumos, that's how I tracked down that embarrasing crap. 😞

I hope to complete a new version soon enough, so I don't have to do a "bugfix release" for it.

Just released: #swad 0.11 -- the session-less swad is done!

Swad is the "Simple Web Authentication Daemon", it adds cookie/form #authentication to your reverse #proxy, designed to work with #nginx' "auth_request". Several modules for checking credentials are included, one of which requires solving a crypto challenge like #Anubis does, to allow "bot-safe" guest logins. Swad is written in pure #C, compiles to a small (200-300kiB) binary, has minimal dependencies (zlib, OpenSSL/LibreSSL and optionally libpam) and should work on many #POSIX-alike systems (#FreeBSD tested a lot, #Linux and #illumos also tested)

This release is the first one not to require a server-side session (which consumes a significant amount of RAM on really busy sites), instead signed Json Web Tokens are now implemented. For now, they are signed using HMAC-SHA256 with a random key generated at startup. A future direction could be support for asymmetric keys (RSA, ED25519), which could open up new possibilities like having your reverse proxy pass the signed token to a backend application, which could then verify it, but still not forge it.

Read more, grab the latest .tar.xz, build and install it ... here: 😎

https://github.com/Zirias/swad

This is going nice so far, I can now correctly sign my #JWT (using #LibreSSL of course, so OpenSSL/LibreSSL will probably be an unconditional dependency for #swad in the next release)

Doing some first experiments, here's how a #JWT for #swad might look like, containing a custom property that has the "auth info" that's currently stored in the server-side #session ... 🤔

Now add a JOSE header, base64-encode and sign that beast...

First step towards implementing #JWT in #swad done, just committed a good 1000 LOC and now my #poser lib can do #JSON 😎
https://github.com/Zirias/poser/commit/7f1772e85c869d544f8a12099ed6545e163dc163

Seems a first step is almost done, adding #JSON support to my #poser lib. This could be the foundation for #JWT support in #swad. 😎

Need to do more thorough testing I guess, but at least the two example documents from #rfc8259 work fine ... the test tool does a full #deserialization / #serialization roundtrip (with specific internal representations of the data types supported by JSON).

edit: Look at the "Longitude" value of the second object in the second example 😏 I only noticed myself right now, but of course that's the desired behavior.

Hopefully, there will be another release of #swad soon!

Looking at my test results again, performance should be okay at least for moderately busy sites ... the 1000 requests per second I observed included actual logins, and I didn't even test whether it would also handle more (it probably would), the only issue was with resolving remote names (with that, around 30% of these requests failed because the thread pool was clogged with jobs all waiting for some DNS response), and the recommendation would be: just disable that feature if your site is a busy one.

But I'm really unhappy with RAM usage going up so much. Almost 100MiB resident set after seeing 1000 unique clients all attempting a login is a lot after all.

So, I'll try to move swad to a session-less design. It can't be fully stateless, a rate limiter will be needed, but maybe I can optimize a bit on that.

But the sessions could be replaced. They're currently used for two things:

* Store actual auth information. This could be stored in signed JWTs (json web tokens) on the client instead. I'm already starting to add JSON support to my poser lib 😉

* Store the random challenge for the #anubis-like proof-of-work checker. Could do the same as anubis here: Derive the challenge from request metadata instead, including a timestamp.

Will be quite some work, but could be doable.

Just released: #swad 0.10

https://github.com/Zirias/swad/releases/tag/v0.10

Swad is the "Simple Web Authentication Daemon". If you're looking for a way to add #authentication (and/or proof-of-work access as known from #anubis) to your #nginx reverse proxy -- without adding yet another reverse proxy -- swad could be for you! It's written in pure #C, has few external dependencies (just zlib, and optionally OpenSSL/Libressl and/or libpam) and compiles to a pretty small binary. It's designed for usage with nginx' 'auth_request'.

Swad is tested on #FreeBSD, some basic functionality tests were also done on #Linux and #illumos (descendant from #solaris). It should build and work on most #POSIX-alike systems.

This release mainly brings performance improvements and a few bugfixes. It's now stress-tested with Apache jmeter, verifying it can deal with at least 1000 requests per second on my personal (somewhat limited) FreeBSD host machine.

Got somewhere:

https://github.com/Zirias/swad/commit/1bbd1e90ff0623d972e8b71c881f590112a9668b

Now, no bot ever causes #swad to create a server-side session, at least from what I can observe in my logs -- these bots don't attempt any login!

I also disabled usage of CSRF tokens for the login form, which I forgot to mention in the commit message. They strictly require a session and are pointless on a login form anyways.

I now decided I'll at least aim for some middle grounds: Rework #swad so it only needs a (server-side) #session once a user is #authenticated!

This does have some implications, e.g. passing a redirect argument to the authentication endpoint won't work any more. But experimentation shows a workaround would be to use an "internal redirect" to the login endpoint in #nginx.

We'll see where I end up. Having sessions only for authenticated users should reduce the need for server-side RAM significantly, so I hope 😉

Good morning! ☕

Now that I can't find any other bugs in #swad any more, I'm thinking again about how I could improve it.

Would anyone consider deploying it on a busy site right now? Either as a replacement for #Anubis (proof-of-work against bots), or for simple non-federated #authentication, or maybe even both?

I'm currently not sure how well it would scale. The reason is the design with server-side sessions, which is simple and very light-weight "on the wire", but needs server-side RAM for each and every client. It's hard to guess how this would turn out on very busy sites.

So, I'm thinking about moving to a stateless design. The obvious technical choice for that would be to issue a signed #JWT (Json Web Token), just like Anubis does it as well. This would have a few consequences though:

* OpenSSL/LibreSSL would be a hard build dependency. Right now, it's only needed if the proof-of-work checker and/or TLS support is enabled.
* You'd need an X509 certificate in any case to operate swad, even without TLS, just for signing the JWTs.
* My current CSRF-protection would stop working (it's based on random tokens stored in the session). Probably not THAT bad, the login itself doesn't need it at all, and once logged in, the only action swad supports is logout, which then COULD be spoofed, but that's more an annoyance than a security threat... 🤔
* I would still need some server-side RAM for each and every client to implement the rate-limits for failed logins. At least, that's not as much RAM as currently.

Any thoughts? Should I work on going (almost) "stateless"?

Adding what was missing for intermediate certificates, I had great fun with #OpenSSL#API again. I mean, it never gets old. First test gave me a nice crash of #swad. Because ....

Well, to use a certificate (type X509 *), you call SSL_CTX_use_certificate(). Docs say "On success the reference counter of the x is incremented." (where x means the certificate). Great, so, call X509_free() directly afterwards to ensure this certificate gets destroyed whenever the SSL context gets destroyed.

So, just call the same function again for the intermediate certificates? No ... but there's SSL_CTX_add_extra_chain_cert() which can be used multiple times. Nice, call it in a loop as long as I find additional certificates in the cert file, and X509_free() them all directly after adding.

And then observe the crash. Well, it's documented, the manpage for SSL_CTX_add_extra_chain_cert() tells:

"The x509 certificate provided to SSL_CTX_add_extra_chain_cert() will be freed by the library when the SSL_CTX is destroyed. An application should not free the x509 object."

So, clearly my fault not reading this before. Consistency in API design is so overrated. 🤪