LibreSSL 4.1.1 and 4.0.1 released https://www.undeadly.org/cgi?action=article;sid=20251002054519 #openbsd #libressl #tls #https #cryptography #security #newrelease #development #freesoftware #libresoftware

LibreSSL 4.1.1 and 4.0.1 released https://www.undeadly.org/cgi?action=article;sid=20251002054519 #openbsd #libressl #tls #https #cryptography #security #newrelease #development #freesoftware #libresoftware

Oh boy, I have a lead! And it's NOT related to #TLS. I finally noticed another pattern: #swad only #crashed when running as a #daemon. The daemonizing wasn't the problem, but the default logging configuration attached to it: "fake async", by letting a #threadpool job do the logging.

Forcing THAT even when running in foreground, I can finally reproduce a crash. And I wouldn't be surprised if that was actually the reason for crashing "pretty quickly" with #LibreSSL (and only rarely with #OpenSSL), I mean, something going rogue in your address space can have the weirdest effects.

For two days straight, I just can't reproduce #swad #crashing with *anything* in place (#clang #sanitizer instrumentation, attached #debugger like #lldb) that could give me the slightest hint what's going wrong. 😡

But it *does* crash when "unobserved". And it looks like this is happening a lot sooner (or, more often?) when using #LibreSSL ... but I also suspect this could be a red herring in the end.

Situation reminds me of my physics teacher back at school, who used to say something in german I just can't ever forget:

"Wer misst, misst Mist."

Feeble attempt in english would be "the one who measures measures crap", it was his humorous way to bring one consequence of #Heisenberg's indeterminacy principle to the point. And indeed, #debugging computer programs always suffers from similar problems...

I need help. First the question: On #FreeBSD, with all ports built with #LibreSSL, can I somehow use the #clang #thread #sanitizer on a binary actually using LibreSSL and get sane output?

What I now observe debugging #swad:

- A version built with #OpenSSL (from base) doesn't crash. At least I tried very hard, really stressing it with #jmeter, to no avail. Built with LibreSSL, it does crash.
- Less relevant: the OpenSSL version also performs slightly better, but needs almost twice the RAM
- The thread sanitizer finds nothing to complain when built with OpenSSL
- It complains a lot with LibreSSL, but the reports look "fishy", e.g. it seems to intercept some OpenSSL API functions (like SHA384_Final)
- It even complains when running with a single-thread event loop.
- I use a single SSL_CTX per listening socket, creating SSL objects from it per connection ... also with multithreading; according to a few sources, this should be supported and safe.
- I can't imagine doing that on a *single* thread could break with LibreSSL, I mean, this would make SSL_CTX pretty much pointless
- I *could* imagine sharing the SSL_CTX with multiple threads to create their SSL objects from *might* not be safe with LibreSSL, but no idea how to verify as long as the thread sanitizer gives me "delusional" output 😳

More interesting progress trying to make #swad suitable for very busy sites!

I realized that #TLS (both with #OpenSSL and #LibreSSL) is a major bottleneck. With TLS enabled, I couldn't cross 3000 requests per second, with somewhat acceptable response times (most below 500ms). Disabling TLS, I could really see the impact of a #lockfree queue as opposed to one protected by a #mutex. With the mutex, up to around 8000 req/s could be reached on the same hardware. And with a lockfree design, that quickly went beyond 10k req/s, but crashed. 😆

So I read some scientific papers 🙈 ... and redesigned a lot (). And now it finally seems to work. My latest test reached a throughput of almost 25k req/s, with response times below 10ms for most requests! I really didn't expect to see this happen. 🤩 Maybe it could do even more, didn't try yet.

Open issue: Can I do something about TLS? There must be some way to make it perform at least a bit better...

() edit: Here's the design I finally used, with a much simplified "dequeue" because the queues in question are guaranteed to have only a single consumer: https://dl.acm.org/doi/10.1145/248052.248106

This is going nice so far, I can now correctly sign my #JWT (using #LibreSSL of course, so OpenSSL/LibreSSL will probably be an unconditional dependency for #swad in the next release)