Discussion
Loading...

Post

  • About
  • Code of conduct
  • Privacy
  • Users
  • Instances
  • About Bonfire
Felix Palmen :freebsd: :c64:
@zirias@mastodon.bsd.cafe  路  activity timestamp 3 months ago

I need help. First the question: On #FreeBSD, with all ports built with #LibreSSL, can I somehow use the #clang #thread #sanitizer on a binary actually using LibreSSL and get sane output?

What I now observe debugging #swad:

- A version built with #OpenSSL (from base) doesn't crash. At least I tried very hard, really stressing it with #jmeter, to no avail. Built with LibreSSL, it does crash.
- Less relevant: the OpenSSL version also performs slightly better, but needs almost twice the RAM
- The thread sanitizer finds nothing to complain when built with OpenSSL
- It complains a lot with LibreSSL, but the reports look "fishy", e.g. it seems to intercept some OpenSSL API functions (like SHA384_Final)
- It even complains when running with a single-thread event loop.
- I use a single SSL_CTX per listening socket, creating SSL objects from it per connection ... also with multithreading; according to a few sources, this should be supported and safe.
- I can't imagine doing that on a *single* thread could break with LibreSSL, I mean, this would make SSL_CTX pretty much pointless
- I *could* imagine sharing the SSL_CTX with multiple threads to create their SSL objects from *might* not be safe with LibreSSL, but no idea how to verify as long as the thread sanitizer gives me "delusional" output 馃槼

  • Copy link
  • Flag this post
  • Block
Felix Palmen :freebsd: :c64:
@zirias@mastodon.bsd.cafe replied  路  activity timestamp 3 months ago

For two days straight, I just can't reproduce #swad #crashing with *anything* in place (#clang #sanitizer instrumentation, attached #debugger like #lldb) that could give me the slightest hint what's going wrong. 馃槨

But it *does* crash when "unobserved". And it looks like this is happening a lot sooner (or, more often?) when using #LibreSSL ... but I also suspect this could be a red herring in the end.

Situation reminds me of my physics teacher back at school, who used to say something in german I just can't ever forget:

"Wer misst, misst Mist."

Feeble attempt in english would be "the one who measures measures crap", it was his humorous way to bring one consequence of #Heisenberg's indeterminacy principle to the point. And indeed, #debugging computer programs always suffers from similar problems...

  • Copy link
  • Flag this comment
  • Block
Felix Palmen :freebsd: :c64:
@zirias@mastodon.bsd.cafe replied  路  activity timestamp 3 months ago

Oh boy, I have a lead! And it's NOT related to #TLS. I finally noticed another pattern: #swad only #crashed when running as a #daemon. The daemonizing wasn't the problem, but the default logging configuration attached to it: "fake async", by letting a #threadpool job do the logging.

Forcing THAT even when running in foreground, I can finally reproduce a crash. And I wouldn't be surprised if that was actually the reason for crashing "pretty quickly" with #LibreSSL (and only rarely with #OpenSSL), I mean, something going rogue in your address space can have the weirdest effects.

  • Copy link
  • Flag this comment
  • Block
Felix Palmen :freebsd: :c64:
@zirias@mastodon.bsd.cafe replied  路  activity timestamp 3 months ago

Analyzed and THIS is the core part of the fix:

https://github.com/Zirias/poser/commit/b1aa9784955db4712d695d217344cb65bbff6ed6

I'll have to revisit that later, it's probably wasteful as it is right now, I should come up with some better idea to implement async logging later. But it DOES fix the bug I finally identified that could make #swad crash.

Now, just running it, observing it, stress-testing it from time to time ... wish me luck that THIS was indeed the only reason left for crashing in "normal operation".

I already know there's more work to do regarding reload (SIGHUP) with the multi-threaded approach, it's currently unsafe unfortunately, must be fixed before the next release.

  • Copy link
  • Flag this comment
  • Block
Felix Palmen :freebsd: :c64:
@zirias@mastodon.bsd.cafe replied  路  activity timestamp 3 months ago

Update: It finally also crashed once with OpenSSL ... of course only testing a build *without* thread sanitizer, after massively stressing it over and over again without crashing. So, there's still also another issue hiding. 馃槥

Nevertheless, with LibreSSL, I can reliably make it crash pretty quickly under heavy load, and the sanitizer output looks useless as described ... so the question remains 馃

  • Copy link
  • Flag this comment
  • Block
Log in

bonfire.cafe

A space for Bonfire maintainers and contributors to communicate

bonfire.cafe: About 路 Code of conduct 路 Privacy 路 Users 路 Instances
Bonfire social 路 1.0.0-rc.2.21 no JS en
Automatic federation enabled
  • Explore
  • About
  • Members
  • Code of Conduct
Home
Login