I want to prevent LLM web-scraping bots from stealing the content from @cryogenix but it's a Web 1.x website that has a no-JavaScript policy (except for designated sections)...

So I can't use Anubis, I don't and won't use CloudFlare (because of JS injecion and privacy concerns), and CrowdSec doesn't seem to protect against it.

What can we realistically do? I wouldn't want to make it a Tor/Onion-only website.

#AI #LLM #Spam #anubis

I learned some things about how #Anubis works today thanks to an article by Tavis Ormandy showing how to obtain a signed auth cookie using 40 lines of C to brute-force the sha256 challenge: https://lock.cmpxchg8b.com/anubis.html Even on my slow A311D MNT Reform running the example only took 0.037 seconds.

@sebsauvage

Oups...

The #AI bots are almost certainly bypassing the #Anubis proxy on my #Kiwix server.

The #AI bots are almost certainly bypassing the #Anubis proxy on my #Kiwix server.

Does anyone use #Anubis behind #OpenBSD’s relayd?

Does anyone use #Anubis behind #OpenBSD’s relayd?

People, please try this #serveronly alternative to #anubis to keep. #ai at bay
Thanks to @alcinnz for the efforts

People, please try this #serveronly alternative to #anubis to keep. #ai at bay
Thanks to @alcinnz for the efforts

poke @sebsauvage
Thank you

I don’t know who needs to hear this but compiling www/anubis from ports-current on #OpenBSD -stable works ok. And indeed, sending the #Anubis package to a remote -stable server and use ˋpkg_add -D unsigned ./anubis(…)` to upgrade also works.

(currently live on the blog which has an unpronounceable name ;-)

I don’t know who needs to hear this but compiling www/anubis from ports-current on #OpenBSD -stable works ok. And indeed, sending the #Anubis package to a remote -stable server and use ˋpkg_add -D unsigned ./anubis(…)` to upgrade also works.

(currently live on the blog which has an unpronounceable name ;-)

#Anubis possède désormais un challenge qui ne nécessite pas Javascript, permettant aux clients sans Javascript (ou avec Javascript désactivé ou filtré) de passer avec succès le challenge.

L'auteur a testé pas mal de navigateur : même les navigateurs en ligne de commande et le navigateur intégré à emacs passent.

Cela devrait réduire la quantité de clients bloqués.
https://anubis.techaro.lol/blog/release/v1.20.0/#no-js-challenge

I just updated #Anubis for the @FreeCAD infrastructure. There's now support for challenges without requiring client-side #JavaScript, and a "success... failure!"-type bug affecting #Chromium should now be fixed. Please let me know if you run into any issues! #FreeCAD

I just updated #Anubis for the @FreeCAD infrastructure. There's now support for challenges without requiring client-side #JavaScript, and a "success... failure!"-type bug affecting #Chromium should now be fixed. Please let me know if you run into any issues! #FreeCAD

Just released: #swad 0.12 🥂

swad is the "Simple Web Authentication Daemon". It basically offers adding form + #cookie #authentication to your reverse proxy (designed for and tested with #nginx "auth_request"). I created it mainly to defend against #malicious_bots, so among other credential checker modules for "real" logins, it offers a proof-of-work mechanism for guest logins doing the same #crypto #challenge known from #Anubis.

swad is written in pure #C with minimal dependencies ( #zlib, #OpenSSL or compatible, and optionally #PAM), and designed to work on any #POSIX system. It compiles to a small binary (200 - 300 kiB depending on compiler and target platform).

This release brings (among a few bugfixes) improvements to make swad fit for "heavy load" scenarios: There's a new option to balance the load across multiple service worker threads, so all cores can be fully utilized if necessary, and it now keeps lots of transient objects in pools for reuse, which helps to avoid memory fragmentation and ultimately results in lower overall memory consumption.

Read more about it, download the .tar.xz, build and install it .... here:

https://github.com/Zirias/swad

Just released: #swad 0.11 -- the session-less swad is done!

Swad is the "Simple Web Authentication Daemon", it adds cookie/form #authentication to your reverse #proxy, designed to work with #nginx' "auth_request". Several modules for checking credentials are included, one of which requires solving a crypto challenge like #Anubis does, to allow "bot-safe" guest logins. Swad is written in pure #C, compiles to a small (200-300kiB) binary, has minimal dependencies (zlib, OpenSSL/LibreSSL and optionally libpam) and should work on many #POSIX-alike systems (#FreeBSD tested a lot, #Linux and #illumos also tested)

This release is the first one not to require a server-side session (which consumes a significant amount of RAM on really busy sites), instead signed Json Web Tokens are now implemented. For now, they are signed using HMAC-SHA256 with a random key generated at startup. A future direction could be support for asymmetric keys (RSA, ED25519), which could open up new possibilities like having your reverse proxy pass the signed token to a backend application, which could then verify it, but still not forge it.

Read more, grab the latest .tar.xz, build and install it ... here: 😎

https://github.com/Zirias/swad

Hopefully, there will be another release of #swad soon!

Looking at my test results again, performance should be okay at least for moderately busy sites ... the 1000 requests per second I observed included actual logins, and I didn't even test whether it would also handle more (it probably would), the only issue was with resolving remote names (with that, around 30% of these requests failed because the thread pool was clogged with jobs all waiting for some DNS response), and the recommendation would be: just disable that feature if your site is a busy one.

But I'm really unhappy with RAM usage going up so much. Almost 100MiB resident set after seeing 1000 unique clients all attempting a login is a lot after all.

So, I'll try to move swad to a session-less design. It can't be fully stateless, a rate limiter will be needed, but maybe I can optimize a bit on that.

But the sessions could be replaced. They're currently used for two things:

* Store actual auth information. This could be stored in signed JWTs (json web tokens) on the client instead. I'm already starting to add JSON support to my poser lib 😉

* Store the random challenge for the #anubis-like proof-of-work checker. Could do the same as anubis here: Derive the challenge from request metadata instead, including a timestamp.

Will be quite some work, but could be doable.

Just released: #swad 0.10

https://github.com/Zirias/swad/releases/tag/v0.10

Swad is the "Simple Web Authentication Daemon". If you're looking for a way to add #authentication (and/or proof-of-work access as known from #anubis) to your #nginx reverse proxy -- without adding yet another reverse proxy -- swad could be for you! It's written in pure #C, has few external dependencies (just zlib, and optionally OpenSSL/Libressl and/or libpam) and compiles to a pretty small binary. It's designed for usage with nginx' 'auth_request'.

Swad is tested on #FreeBSD, some basic functionality tests were also done on #Linux and #illumos (descendant from #solaris). It should build and work on most #POSIX-alike systems.

This release mainly brings performance improvements and a few bugfixes. It's now stress-tested with Apache jmeter, verifying it can deal with at least 1000 requests per second on my personal (somewhat limited) FreeBSD host machine.