#Tag
I just published a blog post summing up my most pertinent thoughts about dealing with badly-behaved web-scraping bots:
https://cryptography.dog/blog/AI-scrapers-request-commented-scripts/
It isn't exactly a Hallowe'en-themed article, but today is the 31st and the topic is concerned with pranking people who come knocking on my website's ports, so it's somewhat appropriate.
#infosec #bots #halloween #scrapers #AI #someMoreHashtagsHere
Since so many people are boosting this thread I think I'll take the opportunity to mention that I'm available for hire on a part-time or contract basis.
Feel free to reach out if you like my ideas about computer-related topics and have both the budget and need of someone who has such ideas.
I can be reached by Fediverse DM or the contact form on my website:
I just published a blog post summing up my most pertinent thoughts about dealing with badly-behaved web-scraping bots:
https://cryptography.dog/blog/AI-scrapers-request-commented-scripts/
It isn't exactly a Hallowe'en-themed article, but today is the 31st and the topic is concerned with pranking people who come knocking on my website's ports, so it's somewhat appropriate.
#infosec #bots #halloween #scrapers #AI #someMoreHashtagsHere
The Code4Lib Journal – Mitigating Aggressive Crawler Traffic in the Age of Generative AI: A Collaborative Approach from the University of North Carolina at Chapel Hill Libraries https://journal.code4lib.org/articles/18489 #AI #libraries #bots #crawlers
The Code4Lib Journal – Mitigating Aggressive Crawler Traffic in the Age of Generative AI: A Collaborative Approach from the University of North Carolina at Chapel Hill Libraries https://journal.code4lib.org/articles/18489 #AI #libraries #bots #crawlers
I just installed the Ultimate Bad Bot Blocker on our Mastodon server. It blocks bad bots, spam referrers, vulnerability scanners, malicious user agents, malware, adware, ransomware, and other harmful bots. It also includes anti-DDoS protection and a Fail2Ban jail for repeat offenders
https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker
🤖 How 73% of Your E-commerce Visitors Could Be Fake
｢ The purpose is likely to manipulate e-commerce metrics, perhaps to influence a site's internal recommendation algorithms or to make cart abandonment rates look normal amidst a sea of other non-purchasing bots ｣
https://joindatacops.com/resources/how-73-of-your-e-commerce-visitors-could-be-fake
Grrr. Getting hit by a fairly aggressive scraper on my blog.
Same user-agent, but different IP addresses - all from China.
It is using JS - but I really don't want to put up a CAPTCHA or similar. Deters too many legitimate visitors.
Anyone have a simple solution for WordPress / PHP?
We're excited to announce the release of BotKit 0.3.0! This release marks a significant milestone as #BotKit now supports #Node.js alongside #Deno, making it accessible to a wider audience. The minimum required Node.js version is 22.0.0. This dual-runtime support means you can now choose your preferred #JavaScript runtime while building #ActivityPub #bots with the same powerful BotKit APIs.
One of the most requested features has landed: poll support! You can now create interactive polls in your #bot messages, allowing followers to vote on questions with single or multiple-choice options. Polls are represented as ActivityPub Question objects with proper expiration times, and your bot can react to votes through the new onVote event handler. This feature enhances engagement possibilities and brings BotKit to feature parity with major #fediverse platforms like Mastodon and Misskey.
// Create a poll with multiple choicesawait session.publish(textWhat's your favorite programming language?, { class: Question, poll: { multiple: true, // Allow multiple selections options: ["JavaScript", "TypeScript", "Python", "Rust"], endTime: Temporal.Now.instant().add({ hours: 24 }), },});// Handle votesbot.onVote = async (session, vote) => { console.log(</span><span>${</span><span>vote</span><span>.</span><span>actor</span><span>}</span><span> voted for "</span><span>${</span><span>vote</span><span>.</span><span>option</span><span>}</span><span>");};The web frontend has been enhanced with a new followers page, thanks to the contribution from Hyeonseo Kim (@gaebalgom)! The /followers route now displays a paginated list of your bot's followers, and the follower count on the main profile page is now clickable, providing better visibility into your bot's audience. This improvement makes the web interface more complete and user-friendly.
For developers looking for alternative storage backends, we've introduced the SqliteRepository through the new @fedify/botkit-sqlite package. This provides a production-ready SQLite-based storage solution with ACID compliance, write-ahead logging (WAL) for optimal performance, and proper indexing. Additionally, the new @fedify/botkit/repository module offers MemoryCachedRepository for adding an in-memory cache layer on top of any repository implementation, improving read performance for frequently accessed data.
This release also includes an important security update: we've upgraded to #Fedify 1.8.8, ensuring your bots stay secure and compatible with the latest ActivityPub standards. The repository pattern has been expanded with new interfaces and types like RepositoryGetMessagesOptions, RepositoryGetFollowersOptions, and proper support for polls storage through the KvStoreRepositoryPrefixes.polls option, providing more flexibility for custom implementations.
We're excited to announce the release of BotKit 0.3.0! This release marks a significant milestone as #BotKit now supports #Node.js alongside #Deno, making it accessible to a wider audience. The minimum required Node.js version is 22.0.0. This dual-runtime support means you can now choose your preferred #JavaScript runtime while building #ActivityPub #bots with the same powerful BotKit APIs.
One of the most requested features has landed: poll support! You can now create interactive polls in your #bot messages, allowing followers to vote on questions with single or multiple-choice options. Polls are represented as ActivityPub Question objects with proper expiration times, and your bot can react to votes through the new onVote event handler. This feature enhances engagement possibilities and brings BotKit to feature parity with major #fediverse platforms like Mastodon and Misskey.
// Create a poll with multiple choicesawait session.publish(textWhat's your favorite programming language?, { class: Question, poll: { multiple: true, // Allow multiple selections options: ["JavaScript", "TypeScript", "Python", "Rust"], endTime: Temporal.Now.instant().add({ hours: 24 }), },});// Handle votesbot.onVote = async (session, vote) => { console.log(</span><span>${</span><span>vote</span><span>.</span><span>actor</span><span>}</span><span> voted for "</span><span>${</span><span>vote</span><span>.</span><span>option</span><span>}</span><span>");};The web frontend has been enhanced with a new followers page, thanks to the contribution from Hyeonseo Kim (@gaebalgom)! The /followers route now displays a paginated list of your bot's followers, and the follower count on the main profile page is now clickable, providing better visibility into your bot's audience. This improvement makes the web interface more complete and user-friendly.
For developers looking for alternative storage backends, we've introduced the SqliteRepository through the new @fedify/botkit-sqlite package. This provides a production-ready SQLite-based storage solution with ACID compliance, write-ahead logging (WAL) for optimal performance, and proper indexing. Additionally, the new @fedify/botkit/repository module offers MemoryCachedRepository for adding an in-memory cache layer on top of any repository implementation, improving read performance for frequently accessed data.
This release also includes an important security update: we've upgraded to #Fedify 1.8.8, ensuring your bots stay secure and compatible with the latest ActivityPub standards. The repository pattern has been expanded with new interfaces and types like RepositoryGetMessagesOptions, RepositoryGetFollowersOptions, and proper support for polls storage through the KvStoreRepositoryPrefixes.polls option, providing more flexibility for custom implementations.
#RIP botsin.space, and bot hosting on Glitch (thanks Fastly);
https://botwiki.org/blog/a-decade-of-botwiki/
I discovered botwiki through a web search, after finding the tombstone on botsin.space. Thanks to @stefan for the blog posts there about all this, and the link to the detailed postmortem by the founder and admin of botsin.space;
https://muffinlabs.com/posts/2024/12/21/12-21-botsin-space-post-mortem/
Lots to think about in there.
No sé muy bien cómo comenzó mi afición por hacer #bots. Es verdad que son un buen ejercicio de #programación, y hay pocas cosas que me gusten más que programar. Además, sirven para divulgar las cosas que me gustan y denunciar las que no.
Cuando hablo de bots, me refiero a cuentas automatizadas que postean regularmente sobre un tema en concreto. Algunos también responden cuando se les menciona.
Voy a hacer un hilo repasando todos mis bots, ordenados según el número de seguidores.
¡Dentro hilo! 👇
No sé muy bien cómo comenzó mi afición por hacer #bots. Es verdad que son un buen ejercicio de #programación, y hay pocas cosas que me gusten más que programar. Además, sirven para divulgar las cosas que me gustan y denunciar las que no.
Cuando hablo de bots, me refiero a cuentas automatizadas que postean regularmente sobre un tema en concreto. Algunos también responden cuando se les menciona.
Voy a hacer un hilo repasando todos mis bots, ordenados según el número de seguidores.
¡Dentro hilo! 👇
The times we live in...
And it's not like people and espechally businesses are unwilling to pay for good tools, cuz not only is HootSuite still around, but @tapbots, who made the excellent TweetBot nowadays make @ivory, an excellent #ActivityPub client with focus on #Mastodon that people are willing to pay a #subscription for!
Like: There's not much of a technical reason something similar to @zulip 's #TUI - based client could exit.
If people can build #bots for shite like #discord, #Telegram, etc. then why not make an actually good client ???
Attention server admins! Yesterday I've read a post by @simon_brooke how nasty AI scraper bots are attacking his self-hosted @forgejo instance. Soon after I'm seeing unusual, periodic traffic spikes on mine and again - dominated by OpenAI, but some other freeloaders too:
20.171.207.41 GPTBot/1.2
85.208.96.211 SemrushBot/7~bl
54.36.148.64 AhrefsBot/7.0
114.119.139.53 PetalBot
With GPTBot and SemrushBot attacking hardest 
They've been hammering my little server periodically today as well, slowing down my instance dramatically as if I was experiencing malicious DDoS attack 
Well, in a sense it is one 
Watch out - it seems corporate AI techbros learned to scrape 
content and starts doing it on a massive scale 
Remember when @Codeberg was (and repeatedly is) hit?
For now blocked IP ranges and User-Agent combinations, not sure for how long that will be enough 
Please boost for visibility and be prepared!
#forgejo #developerlife #coding #attack #techbros #aislop #openai #bots #ddos
🫠 Brave new goo
｢ Google is essentially training bots to communicate with each other and execute tasks, including financial transactions. As Semafor notes, that protocol will likely be the first step in building a brave new ad industry where AI agents become the consumer, since humans have delegated such menial tasks like "buying stuff" to them ｣
Attention server admins! Yesterday I've read a post by @simon_brooke how nasty AI scraper bots are attacking his self-hosted @forgejo instance. Soon after I'm seeing unusual, periodic traffic spikes on mine and again - dominated by OpenAI, but some other freeloaders too:
20.171.207.41 GPTBot/1.2
85.208.96.211 SemrushBot/7~bl
54.36.148.64 AhrefsBot/7.0
114.119.139.53 PetalBot
With GPTBot and SemrushBot attacking hardest
They've been hammering my little server periodically today as well, slowing down my instance dramatically as if I was experiencing malicious DDoS attack Well, in a sense it is one
Watch out - it seems corporate AI techbros learned to scrape content and starts doing it on a massive scale Remember when @Codeberg was (and repeatedly is) hit?
For now blocked IP ranges and User-Agent combinations, not sure for how long that will be enough
Please boost for visibility and be prepared!
#forgejo #developerlife #coding #attack #techbros #aislop #openai #bots #ddos
I noticed that a lot of the crawlers/bots we see on www.bbc.co.uk & www.bbc.com are spoofed e.g. a "Meta" crawler coming from 10s of different small ISPs across the world (the real one comes from a Meta ASN).
I deployed a change this morning which adds source ASN validation (alongside user-agent string analysis) to our "known crawlers/bots" classifier & well, the results speak for themselves. Attached graphs show RPS from "known crawlers/bots" to www.bbc.co.uk & www.bbc.com.
#WebDev#BBC#Bots
There's a lot that could still be improved in #swad, but I don't get that "proof of work" idea out of my mind, so I started a branch to work on it:
https://github.com/Zirias/swad/pull/1
I really think it makes sense when you want some publicly known "guest login" which is still protected against #bots. Not sure yet whether this will succeed, we will see!
It certainly won't be as "fancy" as #anubis, but do the same thing functionally: Require the client to find a #nonce that, combined with a server-provided #challenge, hashes to something with 'n' leading zeros using #sha256. In contrast to anubis, swad won't have to proxy everything (but rely on nginx' auth_request), and no challenge will be issued when the user logs in with credentials some other credentials checker accepts.
A space for Bonfire maintainers and contributors to communicate
