Discussion
Loading...

Post

Log in
  • About
  • Code of conduct
  • Privacy
  • Users
  • Instances
  • About Bonfire
spla
spla
@spla@mastodont.cat  ·  activity timestamp 5 months ago

El robot DotBot era l'únic que seguia intentant "escrapejar" mastodont.cat i dic intentant perquè l'estava bloquejant.
L'he afegit a la llista de robots que no vull que "escrapejin"...

https://mastodont.cat/robots.txt

...a les 10:22 i 10:23 ha xarfadejat dos tuts (sense èxit pel meu bloqueig), a les 11 el robot ja ha vist que era a robots.txt i ha parat de xafardejar. A les 11:44 ha tornat a mirar robots.txt no sé si per assegurar-se però ja no xafardeja.

#BotsXafarders #scraping

{"datetime": "14/Aug/2025:10:22:38 +0200", "ip": "216.244.66.250", "method": "GET", "uri": "/@joan/523289", "status": "403", "user_agent": "Mozilla/5.0 (compatible;
DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)", "referer": "(direct)", "server_name": "mastodont.cat", "request_time": "0.000"}

{"datetime": "14/Aug/2025:10:23:09 +0200", "ip": "216.244.66.250", "method": "GET", "uri": "/@kim/99891381465490612", "status": "403", "user_agent": "Mozilla/5.0 (c
ompatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)", "referer": "(direct)", "server_name": "mastodont.cat", "request_time": "90.000"}
{"datetime": "14/Aug/2025:11:00:20 +0200", "ip": "216.244.66.250", "method": "GET", "uri": "/robots.txt", "status": "200", "bytes": "1908", "user_agent": "Mozilla/5
.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)", "referer": "(direct)", "server_name": "mastodont.cat", "request_time": "0.000", "u
pstream_response_time": ""}

{"datetime": "14/Aug/2025:11:44:27 +0200", "ip": "216.244.66.250", "method": "GET", "uri": "/robots.txt", "status": "200", "bytes": "1908", "user_agent": "Mozilla/5
.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)", "referer": "(direct)", "server_name": "mastodont.cat", "request_time": "0.000", "u
pstream_response_time": ne
{"datetime": "14/Aug/2025:10:22:38 +0200", "ip": "216.244.66.250", "method": "GET", "uri": "/@joan/523289", "status": "403", "user_agent": "Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)", "referer": "(direct)", "server_name": "mastodont.cat", "request_time": "0.000"} {"datetime": "14/Aug/2025:10:23:09 +0200", "ip": "216.244.66.250", "method": "GET", "uri": "/@kim/99891381465490612", "status": "403", "user_agent": "Mozilla/5.0 (c ompatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)", "referer": "(direct)", "server_name": "mastodont.cat", "request_time": "90.000"} {"datetime": "14/Aug/2025:11:00:20 +0200", "ip": "216.244.66.250", "method": "GET", "uri": "/robots.txt", "status": "200", "bytes": "1908", "user_agent": "Mozilla/5 .0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)", "referer": "(direct)", "server_name": "mastodont.cat", "request_time": "0.000", "u pstream_response_time": ""} {"datetime": "14/Aug/2025:11:44:27 +0200", "ip": "216.244.66.250", "method": "GET", "uri": "/robots.txt", "status": "200", "bytes": "1908", "user_agent": "Mozilla/5 .0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)", "referer": "(direct)", "server_name": "mastodont.cat", "request_time": "0.000", "u pstream_response_time": ne
{"datetime": "14/Aug/2025:10:22:38 +0200", "ip": "216.244.66.250", "method": "GET", "uri": "/@joan/523289", "status": "403", "user_agent": "Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)", "referer": "(direct)", "server_name": "mastodont.cat", "request_time": "0.000"} {"datetime": "14/Aug/2025:10:23:09 +0200", "ip": "216.244.66.250", "method": "GET", "uri": "/@kim/99891381465490612", "status": "403", "user_agent": "Mozilla/5.0 (c ompatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)", "referer": "(direct)", "server_name": "mastodont.cat", "request_time": "90.000"} {"datetime": "14/Aug/2025:11:00:20 +0200", "ip": "216.244.66.250", "method": "GET", "uri": "/robots.txt", "status": "200", "bytes": "1908", "user_agent": "Mozilla/5 .0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)", "referer": "(direct)", "server_name": "mastodont.cat", "request_time": "0.000", "u pstream_response_time": ""} {"datetime": "14/Aug/2025:11:44:27 +0200", "ip": "216.244.66.250", "method": "GET", "uri": "/robots.txt", "status": "200", "bytes": "1908", "user_agent": "Mozilla/5 .0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)", "referer": "(direct)", "server_name": "mastodont.cat", "request_time": "0.000", "u pstream_response_time": ne
  • Copy link
  • Flag this post
  • Block
sara :hometown:
sara :hometown:
@Caelumtangi@mastodont.cat replied  ·  activity timestamp 5 months ago
@spla saps que llegit des de l'exterior els teus tuts semblen un guió d'una pel·lícula? Scraping CAT 🍿
(gràcies)
  • Copy link
  • Flag this comment
  • Block
spla
spla
@spla@mastodont.cat replied  ·  activity timestamp 5 months ago
@Caelumtangi 😀 és una serie!
  • Copy link
  • Flag this comment
  • Block
sara :hometown:
sara :hometown:
@Caelumtangi@mastodont.cat replied  ·  activity timestamp 5 months ago
@spla 🤣 !
  • Copy link
  • Flag this comment
  • Block

bonfire.cafe

A space for Bonfire maintainers and contributors to communicate

bonfire.cafe: About · Code of conduct · Privacy · Users · Instances
Bonfire social · 1.0.2-alpha.2 no JS en
Automatic federation enabled
Log in
  • Explore
  • About
  • Members
  • Code of Conduct