I noticed that a lot of the crawlers/bots we see on www.bbc.co.uk & www.bbc.com are spoofed e.g. a "Meta" crawler coming from 10s of different small ISPs across the world (the real one comes from a Meta ASN).
I deployed a change this morning which adds source ASN validation (alongside user-agent string analysis) to our "known crawlers/bots" classifier & well, the results speak for themselves. Attached graphs show RPS from "known crawlers/bots" to www.bbc.co.uk & www.bbc.com.
#WebDev#BBC#Bots
Graph of requests from "known crawlers/bots" over time to www.bbc.com for today.
The graph is relatively steady until about 08:45 UTC when it drops by about 90%
Graph of requests from "known crawlers/bots" over time to www.bbc.co.uk for today.
The graph is relatively steady until about 08:45 UTC when it drops by about 90%