Post · bonfire.cafe

Xeiaso blog: Amazonbot is finally respecting robots.txt

About time! One down, twenty more to go (give or take 1000)

@cadey@pony.social does this mean default bot policy will now let them through? Neat.

About time! One down, twenty more to go (give or take 1000)

@cadey@pony.social does this mean default bot policy will now let them through? Neat.

Amazonbot is finally respecting robots.txt

No replies yet

Be the first to share your thoughts.