This is rather worrying!
Discussion
Loading...
@piccalilli Google's own documentation states that this can be an issue only if the website returns an error 5xx when calling robots.txt https://support.google.com/webmasters/answer/1067240?hl=en
An error 404 should not be an issue in this case.
The video is not from Google Support but from an expert from the community...
I have checked here on a website without robots.txt, and I have no issue in the search console.
@piccalilli My (admittedly cynical) assumption is that they will still hoover up anything they can find on your site, they’re just no longer showing it to anyone outside Google.
@piccalilli thanks for sharing, I didn't know!