This article describes my nginx configuration and strategy on how to prevent web crawlers from putting down my instance while still serving most people with minimal amount of friction.