Drew DeVault: Over the past few months, instead of working on our priorities at SourceHut, I have spent anywhere from 20-100% of my time in any given week mitigating hyper-aggressive LLM crawlers at scale. […] If you think these crawlers respect robots.txt then you are several assumptions of good faith removed from reality. These bots […]