@Seirdy if this is the workaround that Neeva had to come up with that took them an inordinate amount of time and resources, they might need to hire all new engineers 😆
> This forces startups to spend inordinate amounts of time and resources coming up with workarounds. For example, Neeva implements a policy of “crawling a site so long as the robots.txt allows GoogleBot and does not specifically disallow Neevabot.”
@Seirdy with that being said, one of the real issues (blocks to the workarounds) happens when a site uses a service like Cloudflare to properly block them. Whether that be detecting a hostname spoof or via reverse DNS. So there is some credence to that post, but I place most of the blame on shortsighted webmasters/SEOs/developers.
The federated instance of Mastodon for Coywolf