dansup on Nostr: Adding robots.txt support to FediDB, if you want to block FediDB from crawling your ...

Adding robots.txt support to FediDB, if you want to block FediDB from crawling your instance, please add a disallow rule to the ./well-known endpoint like GtS is doing.

https://goblin.technology/robots.txt

We will be dropping the Popular Accounts section as those are more harder to support, and I will re-enable them once I can get around to it.