Eleanor Saitta on Nostr: If your have topic recognizers for queries and for crawlers and they are a) always ...

If your have topic recognizers for queries and for crawlers and they are a) always right, and b) never change, you could shard the index below that point. Crawlers could still be distributed and indexes wouldn't need to reference/communicate, but every crawler would need to update every index index shard. However, A is improbable and B is impossible.

You might be able to hack something together that could somehow meaningfully federate, but the level of computational inefficiency would be staggering. If we don't have a nonprofit that could run a single centralized web scale search engine, we definitely don't have ten thousand nonprofits that could together spend two orders of magnitude more to run a federated one.
Kevin Riggle (npub1lhc…h9l6) The PeARS Project (npub105n…85x4)