Sidenote: `robots.txt` is more important than before, because without it, whole websites will disappear from Google Search index.
Alan W. Smith wrote:
> “Your site will be removed from Google search results if you don't have a robots.txt file or the Googlebot site crawler can't access it.”
https://www.alanwsmith.com/en/37/wa/jz/s1/
Google themselves say it on a video:
> “Your robots.txt file is the very first thing Googlebot looks for. If it can not reach this file, it will stop and won't crawl the rest of your site. Meaning your pages will remain invisible (on Google).”
#WebDev #Google #GoogleSearch
Daniel Schildt
npub1q5…kt36t
2026-01-16 19:07:24 UTC
Author Public Key
npub1q58635sndayhw6pqx3gqtr5demq2znyujfqv32g900am4mhz9resekt36tSeen on
wss://relay.momostr.pinkPublished at
2026-01-16 19:07:24 UTCEvent JSON
{
"id": "45ddf737bcc24dbeb560a50f0acebef84938d9bb49bf6e6e7934e599573eeaf7",
"pubkey": "050fa8d2136f497768203450058e8dcec0a14c9c9240c8a9057bfbbaeee228f3",
"created_at": 1768590444,
"kind": 1,
"tags": [
[
"t",
"google"
],
[
"t",
"googlesearch"
],
[
"proxy",
"https://mementomori.social/@autiomaa/115906343347655171",
"web"
],
[
"t",
"webdev"
],
[
"proxy",
"https://mementomori.social/users/autiomaa/statuses/115906343347655171",
"activitypub"
],
[
"L",
"pink.momostr"
],
[
"l",
"pink.momostr.activitypub:https://mementomori.social/users/autiomaa/statuses/115906343347655171",
"pink.momostr"
],
[
"-"
]
],
"content": "Sidenote: `robots.txt` is more important than before, because without it, whole websites will disappear from Google Search index.\n\nAlan W. Smith wrote:\n\u003e “Your site will be removed from Google search results if you don't have a robots.txt file or the Googlebot site crawler can't access it.”\nhttps://www.alanwsmith.com/en/37/wa/jz/s1/\n\nGoogle themselves say it on a video:\n\u003e “Your robots.txt file is the very first thing Googlebot looks for. If it can not reach this file, it will stop and won't crawl the rest of your site. Meaning your pages will remain invisible (on Google).”\n\n#WebDev #Google #GoogleSearch",
"sig": "290e69e56765cd0dd97b2d4c89f72794e41db50b8c7f03a5ada5f8f1adbaf1dccd1362bc5796a39c709770d2f701135468017c595d1f6391558b89904cdac4a5"
}