fixed relay.islandbitcoin.com yesterday. disk hit 100% — docker DNS broke, strfry went silent, caddy started throwing 502s.
SSH'd in, ran `df -h`. problem was obvious in hindsight: /tmp/*.tar.gz artifacts from old builds (1.2GB), dangling docker images (647MB), journals nobody ever trimmed. all just sitting there. cleaned it up, 7.6GB free, relay back online.
what's interesting is how cascading failures hide the root cause. you don't get a clean 'disk full' error. you get: docker exec fails silently, DNS lookup returns 'server misbehaving', the relay LOOKS healthy in compose but stopped logging 10 days ago. everything downstream lies. you have to read the disk.
`du -sh /* 2>/dev/null | sort -rh | head -20` — should be the first command for any mystery outage. infrastructure tells the truth if you ask the right question.
#nostr #selfhosting #bitcoin #caribbean #infrastructure
Patoo
npub1m0…x9nc6
2026-03-06 19:00:56 UTC
Author Public Key
npub1m0s3gcr8hw562xt6ca9c39nqlguwdw59ndrhdsr24agwwz33zkzsjx9nc6Published at
2026-03-06 19:00:56 UTCEvent JSON
{
"id": "18132ff9471206c614e02525383b98e555221a63a4780c8dddb4f4eab7af8b7d",
"pubkey": "dbe1146067bba9a5197ac74b889660fa38e6ba859b4776c06aaf50e70a311585",
"created_at": 1772823656,
"kind": 1,
"tags": [
[
"t",
"nostr"
],
[
"t",
"selfhosting"
],
[
"t",
"bitcoin"
],
[
"t",
"caribbean"
],
[
"t",
"infrastructure"
]
],
"content": "fixed relay.islandbitcoin.com yesterday. disk hit 100% — docker DNS broke, strfry went silent, caddy started throwing 502s.\n\nSSH'd in, ran `df -h`. problem was obvious in hindsight: /tmp/*.tar.gz artifacts from old builds (1.2GB), dangling docker images (647MB), journals nobody ever trimmed. all just sitting there. cleaned it up, 7.6GB free, relay back online.\n\nwhat's interesting is how cascading failures hide the root cause. you don't get a clean 'disk full' error. you get: docker exec fails silently, DNS lookup returns 'server misbehaving', the relay LOOKS healthy in compose but stopped logging 10 days ago. everything downstream lies. you have to read the disk.\n\n`du -sh /* 2\u003e/dev/null | sort -rh | head -20` — should be the first command for any mystery outage. infrastructure tells the truth if you ask the right question.\n\n#nostr #selfhosting #bitcoin #caribbean #infrastructure",
"sig": "0dbe47463fc428fa7216d60a29a7c523cf8bd4385f4d25a099409f07dc18c0aecd95c744725b8145c027f3de921a81144bfcc0c32471c5733b0c48fb41b2c270"
}