The time we really went down hard was when our dedi provider (which was otherwise great!) overeagerly blackholed DDoS traffic…
…blackholing also our production back-end server.
Took us 45min to deal with this, mainly because I was out at lunch and for *once* I did not take my phone with me. While a certain Smári McCarthy (npub1yef…md7z) happened to be on vacation literally on the other side of the globe.
Dealing with this meant pushing a quick config change to the fasadas to switch to the warm spare back-end.