Site outage. I blame you, Robchu!

[Rob*]

Ah, right. The outage. Funny story behind that...

The recent outage was caused by me - in part because I happened to be sleeping (lazy, I know) when the outage happened at 7am in the UK. I'd come off of working on a project for twenty or so hours so I was, for all intents and purposes, dead to the world.

Currently the disaster recovery works as follows; Something goes down - a script I wrote sends me an SMS saying something along the lines of "Fuck, something broke on: <server hostname>".

Generally within about five minutes of that my phone explodes with WhatsApp messages from the guys saying something like "LEGIT EMERGENCY, seriously not joking. Are you there?" or " I need you. Bad. One of the sites is down and I can't text you or call you" (I wish I was joking but that is the messages I usually receive)

So in this occasion exactly that happened - while I was sleeping. Because I missed the messages and because Petie was granted the root password (as ratified by the elders of the internet) he did what every good tech analyst does - he tried turning it off and on again but he had to do it while running out of the door and so didn't monitor the server coming back online. Turns out that there was an error in the apache config file caused (as you rightly guessed) by the process of moving Muscle Tower and the other sites Petie operates on to the tSRVGFacts server.



When I got up I fixed the config file and started Apache and all was solved.

I should add a disclaimer to my profile now that Petie has root access.