Category: outages

18:05 We’re seeing very high loads on Lannister which has resulted in websites not being served. We are investigating.

18:28 The problem has been found and fixed.

17:40 WordPress MAX server Dothraki has lost connection and is offline. Technicians are investigating.

17:52 Dothraki is back online. No reboot was needed, so there won’t be any post-boot high disk IO slowing sites down. Apologies for the inconvenience.

06:00 AM As a result of the power failure at Hetzner this morning, mysql on Greyworm has failed, and needs to be repaired. We are hoping to get this fixed as soon as we can.

6:50 AM MySQL has been repaired and is running.

03:00AM A number of servers are offline, due to a problem affecting Hetzner’s Samrand datacenter.

Update: Hetzner informs me that the issue is due to a power outage, affecting “a large number of rows [of servers]” and that their technicians are working on the problem.

05:40AM Hetzner appears to have fixed their problem, and servers are booting up. There will be degraded performance (sluggish websites, timeouts) until disk I/O settles down within 30 minutes or so.

13:20 We’re seeing timeouts and slow responses from websites hosted on Bumblebee, and are investigating the problem.

13:30 The problem has been found and fixed, and everything should be back to normal.

9:50PM We’re working hard to mitigate a DDOS targeting two sites on this server. Sites on this server will be unresponsive until the attack has been stopped.

10:20PM Almost there, we’ve stopped 90% of the attack, now blocking the last sources.

10:33PM Things are almost back to normal. Server iowait is a bit high, so sites will be a bit slow until that is resolved. Thanks for your patience, and apologies for this inconvenience.

21 September 2:50PM

Server Bumblebee 156.38.146.98 is offline. Technicians are investigating.

3:15PM The problem has been found, and technicians are working to fix the hardware fault.

3:26PM The problem has been fixed, and the server will be booted up shortly

3:37PM Still waiting for the server to be booted.

3:45 Server is booting up now. For the next 10-20 minutes or so, websites and emails will be slow while disk IO is high after the reboot.

17 September 2018 8:38PM

We’re working to mitigate a DDOS which is preventing the server from serving sites successfully.

8:46 PM

The DDOS has been mitigated. Apologies for the inconvenience.

10:50AM Apache on Lannister is encountering an error, and is not serving any websites on this server. We’re working on the problem.

10:57AM The problem has been fixed, and Apache restarted. All websites should be functioning correctly now.

14:00 Greyworm is currently offline due to a DDOS attack. We’re working on mitigating it.

14:17 We’ve null-routed the target of the attack, and this seems to have stopped the DDOS. Unless another site on this server is targeted, the server should now run normally. Please accept our apologies for this downtime.