Rhaegal High Load / Failure

09:46 We are seeing high loads on the server Rhaegal which is causing sites to time out. We are investigating and should have a fix soon.

10:04 We’ve have found the cause of the high load, and everything should be returning to normal with the next minute or so.

10:10 Our fix has not worked, so we are implementing another fix which requires a reboot of the server. All sites will be offline for at least the next 15 minutes while the server is rebooting.

10:45 Email sent to all clients on this server:

Dear client

The server Rhaegal which hosts the website for {DOMAIN} is offline. We have as yet been unable to find the root cause, but we suspect that it is a hardware failure. While the technicians investigate, we will begin restoring the {DOMAIN} website to the server Hoppler (IP address: 156.38.137.210 ) in the event that the server Rhaegal cannot be restored to a working condition.

We will keep you updated while we work through this.

14:35 We are 30% through the restoration process. If your domain is not using our nameservers, please update its DNS to point to 156.38.137.210 now.

15:30 We are still attempting to recover data from the crashed server, although the chances of that happening are not looking good. If we are unable to recover data from the server, then your email accounts and emails (which are never backed up by Texo) will be lost. As soon as your website is live again, please login to your cpanel and recreate any email accounts you need.

16:15 The restoration of websites to the new server is at 60% and should be done within another few hours

18:25 We have around 50 website left to restore

19:50 All websites have been restored to the new server. If you see any problems with your website, please submit a support ticket. If your domain is not using our nameservers, and you have not yet updated its A and WWW records to point to 156.38.137.210 then please do so now.

Viserion: MySQL failures

4PM MySQL is failing on this server, and a fix is being worked on. Occasional “failed to connect to mysql server” or similar messages might appear from time to time when browsing your website.

8:15PM We are starting an emergency migration of all accounts on this server to the server skizzle.texo.co.za

Multiple Servers Offline

12:50 xneelo appears to have fixed their electrical problem, and our servers are now booting up. I expect everything to be back to normal within 10 to 15 minutes.

12:30 Update from Xneelo (Hetzner):

xneelostatus.co.za/notice/4815/

xneelo network unreachable – Samrand data centre

Notice ID:#4815
Start: 03 Dec 2019 12:15 SAST
Maintenance Window: 30min
Issue: Electrical Failure

12:20PM There appears to be a problem in Hetzner’s Samrand datacentre which has taken all servers, including their own website, offline.

Dothraki offline

17:40 WordPress MAX server Dothraki has lost connection and is offline. Technicians are investigating.

17:52 Dothraki is back online. No reboot was needed, so there won’t be any post-boot high disk IO slowing sites down. Apologies for the inconvenience.

Outage: Johannesburg

03:00AM A number of servers are offline, due to a problem affecting Hetzner’s Samrand datacenter.

Update: Hetzner informs me that the issue is due to a power outage, affecting “a large number of rows [of servers]” and that their technicians are working on the problem.

05:40AM Hetzner appears to have fixed their problem, and servers are booting up. There will be degraded performance (sluggish websites, timeouts) until disk I/O settles down within 30 minutes or so.