Author: Texo [Steve]

7:48PM The MySQL database server on Tankwa has gone offline. We are working to fix the problem.

8:07PM The MySQL server has been repaired and restarted. Apologies for the inconvenience.

11:40 AM The server Bumblebee (156.38.146.98) is offline. We are investigating.

11:55 AM We have found the problem, a DDOS against one website on the server, and have resolved it.

09:46 We are seeing high loads on the server Rhaegal which is causing sites to time out. We are investigating and should have a fix soon.

10:04 We’ve have found the cause of the high load, and everything should be returning to normal with the next minute or so.

10:10 Our fix has not worked, so we are implementing another fix which requires a reboot of the server. All sites will be offline for at least the next 15 minutes while the server is rebooting.

10:45 Email sent to all clients on this server:

Dear client

The server Rhaegal which hosts the website for {DOMAIN} is offline. We have as yet been unable to find the root cause, but we suspect that it is a hardware failure. While the technicians investigate, we will begin restoring the {DOMAIN} website to the server Hoppler (IP address: 156.38.137.210 ) in the event that the server Rhaegal cannot be restored to a working condition.

We will keep you updated while we work through this.

14:35 We are 30% through the restoration process. If your domain is not using our nameservers, please update its DNS to point to 156.38.137.210 now.

15:30 We are still attempting to recover data from the crashed server, although the chances of that happening are not looking good. If we are unable to recover data from the server, then your email accounts and emails (which are never backed up by Texo) will be lost. As soon as your website is live again, please login to your cpanel and recreate any email accounts you need.

16:15 The restoration of websites to the new server is at 60% and should be done within another few hours

18:25 We have around 50 website left to restore

19:50 All websites have been restored to the new server. If you see any problems with your website, please submit a support ticket. If your domain is not using our nameservers, and you have not yet updated its A and WWW records to point to 156.38.137.210 then please do so now.

13:00 We are seeing errors on the server Bumblebee which has taken sites offline. We are investigating .

13:12 We have found the cause, and are applying a fix now.

23:10 All sites on Skizzle are showing MySQL errors because MySQL is failing.
We are investigating the problem and hope to find a fix soon.

23:40 We have found and fixed the problem. Sincere apologies for the inconvenience.

14 July 2020 10:55AM
We’re seeing high loads on the Tyrion server, which is causing slowness and timeouts for websites. We’re investigating.

11:04AM
The problem has been found, and everything should return to normal shortly.

4PM MySQL is failing on this server, and a fix is being worked on. Occasional “failed to connect to mysql server” or similar messages might appear from time to time when browsing your website.

8:15PM We are starting an emergency migration of all accounts on this server to the server skizzle.texo.co.za

12:50 xneelo appears to have fixed their electrical problem, and our servers are now booting up. I expect everything to be back to normal within 10 to 15 minutes.

12:30 Update from Xneelo (Hetzner):

xneelostatus.co.za/notice/4815/

xneelo network unreachable – Samrand data centre

Notice ID:#4815
Start: 03 Dec 2019 12:15 SAST
Maintenance Window: 30min
Issue: Electrical Failure

12:20PM There appears to be a problem in Hetzner’s Samrand datacentre which has taken all servers, including their own website, offline.

18:05 We’re seeing very high loads on Lannister which has resulted in websites not being served. We are investigating.

18:28 The problem has been found and fixed.