6 June 2016 5:17 PM Daenerys appears to be under DDOS attack which has taken the server offline. We’re working on mitigating the attack.
5:31 PM It appears to be another problem in the Hetzner datacentre, probably one of their switches again: https://hetzner.co.za/network-notices/
5:46 PM Hetzner has fixed their network routing problem.
03 June 2016 9:56 AM
Three servers are offline. It appears to be a problem at the datacentre, which is being investigated.
10:03 AM Hetzner has confirmed a networking error in their Samrand DC which is being worked on.
10:05 AM Hetzner seems to have fixed the problem, all servers are now accessible.
2 May 2016 5:38 PM Daenerys (18.104.22.168) is currently inaccessible due to what appears to be a network problem at Seacom / Hetzner. I’m still waiting for feedback from Hetzner.
5:45 PM Daenerys is accessible again, but connections are slow. Still waiting for feedback from Hetzner.
5:56 PM Inaccessible again. Still waiting for Hetzner to respond.
6:02 PM No feedback from Hetzner yet (email or phone) but they have updated their “Network Notices” page with the cause of the problem: a DDOS attack:
Source: Hetzner Network Notices
6:20 PM Daenerys appears to be 1oo% accessible now.
10 February 2016
8:00 Arya is back online. Performance will be degraded (sites will be slow) while the RAID rebuilds for the next hour or so.
7:15PM The server is being shut down now.
7:10PM Backup has completed
5:15 Starting a full server backup now
5PM Hetzner has just informed me that the RAID alarm for the server Arya is sounding.
This means that either a hard drive has failed (and requires replacing) or there is something wrong with the RAID configuration.
I have given them the go-ahead to take the server offline to diagnose and fix the problem.
I expect up to 30 minutes of downtime while they do this, for which I apologise.
Clients using CloudFlare for their website might be affected by an outage affecting two undersea cables and CloudFlare.
CloudFlare status: https://www.cloudflarestatus.com
More information: http://www.techcentral.co.za/seacom-wacs-problems-hit-sa-internet/62649/
Tue 11 August 4:40PM Hetzner has informed us that the RAID alarm is currently sounding, and that KEATS needs to be switched off in order to diagnose and fix the problem.
5:28PM Keats is being shutdown now
6:02PM Response from Hetzner:
This mail serves to confirm that the maintenance on your server tex001_truservcomm_jhb1_009, was completed successfully.
SDA was swapped, and RAID is currently rebuilding.
6:15PM The RAID rebuild is at 24%
6:27PM RAID rebuild is at 30%
6:53PM RAID rebuild is at 50%
8:07PM RAID rebuild is at 90%
10:36PM Hetzner says that the server is “fixed”. Unfortunately, it won’t boot. I am therefore going to reinstall the server, and then restore all hosting accounts from backup. Please accept my sincere apologies for this. I will work through the night and tomorrow to get all sites up as soon as possible.
2AM Wed 12 August OS has been rebuilt, cPanel and Cloudlinux have been installed. Restoration of hosting accounts is starting now.
4:30AM All accounts have been restored from backup.
20 July 2015 18:00
There appears to be a network problem in Hetzner’s JHB DC, which is affecting the servers Cersei, Yeats and Email 4
18:11 They seem to have fixed the problem, and all servers are back online.