Official off-network notifications for Texo clients

Email 5 Offline

12:04PM

Email 5 is currently offline, due to a CPU temperature spike. Hetzner technicians are currently attending.

12:57 Hetzner techs have replaced the faulty hardware, and server is booting up now.

Tyrion offline

10:12AM The server tyrion.texo.co.za has just gone offline. We’re investigating.

10:20 We’ve found and fixed the problem, but websites might be a little slow for the next 10-20 minutes while we do some testing.

Daenerys offline

6 June 2016  5:17 PM Daenerys appears to be under DDOS attack which has taken the server offline. We’re working on mitigating the attack.

5:31 PM It appears to be another problem in the Hetzner datacentre, probably one of their switches again: https://hetzner.co.za/network-notices/

5:46 PM Hetzner has fixed their network routing problem.

Servers offline

03 June 2016 9:56 AM

Three servers are offline. It appears to be a problem at the datacentre, which is being investigated.

Affected servers:

Daenerys
Lannister
Email 4

10:03 AM Hetzner has confirmed a networking error in their Samrand DC which is being worked on.

10:05 AM Hetzner seems to have fixed the problem, all servers are now accessible.

Daenerys inaccessible

2 May 2016 5:38 PM Daenerys (197.189.230.226) is currently inaccessible due to what appears to be a network problem at Seacom / Hetzner. I’m still waiting for feedback from Hetzner.

5:45 PM Daenerys is accessible again, but connections are slow. Still waiting for feedback from Hetzner.

5:56 PM Inaccessible again. Still waiting for Hetzner to respond.

6:02 PM No feedback from Hetzner yet (email or phone) but they have updated their “Network Notices” page with the cause of the problem: a DDOS attack:

  • DDOS – Network connectivity – Johannesburg data centre (2016-05-02, ATTENDING)

    Start: 2016-05-02 5:53:48 SAST
    Resolved: TBA
    Status: attending
    Point of impact: Truserv and Co-Location customers
    Symptoms: Truserv and Co-location customers will have intermittent connectivity to their servers.
    Cause of problem: DDos
    Estimated time of repair: TBA
    Attending: Hetzner Engineers

Source: Hetzner Network Notices

6:20 PM Daenerys appears to be 1oo% accessible now.

Arya: Emergency maintenance

10 February 2016

8:00 Arya is back online. Performance will be degraded (sites will be slow) while the RAID rebuilds for the next hour or so.

7:15PM The server is being shut down now.

7:10PM Backup has completed

5:15 Starting a full server backup now

5PM Hetzner has just informed me that the RAID alarm for the server Arya is sounding.
This means that either a hard drive has failed (and requires replacing) or there is something wrong with the RAID configuration.
I have given them the go-ahead to take the server offline to diagnose and fix the problem.
I expect up to 30 minutes of downtime while they do this, for which I apologise.

Clients using CloudFlare for their website might be affected by an outage affecting two undersea cables and CloudFlare.

CloudFlare status: https://www.cloudflarestatus.com

More information: http://www.techcentral.co.za/seacom-wacs-problems-hit-sa-internet/62649/