Bumblebee offline

21 September 2:50PM

Server Bumblebee is offline. Technicians are investigating.

3:15PM The problem has been found, and technicians are working to fix the hardware fault.

3:26PM The problem has been fixed, and the server will be booted up shortly

3:37PM Still waiting for the server to be booted.

3:45 Server is booting up now. For the next 10-20 minutes or so, websites and emails will be slow while disk IO is high after the reboot.

Emergency maintenance: Baratheon

12:55PM The server Baratheon will be rebooted within the next 10 minutes to address a software error. We expect downtime to be no more than 30 minutes, unless the server requires a file check on startup, which could take up to 2 hours. Please accept our apologies for the inconvenience.

13:16PM The server has booted successfully, and services are returning to normal. Websites will be slow until the server load drops over the next 10-20 minutes.

Greyworm offline: DDOS

14:00 Greyworm is currently offline due to a DDOS attack. We’re working on mitigating it.

14:17 We’ve null-routed the target of the attack, and this seems to have stopped the DDOS. Unless another site on this server is targeted, the server should now run normally. Please accept our apologies for this downtime.

Baratheon: error

Baratheon is currently experiencing a problem, with some sites not loading.
We are investigating.

I have found a temporary fix, until cPanel can provide a fix or patch for the problem.

My “temporary” fix appears to be the only fix for this, until Comodo sorts their problem out. Response from cPanel support:


I can see you've already corrected this issue:

# grep -i stapl /usr/local/apache/conf/includes/pre_virtualhost_global.conf
SSLUseStapling off

Unfortunately, the 'ocsp.comodoca.com' server is currently facing issues, which would prevent SSL sites from being served. You've disabled SSL Stapling properly. Unfortunately, there isn't much else we can do until Comodo investigates why their OCSP server is not responding.

I certainly apologize for the inconvenience. Please let me know if you have anymore questions or issues.


Arya offline

14:32 The server Arya has gone offline – we’re busy troubleshooting now

14:43 The problem has been found and fixed. A problem with the firewall was denying all connections, Sincere apologies for the inconvenience.