Server Load: Lizzy

12:10 We’re seeing very high loads on the server “Lizzy” which is causing connection errors and timeouts.

We’re working to find and fix the problem.

update 12:20 We’ve found the account causing the problem, and have suspended it while we investigate the root cause.

update 12:35 Lizzy is running normally again.

Network Issue

16 February 2012

We’re aware of a network issue affecting two servers:

Fudge
Mojito

which has made them inaccessible, and we’re working on it.

update 07:49 It’s been identified as a RAID issue, the DC’s engineers are on the phone to HP to work on a fix.
We have no ETA at the moment.

update 08:20 HP engineers are onsite now and we should have a fix shortly.

update 08:50 the HP engineers are still working on the problem…

update 09:14 The engineers say they’ve fixed the problem, so we’ll be booting these two servers now.

update 09:22 both servers are back up, but until the disk IO settles down after the reboot, please expect slow page loads. Everything should be back to normal within 15-20 minutes.

update 09:56 Fudge is back to normal, but Mojito is still experiencing very high load. We’re working to get it down.

update 10:15 Mojito is running normally now.

Emergency Maintenance

15 February 2012

19:55 We’re having a software issue on the server Wordsworth (causing the server to be unresponsive), which we’re attempting to track down.

update 20:05 The problem has been found and we’re working on a fix

update 20:12 we’ve applied the fix and will be rebooting Wordsworth now.

update 20:15 Wordsworth has booted up, but please expect slower performance (and intermittent gateway errors) for the next 15 -20 minutes while everything stabilises

update 20:30 we’re aware of the mysql (database) issues, and are working to fix

update 20:42 everything is working correctly now – sincere apologies for the downtime.

Servers offline

09 February 2012

10:38PM MWEB’s Cape Town datacentre is again experiencing issues, and has caused the following two servers to go offline again:

Chauler
Lizzy

Unfortunately we’re still at the planning stages of our migration away from MWEB, and no sites have been moved off these servers yet.

Apologies for the inconvenience (again) – we’ll keep you updated.

00:05 There is still huge packet loss to MWEB’s Cape Town DC

00:42 Everything appears to be returning to normal, but we’ll continue to monitor the situation.

Servers offline

MWEB’s Cape Town datacentre is offline. This has affected the following servers, which are currently unreachable:

Chauler
Lizzy

The MWEB website and MWEB MobiSite are also affected and are also offline.

We apologise for the inconvenience, and hope to get the servers back online asap.

Update 14:10
MWEB technical support says “please give it another hour” which is unacceptable. Once they’ve sorted out their problem, we’re going to start planning a migration of all accounts off MWEB servers. If you’d like to have your account moved as soon as possible, please send an email to [email protected] so that we can take care of it for you.

Update 14:34
Servers are back online.