Silly Globix! (by alaric)
One of the companies I do work for hosts their equipment at Globix's Prospect House datacentre in New Oxford Street, London.
And it's become increasingly apparent over the past couple of years that Globix aren't doing a great job.
It all began to crumble on the day everything went down. Now, Globix pride themselves on the lovely backup power system. They have a giant generator, roomfulls of batteries, and all the usual stuff. They explained to us how they're better than many such places because they test their generator under a simulated load with a giant resistor on the roof.
However, still, the power to the entire floor went down for many hours. Apparently, the external power had gone down. The backup batteries had flawlessly switched in while the generator started, then the generator took over and started recharging the batteries. However, it turns out that the power switching system that connects generator, batteries, incoming power, and the load had a special cooling system, which was... connected to the external power feed. And so it wasn't running. The switch unit overheated, and shut itself down, killing the machine floor.
Whoever installed that cooling system was, therefore, somewhat incompetent.
But it gets worse. Power was restored to the floor. I went in to talk to the machines that didn't come back up cleanly, and couldn't help but notice that it was:
- Hot
- Dark
It became apparent that they were running on a reduced load, to keep their switch cool enough. To reduce the load, they'd killed the AIR CONDITIONING SYSTEM. And the lights.
In order to reduce their on-paper downtime, they'd brought the computers back up without aircon, for about a day.
In the year or so since that happened, we've had a lot of hardware failures - which we'd not been plagued with before. It's pretty likely that the roasting the machines had didn't do them much good.
Then about six months ago, we were going to install a new machine. We followed the usual process; on the day, we emailled in to say we were bringing in a machine. However, unlike the zillion past times we'd brought in machines, we got an email back saying we had to wait until there was power available.
Eh?
It turned out that they realised they were running short of power in the data centre. Now, they'd not told anyone this, perhaps because it's embarrassing. Instead, they waited until we tried to put a machine in, then revealed that they'd introduced new restrictions. Since we'd been planning this machine insertion for several weeks, had we known, we'd have gotten into the "power queue" sooner. Instead, we had to wait weeks until somebody else removed a machine, freeing up amps for ours.
We have space in our rack. We have spare IP addresses. We have spare sockets on the switch. Yet we are waiting on other customers to remove equipment before we can put our own in.
But it gets worse.
We receive an email telling us that our machines are in the rack the wrong way round. Apparently they want to line machines up so that alternate aisles are used to take air from, so the other aisles receive all the hot air for taking away. And they've just noticed that our machines don't fit their plan.
Now, the machines were initially installed long before my time at the company. Since I took over, I've just been putting machines in the same way as the original ones. And, in fact, putting them in the other way would be very hard, since Globix's power strips are down the mounting rails on the 'rear' of the rack. To put machines in facing the other way would involve rerouting the power supply to the rack.
Oh, they say, perhaps our engineers put the power strips down the wrong side of the rack when they set it up for you. Sorry. Please spend a day powering down all your machines, taking them out, moving the power strips, and putting them all back in. As soon as possible.
I understand your point of view. Globix had obviously placed the power strips in the wrong side of the cab leading your technicians to understandably put the kit the wrong way round. However in order to ensure that the data-centre is practicing best practice and ensure that all customers are having their cabs cooled on the correct side it is imperative that all customers are facing the right way round. We will offer you hands so all we really need is for someone to over see it but I'm afraid we have no other option other than to turn the kit around.
Grrrrr!!!!
Tue 25th Apr 2006 2:01 pm 
Subscribe
By Becca, Tue 25th Apr 2006 @ 6:03 pm
The older i get the more i realise how incompetant and inept the vast majority of ppl around me actually are. Its incredibly frustrating. Through various incidents i have learned that if you want something done (correctly) you have to do it yourself! The problem is that you can´t do everything yourself....such is life. Try not to get too stressed....i´m out in mallorca atm teaching undergrad astronomers...very relaxed environment...incredibly frustrating job though!