Thursday, 11 November 2010

Fuel starved resilience

One of the measures I recommend to clients is to ensure they undertake regular power tests of their standby power systems for their trading floors and data centres. It should be a complete power test (say once every 6 months) where grid mains power is cut totally from the test building. The test should force the UPS system (if any) and the standby generator to take the load of supplying the business for at least an hour. Usually of course this test will have to be performed at a weekend so that main trading activities are not disrupted. The first time the test is run you’ll be pretty much able to find some important system that has not been fully connected to protected power. At the same time technicians should tour the building looking for any non-essential items connected to the critical protect power supply.

Part of the test should observe that the generator and UPS will successfully back off when the main grid power supply is restored and stable. During such tests the technicians should inspect air conditioning services to ensure that critical systems are thermally protected. It is no good having a room full of servers that will melt down in 20 minutes on standby power if there is no air conditioning to the room.

In one total power down test I witnessed the standby power source was a large generator mounted on the roof of a building. The first total power down test established that the generator would start and run for a few minutes before chugging to a halt. Subsequent investigation discovered the main fuel pump that delivers from the basement fuel tank to the fuel header tank on the roof was in fact connected to unprotected mains power. When the mains power failed there was no power to the essential pump that pumped fuel to the generator.

There's a sequel to this event here and another event, where we discovered some unpleasant neighbours, highlighting the need to test here.

No comments:

Post a Comment