Sunday 14 November 2010

Too much fuel for trading floor

There's a sequel to the earlier blog entry where the generator failed due a poorly wired fuel supply.

Clearly some engineering works were necessary to ensure the fuel pump supplying the generator still had power during a mains outage. Some additional power cabling had to be run down to the basement to feed the fuel tank pump. At the same time the engineering company undertook some routine maintenance on the roof mounted generator. It was decided to run some tests to ensure the generator would start properly and that fuel would be properly supplied from the basement even if grid mains power was removed. Unfortunately it was decided to run the generator test during the trading day, presumably to save costs on engineering cover rather than paying overtime rates.

The traders were warned they might hear some slight noise from the generator, but there would be no interruption in power to the trading floor or other services. The generator test was scheduled for a quiet period in the trading day. The generator started flawlessly and ran well for at least 20 minutes. Stop and restart was tested. All appeared to be going well. I hadn't mentioned earlier, but the trading floor was on the top floor of the building and the generator was mounted on the roof above the trading floor.

People first started to notice problems when diesel fuel started to drip then pour through the ceiling of the trading floor on to the trading desks and traders below. Some urgent investigations discovered that diesel fuel was pooling in the containment bund/wall surrounding the roof mounted generator. During the maintenance work someone had left the end cap of the fuel pipe for the generator, unfortunately enough fuel was reaching the generator for it to continue to run. Small cracks in the roof below the generator was allowing fuel to leak through on to the trading floor below. The fire brigade were called. They had great difficulty in evacuating the traders who did not see why they should worry about a "little bit" of diesel. We had to close that part of the trading floor, but some urgent support staff intervention we were back in business within two hours.

The moral of this event? Don't run tests in a live environment unless it is really unavoidable. Potential saving on man hour costs can soon get wiped out by the consequences of an outage.


Thursday 11 November 2010

Millbank House - Student Riots

From watching the television news reports of the Student Fees protest it was painfully obvious that the Millbank building had woefully poor security design. A small group of extremists and misguided broke away from the main body of the peaceful protest to attack the building containing the Conservative Party head quarters. I'll not comment on the Police preparedness, there are other people doing that already.
What particularly drew my attention was the cheap quality of glass walls at the ground floor reception area. Some of the glass had been treated with blast protection film, but the large windows, accessible at street level, were not made from laminated glass. It looks as though standard float plate glass had been used on the static windows and tempered glass on the doors.
Considering the risk created by the occupants (Tory Party) of the building the glass should have been replaced by glass strengthened by lamination to provide better security protection.
I also note the windows higher in the building appear to be plain sheet glass judging by the breakage patterns, not even any anti-blast film.
The layout of the reception area needs to be redesigned to prevent rush through attacks.

Fuel starved resilience

One of the measures I recommend to clients is to ensure they undertake regular power tests of their standby power systems for their trading floors and data centres. It should be a complete power test (say once every 6 months) where grid mains power is cut totally from the test building. The test should force the UPS system (if any) and the standby generator to take the load of supplying the business for at least an hour. Usually of course this test will have to be performed at a weekend so that main trading activities are not disrupted. The first time the test is run you’ll be pretty much able to find some important system that has not been fully connected to protected power. At the same time technicians should tour the building looking for any non-essential items connected to the critical protect power supply.

Part of the test should observe that the generator and UPS will successfully back off when the main grid power supply is restored and stable. During such tests the technicians should inspect air conditioning services to ensure that critical systems are thermally protected. It is no good having a room full of servers that will melt down in 20 minutes on standby power if there is no air conditioning to the room.

In one total power down test I witnessed the standby power source was a large generator mounted on the roof of a building. The first total power down test established that the generator would start and run for a few minutes before chugging to a halt. Subsequent investigation discovered the main fuel pump that delivers from the basement fuel tank to the fuel header tank on the roof was in fact connected to unprotected mains power. When the mains power failed there was no power to the essential pump that pumped fuel to the generator.

There's a sequel to this event here and another event, where we discovered some unpleasant neighbours, highlighting the need to test here.

Tullett Prebon & BGC

I see Tullett Prebon are in a spat with BGC about proprietary pricing data on Treasury Swaps. I used to be involved in the Tullett IT before the period mentioned in the case. Typically these pricing tools draw their data from many sources. Tullett were particularly careful in those days about not infringing IP of data feeds, I doubt very much if they lost any of those skills.
As to the damage being worth hundreds of millions of dollars? - I think the lawyers are having a laugh.

Monday 8 November 2010

Ahh! The confidence of East Coast USA techs

There are times when the confidence of IT Support technicians hailing from the East Coast of USA can raise a smile. (I mean Noo Joisey of course).
We had regional IT teams supporting a global trading activity. The New York team had invented the PC based application (real time price calculation) that would double business [yawn]. Of course it needed special processor and graphics card. The time had arrived when this application would be [imposed] supplied to their Brit colleagues.
"Do you want us to supply the servers for this?"
"No, we can get them more cheaply in the US and we know how to configure them. You can help by unpacking the boxes when we send them over."
"Sure thing Bob."
After several false starts eventually our USA colleagues succeed in delivering the PC/Servers to the London offices. We duly unpack them and store in a secure room. Soon the american geniuses fly over [Club Class] to London to demonstrate their wonder product. They find their treasured servers and plug the first and then the second into the mains power with suitable network connection.
You'd think the loud click and puff of smoke from the first server box when first switched on might just give a hint that there is a slight technical problem here?! Nope, they boldly switch on the second box. Click, and a puff of foul smell smoke from the second box.
"Ah Bob, you did remember that the mains voltage here is 240V Ac and not 120V didn't you...? You probably need to make sure you changed the voltage setting on the Server PSU."

Friday 5 November 2010

How do you calculate the PUE for that...?

PUE is a ratio of power used effectively against power delivered to an organisation such as a data centre. In concept it is simple, but the process of calculating a real figure is fraught with complexities, assumptions and varying interpretations.

Calculating a PUE for a trading floor is not always easy. Not a lot of the equipment has individually metred power, so any figure calculated tends to be quite approximate. It does however bring back memories of a company where I used to work. One branch of the company was located in two floors of a multi-tenanted building. Our company had grown by a process of acquisition of other trading firms. A consequence of that was the the IT infrastructure and the supporting building services infrastructure was quite fragmented.

The business decided it could improve the level of trading by building and implementing a new trading floor within the existing building. Plans were duly made and agreed with the landlord. It was not until later in the project that it was realised the building electric power supply would need to be upgraded to meet the power demands of the new trading floor and its associated equipment. The directors of the business were not best pleased with the additional expense on the project but agreed the (new) premises manager could contact the utility company (via the landlord) to ask for additional power cables to be installed.

The shocking news (unintentional pun) came back from the power company. "Sorry guys but we cannot deliver that level of power to your building unless we build a high voltage sub-station." The price quoted would have shocked NASA. Eventually after much discussion it was decided our company would install its own high voltage sub-station in the building. Our company took on the role of supplying power to the other tenants from our sub-station. We had to recruit an additional facilities manager who was trained and certified in the operation of high voltage equipment. Needless to say the project went over budget! I was not the project manager - phew!

Calculating PUE on the basis of power supplied to our company would be interesting to say the least.

Creating a trading floor

I've just returned from Hong Kong where I was speaking at a conference. My topic was entitled "Creating a Trading Floor" which is conveniently the same as my book about trading floor construction. I was speaking for only 40 minutes and found great difficulty in levering enough detail into the session. The book is a 500 page reference book and that only really skims over the sdurface of some of the complex technology and challenges facing people working on such a project. The session was very well attended, which shows that either the other parallel sessions were of low interest or there is a degree of interest in trading floors in Asia.
I was a bit constrained in not knowing the skill levels of the audience. I'd have loved to have gone deeper into some of the more complex issues and also some of the current issues such as sustainability.
All in all an interesting visit, but the 10 hour flight is a bit of a killer.

Monday 1 November 2010

Standby power testing

I visited the bank's branch office on the edge of Istanbul. As part of my brief I inspected the building and IT infrastructure supporting the local trading floor. I was being shown around by the local IT Director. It came to the point where I was investigating the standby power facilities.
"Show me the standby power facilities please?"
"Oh you mean the generator? It is out the back in the shed on its own."
Three minutes later...
"This looks to be in good order. How frequently do you test the generator? Once a month?"
"Oh no Sir, we don't ever do test runs."
"Why not? How are you confident the generator will start if there is an outage?"
"It is not a problem. The mains power here fails at least once a month. So we know the generator works just fine."
Stunned silence on my part ...