New York Outages

New York and New Jersey suffered a lot of damage from Hurricane Sandy. Even now a couple of weeks after the impact of the damage is still effecting  businesses. Apart from the obvious damage to timber frame buildings, the main damage effects appears to be:

• Loss of power
• Lack of fuel
• Results of flooding.

While I have enormous sympathy for those involved I can't but help think many of the after effects of this storm could have been mitigated or avoided if there had been some proper planning and infrastructure provision beforehand. Much of the problems seem to arise from trying to provide infrastructure on the cheap. Saving pennies has cost many more dollars. At some point in history an accountant may have shaved a few dollars from the budget for a company's infrastructure, but that same accountant will not stand up and proclaim "Mea Culpa" when the business is out of action for a couple of weeks as a consequence of those "savings". There are those who say this could be covered by business continuity insurance because it is only a 100 Year Storm. You'd trust the insurers to cover all costs and rebuild your company? Not!

Let's first look at fuel. In the case of Gasoline there were service stations which could not pump fuel because the pumps depended on mains electricity from the Utility Company. When the power went down the service stations became useless. There should be federal laws which require the provision of  weather protected standby generators at service stations to provide protected power to pump the fuel.

Then there was the problem of fuel shortages. The supply of gasoline in normal circumstances is a "just-in-time" facility where the service stations depend on regular delivery of fuel every few days by tanker trucks. If that delivery service is interrupted the service stations hold only a couple of days supply and quickly dry up. The situation is worsened by panic buying by consumers. When everyone decides to fill their gas tank to the top it places strains on the limited supplies causing them to run out more quickly. It leads to rationing such as this case. This whole situation could be mitigated if service stations were forced by law to hold a minimum of 15 days stock in their underground tanks. The 15 day buffer would make fuel supplies considerably more resilient to supply shock. The finance cost of holding 12 extra day's supply is not massive.

Your organisation may have standby power generators with a 24 hour fuel supply with a resupply contract with a local fuel depot. Have you actually checked whether the fuel supplier can honour those commitments in the event of a regional disaster such as Hurricane Sandy? Flooding might prevent despatch of the trucks. I recently saw a comment on a LinkedIn forum where an organisation fuel resupply company could not pump fuel into the delivery tankers because they also had lost mains power from the utility company. You should diversify suppliers and actually check their delivery arrangements in the event of a regional disaster, don't rely on contractual obligations. If you can extend your own fuel capacity to five days you'll give yourself more flexibility in the event of a disaster.

The regional loss of power is almost inevitable in the event of a disaster such as Hurricane Sandy. Power lines will collapse and the utility company will cut power to protect substations. You should ensure access to local power generation facilities. Perhaps your own standby generator or maybe a facility shared between local businesses. The standby facility should be regularly tested with total power down scenario's. You should walk through what would happen if the basement of your building becomes flooded. A co-location hosting centre discovered this to their cost. In some cases heroic efforts such as a human bucket chain may be necessary to keep the generators running.

Lloyds - the insurance people; have written a telling paper on energy security paper. It is worth visiting their site to download a copy to read.

Flooding can strike most businesses. You may be in a low lying coastal area, a river flood plain. You may have a major water main pipe running under the street outside of your front door with the risk of pipe burst. In storm conditions damaged roofing can allow rain water leakage. If there is risk you should look at: 

• the arrangements you have to prevent the water from reaching your key machinery; this might include high sill doorways, sealing of leakage points, tanking of the room floor and walls.
• the arrangements you have to quickly drain any flooding in your building. It might include pre-laid pipes, drainage sumps in equipment rooms and submersible drainage pumps. 
• leak detection equipment and alerts;
• overhead drip trays which can help prevent overhead water from leaking on to sensitive equipment. 

They plan for these type of flooding incidents in Australia. In New York a substantial about of phone/data cabling was destroyed by salt water flooding in a Verizon below ground level cable vault (90,000 cubic feet). The compressed air pumps, which help to keep the copper data cables dry, failed allowing salt water to saturate the insulation and destroy the data communications. Most of the old cable will have to be chopped away and replaced with optical fibre.

Blast mitigation

One development in physical security is the use of spray on polymer coatings to mitigate the effect of explosion blast on building walls. This can be a retrofit option is required. It helps to reduce penetration and spalling on the inside of the wall.

Here's a couple of suppliers Rhino Linings,  Dynasystems (Dynashield) and Line-X (Paxcon) and a video example. The polymer was originally developed as a coating to protect the the floor beds of pickup trucks from damage. It doesn't greatly strengthen the concrete/blocks/steel per se, but it does reduce the debris on the side of the wall opposite to the explosion. The reduces the risk to people within the building.

If you need some comparative research, there's a paper here.

If you are in the position to influence the construction of the building ,the blast resistance can be enhanced by adding polymer fibres and steel fibre (approx 7%) into the concrete during the mixing process. There is a great range of suppliers for these fibres. Such technology can also help protect concrete from the spalling effects of fire.

Passive fire protection

This week I called in at the Facilities Management exhibition in Olympia Conference Centre in London UK.  Mostly I find such exhibitions a bit boring, but there are often useful nuggets of information to be discovered. 

One nuggets of these was the Promat stand, There I was shown products which can provide four hour fire protection and also limited blast protection too. If this topic interests you it is well worth visiting their website to browse for 30 minutes or so. They have some particularly useful handbooks available on-line free of charge. If you need to plan building firewalls or "fire stopping", this is the site to visit.

Who are the software pirates?

I'm pretty sure that if a software manufacturer discovered a large organisation where software piracy is rampant there would soon be a court action and a claim for damages. For example if one in five of the users had an illegal copy of their software running on their desktop PCs the supplier would demand payment and compensation on top.

However several of the USA based suppliers are guilty of that same piracy when it comes to the differential between the USA price (USD) and the UK price (GBP) charged to customers. Take for example the latest release of the Adobe Acrobat product version XI. Its USA advertised price is $299, yet the UK price is the equivalent of $375 after the effects of sales tax (VAT) and exchange differences are taken into account.  It makes the same product bought in the UK (by download) 25% more expensive. There is no justification for this corporate greed by Adobe, it is in effect piracy by the suppliers.

I don't break the law by using pirated copies of Adobe software. I simply buy an alternative called Nitro PDF. It is just as good as Adobe Acrobat, yet it is one third of the price and they don't play games with the price in different currencies. Hence Adobe loses out. If they'd shown some integrity I might have done business with them. It could be worth checking what your own business policy is in regard to USA/UK pricing differential.

Cheap Energy?

An article in Wired points to progress in cold fusion. Maybe it works or maybe it is like write only memory? It will be interesting to see. Here's another information source.

Dubious business practices.

Over the years I've developed the view that common courtesy is an essential part of client relationships. Part of the process is to show respect to clients/prospects. Wasting the time of a client/prospect shows a lack of respect.  When I see those signs of discourtesy I tend to assign the supplier to the blacklist/scam pile and refuse to do business with them in the future.

Take for example Clifton Asset Management Plc based in Bristol UK. They have the practice of periodically sending me prospecting letters offering Overdraft and Loan facilities for my business. As it happens we do not run our business using the debt model and we have no interest in such offers. What really annoys me is that the signed letters from Adam Tavener Chairman of Clifton Asset Management Plc are marked "Private and Confidential" on the outside of the envelope. The contents are clearly neither private nor confidential. Mr Tavener has no prior business relationship with me or my company. The wrongful labelling of the envelope is a deceitful practice designed to help the letter bypass any secretarial filter.

Such deceit does not bode well for any future business relations with this company so they are now blacklisted. 

However their letters continue to arrive. There is a note at the foot of the letters saying please contact Clifton "should you wish to unsubscribe"; yet another clear indicator of unsolicited mail as we didn't subscribe in the first place.  We've contacted them on previous occasions to ask them to stop sending such letters to our organisation to no avail.  

Earlier this week I phoned Clifton Asset Management to speak with Mr Adam Tavener as invited to do so by his letter. The call was answered by a telephone clerk. I made it clear I  wanted to discuss the dubious marketing practices with Mr Tavener, but I was told he's a busy man and would not be available.  I left a message for him to call me. I'm not surprised to have not been called by Mr Tavener. Ignoring my complaint is a further sign of disrespect.

Since then I've undertaken some research into Clifton Asset Management,  the associated companies, the management team and their history. They are certainly not the type of company that I'd consider using in any shape or form in the future.  It just goes to show that a lack of courtesy is a reliable business indicator. I have no evidence that their services are a scam, indeed they are registered and regulated by the Financial Services Authority, but it would be a cold day in Hell before I'd consider using them.

It seems like I'm not alone in these thoughts, and here.

Edit: 14/11/2013 We just can't seem to get away from these people. We've moved to a new location and are receiving mail for the previous occupant including letters marked "private and confidential". The return address on those letters is  TMS PO Box 10044, Hucknall NG15 5DQ which appears to trace back to Clifton Asset Management. We've marked them Return to Sender, but have no great hopes about the letters ceasing.

Edit: 31/01/2014 It seems like the Clifton Asset Management is still up to its old tricks. Despite requests from us to cease and desist sending us their junk mail we had another letter from them dated 27th January 2014. Marked "Private and Confidential" which it is not, the letter puts forward schemes to raid our pension funds by borrowing against them and along the way paying large fees to Clifton Asset Management. It seems a rather parasitic use of funds which should be held in trust for the fund members. This time however their website has changed to pensionledfunding.com. There appears to be a new name in the frame of Neil Greenaway as the Managing Director. I wonder what it will take for Clifton Asset Management to leave us in peace? 

Planning for power (2)

In the previous blog entry on planning for power in a data centre I considered the need for a power budget and the need for a good change control process for variation in the power budget.

Most new data centres are initially only partially filled with equipment with space, facilities and resources left free for expansion. The power budget for the equipment expansion areas needs to be estimated in the early planning stages of the main construction project. It might be possible to postpone both capital and operational expenditure by fitting only the power infrastructure required at the moment, but in this case engineering space needs to designated and protected for future expansion. Additionally consideration needs to be given to the route through which electrical and mechanical plant will be moved during upgrades and maintenance.

When the power load has been established it will be possible complete the planning for air conditioning requirement.  The design of the air conditioning system will depend on what facilities exist for chilled water and what additional equipment can be installed within the building.

Consideration will needed on how long standby power will need to endure during a mains outage. This decision is not simply how long it takes for a standby generator takes to start and provide power. Consideration needs to be given as to how long it takes to gracefully shut down a data centre should the standby generators fail to start. An uncontrolled power crash can have devastating effects on ICT processing and the dependent businesses. The endurance requirements of the business will dictate how much energy needs to be stored in batteries. Consideration also needs to be given on how cooling will be provided if the generators fail to start. The required battery capacity affects the design of the battery room, but I'll cover that in a later blog article.

A joint decision will have to be taken, with the electrical engineer, on the power transfer switching arrangements will handover power to the standby arrangements and consequently back when Utility mains power is restored. For a data centre this switching is usually automatic with manual over-ride capability.

When the power requirement has been agreed, the engineers should establish the space requirements and weight of the air conditioning plant, machinery, transformers, switching gear, UPSs, generators, fuel tanks, batteries, ducting, piping and cabling  required. Agreement should be sought on where all of this equipment will be housed and mounted. The electrical equipment and air conditioning plant will impose an additional weight load on the building structure. The positioning of this weight/vibration load should be reviewed by a structural engineer to ensure the load is within the capacity of the building. Machinery such as generators, compressors and air handling units will create vibration in operation. This vibration might affect the building. The plans for damping and handling such vibrations should be reviewed by the building structural engineer.

Throughout the planning process it is highly likely that there will be obstacles to providing the equipment necessary to provide the power and cooling. For example there may not be enough space or the equipment is too heavy. In the worst case these obstructions may limit the amount of technology which could be deployed within a building. In any event there are likely to be many cycles of the planning process before an agreed solution can be achieved. Depending on the complexity of the data centre this process may take a few months. The process may involve agreement from the owners of the building and also regulatory planning authorities. If the original assumptions on power requirements change after the design has been settled it can be an expensive problem to fix.

Part of the planning process is to provide a solution which is affordable to the business client. It is in effect a process of risk assessment with the client where the proposed capital and operational costs are weighed against the risks and what the business can afford.