While we do post outage reports to the ICIT outages page, they are very short and to the point. I 
thought it would be helpful to give more details about today's major outage and outline some of 
the lessions learned. 

There was a ConEd power surge this morning at 6:07 am. It knocked some of the power at Hunter 
and elsewhere around the city. Unfortunately, the power in the machine room went down and the 
back-up generator system did not kick in as it should. The component that failed was the transfer 
switch, which automatically senses a power outage and changes the power feed from the line to 
the generator. Our UPS did immediately take over the electrical supply, but it is only capable of 
covering the time it takes for the transfer switch to route the power from the generator. After 
about 10 minutes the UPS ran out of power, and since the generator was not activated, all of our 
systems crashed, including the network switches. Power came back on at 6:28, and systems 
started to reboot. The network switches rebooted and the network came back about 7:00 am. The 
Proofpoint server also rebooted immediately and started queueing email. Unfortunately, the 
servers for email and the web did not come back up for different reasons. The email server 
rebooted faster than the storage system and hung waiting for the storage system. A switch into 
the main web server blew out when the outage occured. Our staff was in very soon after the 
problems appeared, but it took some time to get all of the systems up and to trouble-shoot 
specific problems. Many systems needed attention. Everything was back up by about 11:00 am.  

No email was lost during this outage; email was queued by Proofpoint as soon as the Proofpoint 
server came up. Email that was sent during the time between the start of the power failure and the 
the time the Proofpoint server came up was bounced, but the period of time was short enough 
that it would have been resent by the originating mailserver.  Thanks go to our staff who got 
everything going. This was the first outage for some of our systems and they got to the root of the 
problems and solved them quickly.

Lessons learned: It is clear that the transfer switch needs to be repaired immediately. The backup 
power systems are tested monthly, but we need to work with Hunter Facilities Management to 
carefullly monitor what happens inbetween. Our staff learned about ways to deal with problems 
when our new systems crash, which should make future outages (which we hope do not happen) 
shorter. We will also look at the possibility of getting more battery power to keep the UPS systems 
working longer.