Hurricane Sandy: Disaster Recovery Best Practices & Lessons Learned. Robert DiLossi Director Crisis Management SunGard. Agenda

Transcription

1 Hurricane Sandy: Disaster Recovery Best Practices & Lessons Learned Robert DiLossi Director Crisis Management SunGard Agenda SunGard Response and Readiness Hurricane Sandy Timeline Lessons from Sandy and Prior Events Statistics and Wrap up 1

2 We never thought this would happen to us It s estimated that 25% of businesses will not re open after a disaster The National Business Association estimates that only 38% of businesses have a viable emergency plan. No organization is exempt SunGard Operations/Crisis & Hurricane Sandy Timeline 10/22 NHC TD 18 becomes Hurricane Sandy 10/24 SunGard commenced 72 hr Facility readiness procedures Facility Operational testing UPS, Diesel Generator validation Top off fuel tanks Emergency Support Validation Communication & Emergency planning Internal NotiFind alerts activated Backfill staff into affected facilities Review third party response plans Activated laptop shipments Deployed Mobile Trucks to Copley, OH 2

3 SunGard Operations/Crisis & Hurricane Sandy Timeline 10/25 FEMA and Red Cross encourage Northeast to take steps to get prepared 10/26 FEMA briefs Government Agencies SunGard proactive communications commence Reroute Network Activity Additional Fuel Vendors Contracted Facility Access and Situation Awareness Portal announcements sent to our Customers SunGard Operations/Crisis & Hurricane Sandy Timeline 10/27 FEMA's National Response Coordination Center is activated 10/27 SunGard s Crisis Management Center (CMC) is activated 24x7 support Manned in Philadelphia and Carlstadt Upgrades and change control seize until after events passes Initiated hotel room blocks 3

4 SunGard Operations/Crisis & Hurricane Sandy Timeline 10/28 President signed emergency declarations for CT, MD, MA, NJ, NY & District of Columbia, more to follow 10/29 Sandy struck Atlantic City, NJ Full Moon and high tide increases surge to 14 feet Biggest winds are North of eye, hit NY Harbor 10/30 01:00 ET Local levee breached (Moonachie, Carlstadt, and Little Ferry N.J. ) Three towns under 5 feet of water within 30 minutes 10/30 02:00 ET Local Emergency Responder Command Post initiated in Carlstadt 760 Supported local emergency responders as a command post Used for triage and served as an evacuation center Provided medical attention, food and provided shelter for 100+ evacuees SunGard Hurricane Superstorm Sandy Statistics 342 Alerts (customers may need services) 7.5 million messages sent out via SunGard NotiFind 117 Disaster Declarations (customers do need services) Average of 13 days in disaster mode in our facility Average of 25 days in disaster mode for Mobile recovery 9 Facilities utilized Carlstadt Campus utilized 162,500 gallons of fuel Carlstadt Campus 529 incoming / 388 outgoing packages between Oct 29 and Nov 11 4

5 SunGard Hurricane Superstorm Sandy Statistics 1,500 people in our facilities Increase food, facility needs, security, parking Deployed one third of our staff 100% up time through the event CMC active for 17 days Space reallocated to build out additional workgroup recovery positions in Carlstadt and Philadelphia (keep track of team members deployment) SunGard Mobile Truck Response Deployed 5 Mobile Recovery Units to augment space in Carlstadt Traveled 11,000+ miles to deliver units Two units still at Manhattan customer site Five customers utilized the units, augmenting Carlstadt space Use of new VOIP technology Trucks deployed ahead of storm for quicker response 5

6 SunGard Response Customers Recovered Shared Seats Dedicated Seats Systems Recovery How many days are you prepared to remain in recovery mode Hotsite Mobile Mobile Data/Metro Center Average # of Days 6

7 Lessons Learned People and Infrastructure Team players missing or unavailable Work from home doesn t work without power or connectivity Employee / family first Not focused on recoveries Lack of personal necessities Raw sewage and toxic waste created health risk ATM s down, lack of funds No lodging / logistics Who s in charge? / lack of communication Power infrastructure older Lack of corporate presence at recovery site Lessons Learned People and Infrastructure Utilize alternate location employees, if available Have your own food supply on hand, MRE s (Traveling or shelter in place) Hand out Go Kits for personnel (Traveling or shelter in place) Develop/test your incident / crisis communication plans Document, document, document Socialize the test outcomes & developed plans Engage all business units earlier / ownership of plan Rally Corporate involvement / include while testing Validate vendors contacts quarterly (Tape, Hotel, Fuel, Transport Co., etc.) Engage local agencies (i.e. Fire, Police, Red Cross, EMS, Local Government, etc.) 7

8 Lessons Learned Travel & Communications Mass Transit, Roads/Highways, Bridges, Airports are closed / heavily damaged Limited fuel available (Gas & Diesel) Car rentals unavailable, vehicles destroyed Evacuations need to be enforced earlier Long delays Cell / land lines down or shutdown Press/media statements for public sectors Lessons Learned Travel & Communications Deploy team earlier / use alternate travel means Identify alternate routes Understand evacuation routes safety Initialize agreements with fuel providers Satellite phones Utilize local government agencies for support Consider alternate communication methods Establish Media Engagement Process 8

9 Lessons Learned The Plan Documentation outdated Required additional seats than specified on contract (customer growth) Lack of involvement during planning/testing Business unit requirements not accurate Regulations can not be forgotten Not planned for extended recovery Corporate participation minimal Vendor contacts outdated External agencies not engaged Lessons Learned The Plan Enforce change control / mandatory Understand your regulations, may prevent sharing an area Challenge your business units during testing for discovery Develop and test your incident and crisis communication plans Develop robust plans for longer stays Rally Corporate involvement / include while testing Validate vendors contacts quarterly Engage local agencies 9

10 Lessons Learned After the Disaster Insurance coverage unknown or not current Inventory out of sync Type or amount of equipment not know Temporary or permanent relocation Redirection of mail service Return home?? Lessons Learned After the Disaster Understand your policies and documented risks Utilize independent review of insurance Remember to upgrade or revise your coverage when moving or upgrading property Record equipment serial # s / photograph Inventory auditing, review coverage annually Use external security agency to monitor, if applicable Maintain running status of disaster events Develop a strategy for your return home 10

11 Even the best tested plans There will be challenges Focus on critical needs, prioritize You are senior management s advisors All one team, one goal Test your plans Evaluate tests & close gaps Introduce use of new tools where needed Include all business units and personnel Prepare for extended recovery Demand change control document, document, document Utilize alternate people for tests, give key players the day off Lessons Learned Customer Regulations cannot be forgotten Understand your regulations, may prevent sharing an area Challenge your business units during testing for discovery Required additional seats than specified on contract (customer growth) Not able to travel out of area Underestimated disk requirements, tape technology, etc. (growth and internal updates) Discovered during testing question your teams on requirements Communication issues continued 11

12 Lessons Learned Customer Value of Dedicated seats Shared seats have a higher risk profile in a regional event Location Regulations Readiness Specialized business units Dedicated customers were able to meet their workgroup needs Recent Disruption Boston Marathon Bombing April 15, 2013 (Patriots Day) Struck near the Boston Marathon finish line area on Boylston Street. Widespread panic for several days Closed subway stations/mass transit Roads blocked, evacuation of the area for several days Evacuation of several buildings FBI quarantine area 12

13 Boston ManHunt Recent Disruption On April 19, 2013 the entire Boston area is shut down, shelter in place issued. Widespread panic for several days Closed subway stations/mass transit Roads blocked, evacuations Evacuation of several buildings FBI quarantine area Recent Disruption Boston Back Bay Fire/Power Outage Struck on March 14, 2012, covering the heart of Boston with black smoke, causing: Widespread power outage for several days Closed subway stations/mass transit limited or down Roads blocked, leaving no access to area Evacuation of several buildings 13

14 Recent Disruption Tornado damage A large Tornado outbreak in May 2012 ravaged the midwest, causing damages in the millions. SunGard monitored the outbreak, handling several alerts and 1 disaster. St. Louis Recovery Center: Provided 30 Workforce Continuity seats, maintaining their business Customer sustained extensive building and roof damages Customer utilized Voice, Network and DIS services Occupied St. Louis recovery center for 105 days. Recap top actionable items Lack of Transportation/Lodging/Logistics Key Players Unavailable or Missing Communications down Health risks created by event Out of Date Contracts/Agreements Any Vendor Media Storage, Insurance, Fuel, Business Resumption Vendor, etc. Outdated Documentation/Scripts Technical Scripts, Employee Contact Lists, BCP, etc. Extended Recoveries not in plan Difficulty Accessing Tapes/Data/Documentation Lack of Content Business Continuity Plans (All company needs not fully identified, lack of testing, etc.) 14

15 Planning Steps for Business Continuity Obtain Executive Management s Commitment Establish a planning committee Perform a risk assessment Establish priorities Determine Recovery Strategies & Prioritize Perform Data Collection Document the plan (document, document, document) Develop testing criteria and procedures Test the Plan!!! Review the plan, close the gaps!!! The future of managing weather alerts / notifications Earthquake Event 15

16 Bob DiLossi Office 16