150 likes | 244 Views
Critical Infrastructures: Working Together In A New World. Morgan Stanley 9/11 Lessons Learned. Steve Ruegnitz, Managing Director, Institutional Enterprise Applications, Security and Telecommunications . Agenda. Three Events Shaped Our Contingency Plan Pre-Crash Overhead View
E N D
Critical Infrastructures: Working Together In A New World Morgan Stanley 9/11 Lessons Learned Steve Ruegnitz, Managing Director, Institutional Enterprise Applications, Security and Telecommunications
Agenda • Three Events Shaped Our Contingency Plan • Pre-Crash Overhead View • Our People At The World Trade Center • WTC Tenants • Chronology Of Events • Locating Our People • Caring For Our Employees • Relocating Our People • Lessons Learned – Be Prepared • Lessons Learned -- Logistics • Lessons Learned -- The Human Factor • Going Forward • Small Enterprise Suggestions
Three Events Shaped Our Contingency Plan 1991 Persian Gulf War 1993 World Trade Center Bombing Y2K
Pre-Crash Overhead View WTC Complex TO CHAMBERS ST. EXIT Morgan Stanley Escape Route • 1 WTC • 2 WTC • Marriott Hotel • 4 WTC • 5 WTC • 6 WTC • 7 WTC • 90 West St. • 1 WFC • 2 WFC • 3 WFC • 4 WFC Citi Asset Management 7 AXP Merrill Lynch VESEY ST. WEST ST. 5 6 12 11 1 10 CHURCH ST. 4 2 3 Lehman Brothers 8 9
Our People At The World Trade Center • 3,528 total employees • 2,834 in IIG (82% of IIG non-field staff) • 694 in IM (17% of IM staff) • 31 employees treated for injuries • 6 missing
WTC Tenants 110 AON 1100 employees WINDOWS ON THE WORLD 70 employees 103 SANDLER O'NEILL & PARTNERS 177 employees CANTOR FITZGERALD 1000 employees FIDUCIARY TRUST COMPANY INTL. 645 employees MARSH & MCLENNAN 1700 employees 96 AMERICAN BUREAU OF SHIPPING 16 employees FRED ALGER MANAGEMENT 55 employees 93 WASHINGTON GROUP INTL. 190 employees CARR FUTURES 141 employees 87 KEEFE, BRUYETTE & WOODS 171 employees OHRENSTEIN & BROWN 91 employees NY STATE DEPT. OF TAXATION & FINANCE 222 employees NETWORK PLUS 46 employees HARRIS BEACH 113 employees 74 EURO BROKERS / MAXCOR FINANCIALGROUP 285 employees FUJI BANK 625 employees PORT AUTHORITY 2000 employees MORGAN STANLEY 2778 employees 59 BRIDGE INFORMATION SYSTEMS SIDLEY AUSTIN BROWN & WOOD 600 employees 56 ICAP 750 employees DAI-ICHI KANGYO BANK 300 employees FIREMAN'S FUND 190 employees 46 WACHOVIA / FIRST UNION 48 employees 43 LEHMAN BROTHERS 618 employees THACHER PROFFITT & WOOD 300 employees FRENKEL & CO. KEMPER INSURANCE 36 employees OPPENHEIMER FUNDS 598 employees EMPIRE BLUE CROSS& BLUE SHIELD 1914 employees SUN MICROSYSTEMS 300 employees 25 ICAP 750 employees SCOR-US 120 employees KEY BANK OF AMERICA 400 employees More than 100 people missing 10 11 to 100 people missing 10 or fewer people missing All employees accounted for Areas of impact 1 WTC 2 WTC Morgan Stanley offices
Chronology Of Events 9:15am Senior Management relocates to command facilities in 750 7th Ave Establish communications Begin process of locating our people Employee emergency number aired on broadcast media IIG & IM Senior Management at 1585 9:10am Lost contact to 2 WTC 7:30am 8:00am 8:30am 9:00am 9:30am 10:00am 10:30am 11:00am 11:30am 9:50am Last MS employee leaves 2 WTC 9:59am 2 WTC collapses 10:28am 1 WTC collapses 11:30am First survivor arrives at command facility (750 7th Ave) 8:48am 1 WTC hit by AA#11 8:49am 2 WTC Evacuation begins Contingency Plan implemented 9:20am Varick St. back-up systems activated Harborside starts purchasing key business equipment 9:03am 2 WTC hit by UA#175
Locating Our People Day Event Time Employee Update • Emergency 800# aired on broadcast media • Discover & Van Kampen Call Centers • First employee call received at call center • MS Online Call Center • 2,500 phone calls received • Out-bound calls to employees and family members begin 11:00 am 12:25 pm 1:25 pm 7:30 pm 11-Sep 2,528 unaccounted Continue to receive in-bound employee calls Out-bound calling campaign continues Employee ATM activities reviewed Home telephone numbers matched against Discover Card telephone numbers 12-Sep 300-400 unaccounted Door-to-door search for missing employees commences 13-Sep 30-40 unaccounted 6 unaccounted 14-Sep
Caring For Our Employees • Employee Assistance Program began work immediately • part of the long-term employee benefit plan • resources expanded to accommodate increased call volume • group sessions held in all major locations • Full-time counselors were on-site at nine of our major locations in the New York area from September 12th to October 5th; currently, supplied on an “as-needed” basis • Additional counselors dispatched to emergency call centers • Results: • 1,476 employees received phone counseling via EAP • 6,577 employees received group counseling via EAP • 307 employees received one-on-one counseling via EAP
Relocating Our People 1585/1221/1633/750 Hudson River NEWJERSEY Morristown Bronx GreatNeck Passaic River JerseyCity Varick St. Manhattan Newark Queens NewarkInt'l. Airport Brooklyn Bay Other Locations Harborside Elizabeth John F. Kennedy Int'l. Airport Brooklyn . Staten Island N New York, NY 0 3 Mi 0 3 Km
Lessons Learned – Be Prepared • We were prepared -- With an active, practiced disaster plan • Diversity of locations • We have a high capacity resilient infrastructure • HA everything • Capacity is your friend • For network, diverse carriers, ISP’s and routing • Telephone is the #1 priority service -- The public network down town was literally blown away, but our “on net” infrastructure kept our internal service intact
Lessons Learned -- Logistics • Every second counts -- We didn’t vacillate with respect to evacuation after WT1 was hit • Command and control – Within minutes a senior management command center was manned and our operations command centers in Mid-Town and London were in disaster mode • First order of business was welfare of our people • A central clearing house was set up to receive, track and prioritize recovery related requests (also valuable source of inventory tally for insurance reimbursement)
Lessons Learned -- The Human Factor • Across the board, our people were compassionate, heroic and professional • There is no substitute for leadership • We work with quality vendors and they proved their mettle • All firms in the Financial District put aside differences and pulled together to rescue our industry
Going Forward • Continue with our existing high capacity resilient infrastructure strategy that worked so well during the 9/11 disaster • Regional dispersion of staff and technology • More focus on BCP and, in particular, as a result of terrorism, including cyber terror • Implement a firm-wide incident response (based upon IT Sec’s STAR model) • Any low hanging loose ends in the security technical infrastructure are being nailed down (password quality, internal cross-group access, internal malicious code scanning, encryption, etc)
Small Enterprise Suggestions • Fractional bandwidth from global carriers (e.g., Equant) • Avaya IP phone • Heightened awareness of terrorism alerts and attack modes (FS/ISAC -- http://www.fsisac.com/) • Work out a BCP support mechanism in your respective industries – when it comes to large scale disasters or terrorism, your competitors are on your side