FUTURES INDUSTRY BUSINESS CONTINUITY AND DISASTER RECOVERY

Transcription

1 FUTURES INDUSTRY BUSINESS CONTINUITY AND DISASTER RECOVERY OCTOBER 9, 2004 DR TEST RESULTS October 15, 2004 Compiled and Developed by The

2 BACKGROUND The FIA Information Technology Division established a Business Continuity and Disaster Recovery Planning (BC/DR) Committee in The Committee was chartered with identifying and developing industry-wide testing and preparedness among FCMs, clearing firms, exchanges, clearinghouses and key service providers. The Committee fostered the creation of a Business Continuity and Disaster Recovery Planning Survey in Q4, Detailed preparations for industry-wide testing began in Q2, 2004 and BC/DR briefings were held to discuss and agree on a date, goals, objectives, etc.

3 BACKGROUND (Cont d) Starting in April, a number of futures Industry BC/DR symposiums were held in New York and Chicago. A Project Manager was appointed to be the overall test coordinator. A BC/DR Working Committee was established, including representatives from: Clearinghouses Exchanges FCMs, clearing firms and non-clearing firms Key service providers Independent software vendors

4 BACKGROUND (Cont d) The Committee established a common date for testing and determined the scope of the test: Test firm back-up to exchange back-up sites (DR-DR). Verify connectivity. Test round-trip communications capabilities. Regular Committee conference calls were held between May and October (bi-weekly and weekly).

5 BACKGROUND (Cont d) The test was held Saturday, October 9 from 8:00 a.m. until 4:30 p.m. EDT Participation in the test was not mandatory. Each exchange developed its own test script and recruited member firms. To encourage participation, the FIA agreed not to release any firm names. Post-test interviews with firms and survey forms were used to assess feedback and determine results.

6 EXECUTIVE SUMMARY First industry-wide disaster recovery test in the U.S. financial services sector (due largely to the good working relationship between exchanges and firms). The major U.S. futures exchanges and clearinghouses and 90 FCMs, clearing and non-clearing firms participated in the test. Participant firms represent a significant critical mass of futures order flow and liquidity. Exchanges and clearinghouses demonstrated that their systems, processes and procedures simultaneously worked very well. Under real life situations, most problems encountered could probably be resolved within hours or by the next business day.

7 EXECUTIVE SUMMARY (CONT D) Feedback from member firms, exchanges and clearinghouses indicates this was a valuable and worthwhile exercise and that the industry should do more common testing in the future. Firms indicated that the test helped them: Identify some internal single points of failure Tighten up and document their business continuity procedures Better understand the need for cross-training Test connectivity with exchanges DR sites Pre-test connectivity testing was of value and should be integral for future test exercises.

8 OVERALL TEST RESULTS 90 firms* participated in the test. All major U.S. futures exchanges and clearinghouses participated in the test: CBOE Futures Exchange CBOT CME/CME Clearing Clearing Corp. Eurex US NYBOT/NYCC NYMEX/COMEX/NYMEX Clearing Options Clearing Corp. * Some firms are common members of multiple exchanges and clearing houses.

9 OVERALL TEST RESULTS (CONT D) Test participants included both clearing firms and non-clearing firms. Approximately 77% of all futures clearing firms participated representing approximately 96% of the volume cleared on U.S. futures exchanges. 88% tested from their Disaster Recovery sites and/or via back up communications facilities. The remainder tested from their primary sites.

10 CBOE FUTURES EXCHANGE Tested via member firms back up connectivity to production CBOEdirect trading platform. Scripted trade entry for VIX futures contracts. Transmitted trades to the Clearing Corp s. back- up systems. Clearing Corp. produced end-of-day files and transmitted them to the Options Clearing Corp. back-up systems.

11 CHICAGO BOARD OF TRADE Tested via member firms back up connectivity to production ecbot trading platform. Confirmed connectivity via successful system login message no order entry. Generated a subset of October 8 trades to member firms from CME Clearing: 10-year Treasury, Corn and Dow Futures contracts Transmitted trade confirmations from CME Clearing. Accepted EFP, change records, PCS and large trader records.

12 CHICAGO MERCANTILE EXCHANGE Tested member firms back-up connectivity to production GLOBEX trading platform via CME remote back-up site. Scripted order entry in eurodollars, euro FX, Japanese Yen, CPI, Goldman Sachs Index, hogs, NASDAQ, NASDAQ emini, S&P and S&P emini futures contracts. Received ex-pit, block trade information via CME remote site portal URL. Generated trade reporting and SPAN files from 10/7 trade date. Received PCS and large trader information from member firms. Transmitted trade registry data and SPAN files via FTP.

13 THE CLEARING CORP. Tested member firms back-up connectivity to backup clearing system. Confirmed system connectivity via successful system login message no scripted trade entry. Transmitted previous business day clearing reports to member firms: Match off files Report files SPAN files PCS, ex-pit and large trader information was not accepted.

14 EUREX US Tested member firms back up connectivity to production XCEED trading platform. Connectivity was confirmed via successful system log in message no scripted order entry. Transmitted the previous business day s clearing reports from the Clearing Corp: Match off files Report files SPAN files

15 NEW YORK BOARD OF TRADE Tested member firms back-up connectivity to NYBOT back-up trading floor and back up electronic order routing (EOR) systems. Scripted order entry for cotton and sugar futures contracts. Order entry via back-up phone lines, NYBOT EOR and member firms order routing systems. Forwarded members phones to NYBOT back-up trading floor facility: The trading floor was supported by NYBOT Floor Operations staff. Trade entry via TIPS system for test trades, ex-pit, large trader and PCS information. NYCC produced end-of-day files: Match off files, report files and SPAN files.

16 NEW YORK MERCANTILE EXCHANGE Tested member firms back-up connectivity to eaccess back-up trading platform. Order entry via scripted trade entry in crude oil, natural gas, heating oil, gasoline, silver and gold futures contracts. Reported trade fill information to member firms. Transmitted end-of-day clearing reports via NYMEX Clearing: Ex-pit, PCS, large trader information 591 Member Trade Register report

17 PROBLEMS ENCOUNTERED The type of problems encountered included: Communications connectivity, incorrect IP addresses FTP, RJE communications Incorrect contact phone numbers Invalid or expired passwords ISV application software/support Misinterpretations of test protocol/script Staffing/lack of technical or domain knowledge Trade management applications connectivity Most were resolved quickly, although some caused an unexpected delay to test progress.

18 PROBLEMS ENCOUNTERED (CONT D) Several firms had connectivity problems with accessing a clearinghouse post-trade system: One could not connect at all One did not have a back-up server replicated or configured for the test One could not make the changes during the test

19 PROBLEMS ENCOUNTERED (CONT D) Some firms experienced procedural problems with invalid or expired passwords, looking at the wrong system screens, etc. A clearinghouse had a network IP configuration addressing problem that prevented traffic flow for the first hour of the test: It was subsequently fixed and all firms were then able to communicate. One exchange could not transmit trade data to its clearinghouse because a trade management process did not propagate properly: They could not transmit trade data to their members. Several exchanges reported support was stretched because a number of firms started to access FTP and/or RJE processing before they were ready.

20 LESSONS LEARNED Feedback from member firms, exchanges and clearinghouses indicates this was a valuable and worthwhile exercise and that the industry should do more common testing in the future. Most problems that were encountered were rectified quickly, although some caused an unexpected delay to the restoration of full processing capabilities. Some firms may not have completed all the test script steps and some did not call the exchange or clearinghouse to acknowledge the end of the test. Firms, exchanges and clearinghouses need to better cross-train staff and have more resources onsite to support future testing efforts.

21 LESSONS LEARNED (CONT D) Firms indicated that the test helped them: Identify some internal single points of failure Tighten up and document their business continuity and system fail over procedures Better understand the need for cross-training Test connectivity with exchanges DR sites Pre-test connectivity testing should be a pre-requisite for future tests. The exchanges and clearinghouses internal support processes and procedures worked well.

22 LESSONS LEARNED (CONT D) Firms must be aware of any environmental impact on their network firewalls caused by test requirements: Have proper network staffing support actively engaged during the test. Be aware of any firewall impact and make changes accordingly to accommodate testing. Some firms indicated that it would be helpful to have a test book that included information and scripts for each exchange and that it be frozen about one week before the actual test. Exchanges and clearinghouses should be staffed to potentially handle a high volume of troubleshooting calls from member firms.

23 LESSONS LEARNED (CONT D) A large number of firms apparently did not pre-register and showed up for the test at several exchanges: They were set up and most tested successfully, however, this placed an additional demand on exchanges and clearinghouses support staffs. There is a need to better coordinate testing activities: Those firms that are common members of two or more exchanges indicated that the time frames for completing various test activities strained their operations and support capabilities More time should be allocated for this during future tests. Corroborate all test contacts prior to the tests.

24 SUGGESTED NEXT STEPS Survey firms and exchanges for input to future testing Establish a steering committee to: Define the scope of 2005 industry test initiatives Establish the goals and objectives of 2005 test initiatives Broadly market future test initiatives to firms IT, operations, trading and senior management, to ensure a wider dissemination of test information within all firms.