U S E M ETRICS DASHBOA R D S E N T E R P R I SE SEC URITY RISKS. Session 5302

Transcription

1 1 U S E M ETRICS DASHBOA R D S TO M ANAGE E N T E R P R I SE SEC URITY RISKS Session 5302

2 M E T R I C S D A S H B O A R D S 2 Moderator Cheryl Stone Director, Corporate Security & Safety RAND Corporation Presenter Peter Ohlhausen President Ohlhausen Research, Inc. Presenter Daniel McGarvey Senior Principal Business Process Analyst Alion Science and Technology Presenter Richard Weaver Chief Security Officer Head, Security Svcs Dept. Johns Hopkins University Applied Physics Laboratory S E S S I O N S U P P O R T E D B Y A S I S F O U N D AT I O N A N D A S I S D E F E N S E A N D I N T E L L I G E N C E C O U N C I L 2

3 M E T R I C S D A S H B O A R D S 3 O U T L I N E I. ASIS Foundation metrics research project, Security Metrics Evaluation Tool, and ongoing research II. III. Presenting metrics data to C-suite Metrics dashboards for decision making and ROI demonstration 3

4 M E T R I C S D A S H B O A R D S 4 A S I S F O U N DAT I O N M E T R I C S R E S E A R C H Persuading Senior Management with Effective, Evaluated Security Metrics (2014) Nine criteria for evaluating metrics --Reliability, Validity, Generalizability --Cost, Timeliness, Manipulation --ROI, Org. Relevance, Communication Library of evaluated metrics Please contribute your metric at 4

5 M E T R I C S D A S H B O A R D S 5 S E C U R I T Y M E T R I C S E VA L UAT I O N T O O L ( S E C M E T ) Discern strong and weak points of a security metric Refine metric to optimize its scientific merit, operational reasonableness, and strategic relevance More persuasive to senior management 5

6 6 C R I T E R I O N 5 : T I M E L I N E S S Extent to which metric data can be gathered in a timely fashion so the results can have an impact. The data for this metric is out-of-date by the time it can be gathered and interpreted; the data collection process is very time-consuming; the data is unlikely to have an impact (as it does not reflect current conditions). The data for this metric is fairly up-to-date by the time it can be gathered and interpreted; the data collection process is somewhat time-consuming; the data is somewhat likely to have an impact (as it somewhat reflects current conditions). The data for this metric is very up-to-date when gathered and interpreted; the data collection process is not timeconsuming; the data is very likely to have an impact (as it reflects current conditions)

7 M E T R I C S D A S H B O A R D S 7 I T S I M P O R TA N T T O T H I N K A B O U T T H E WAY O N E M A K E S D E C I S I O N S Halo effect Outcome/hindsight bias Confirmation bias Regression to the mean Wet bias A I M F O R L E S S W R O N G N E S S 7

8 M E T R I C S D A S H B O A R D S 8 K A R L P O P P E R, P H I L O S O P H E R O F S C I E N C E How can we hope to detect and eliminate error? By criticizing the theories or guesses of others and if we can train ourselves to do so by criticizing our own theories or guesses. Conjectures and Refutations: The Growth of Scientific Knowledge,

9 9 O B S E R VAT I O N S O N N E W M E T R I C S Metric of completed guard tours: does it discourage stopping to address a problem? Metric of driving time saved by conducting investigations long-distance: does it adequately consider quality factors or lean toward speed, convenience, and cost? 9

10 10 P R E S E N T I N G M E T R I C S TO C - S U I T E Corporate management tends to view security as overhead (cost center, not production center) and security metrics as merely measuring activity, not value. Security benefits are difficult to measure compared to the benefits of profit centers. Security professionals often lack the skills or time to create and administer effective metrics. Thus, current security metrics, in practice, are generally not compelling and are often not taken seriously (Rothke, 2009). 10

11 11 P R ESENTING METRICS TO C - S U I T E Make Metrics Compelling: an overview Present metrics that are aligned with the organization s objectives or risks or that measure the specific issues management is most interested in. Present metrics that meet measurement standards. Tell a story. Use graphics, and keep presentations short. Present metric data regularly. 11

12 12 P R ESENTING METRICS TO C - S U I T E Align with Organizational Objectives and Risks Risk: Metrics-based approach helps senior management understand the level of risk in site selection and make informed decisions on risk management. ROI: There is a clear link between reducing shrinkage and saving money. Your metrics must demonstrate that investment in security technology led to reduced losses. 12

13 13 Risk Vs Return on Investment 13

14 14 P R ESENTING METRICS TO C - S U I T E Present Metrics That Meet Measurement Standards Metrics are quantitative and exude scientific authority. However, if metric is based on invalid or unreliable data, you cannot draw accurate conclusions from it and it will lack external credibility. A metric that has been properly designed from a scientific point of view and that has been evaluated against a testing tool (such as the Security MET) may appear more valuable and persuasive to senior management. Using a metric that meets measurement standards also provides an objectivity that aids decision-making. 14

15 15 Risk Measurement Standards Pherson associates, llc. all rights reserved. 15

16 16 P R ESENTING METRICS TO C - S U I T E Tell a Story Can be a story about the specific risk that security is attempting to mitigate, as well as the consequences if the event occurs. Be straightforward about risk and uncertainties. Part of a compelling story is the unfolding of events over time. Metrics can show progress toward meeting a specific strategic goal. Benchmarking can enrich a story if it is aligned with strategic organizational goals. Benchmarking provides the opportunity to ascertain where company stands on a given metric in relation to its competitors. 16

17 17 Security Threat Risk Mitigation 17

18 18 P R ESENTING METRICS TO C - S U I T E Use Graphics, and Keep Presentations Short Keep it simple and clear. Present a few short bullet points top-level information only, rather than complex charts and graphs. Less is more. Pick graphics that get your points across. One graphic = 1,000 words. Keep presentation short (but still tell a story). Present metrics in the style or format management uses. 18

19 19 19

20 20 P R ESENTING METRICS TO C - S U I T E Present Metric Data Regularly Data ages over time. Distinguishing metrics that are time-sensitive from those that provide value over time will enhance the overall value of metrics. Comparing historical data against current data will show trends. Do not hide painful data from management. Good metrics are the key to demonstrate ROI. 20

21 21 S E C U R I T Y M E T R I C S : W H AT T O M E A S U R E? S O M E G U I D I N G P R I N C I P L E S Be mindful that the process of collecting data and reporting metrics can be extremely time consuming and may unintentionally divert staff from performing work that needs to get done Therefore, confine metrics to only those things that provide useful insight into aspects of operations that are actionable and will lead to delivering improved service to customers and/or will reduce security risks Make every effort to determine the most critical concerns of senior management, and implement metrics that link to those concerns and that will demonstrate value and return on investment To the extent possible, leverage technology and automation to collect and analyze metrics data, thereby avoiding or minimizing manually intensive processes 21

22 22 T H E $ 6 4, Q U E S T I O N : W H AT TO M E A S U R E? Example: Enterprise Classified IT Security - Number of Systems Administrators and ISSOs - Number of Systems (overall) - Number of Networked Systems - Number of WANs - Number of Classified SSP Submissions - Number of Incomplete SSP Submissions Returned - Number of Classified ATOs Received - Number of Users Trained - Number of Authentication Tokens Distributed - Media Write Access Authorized - Number of Privileged Users - Number of Authorized Data Transfer Agents - Number of Classified VTCs Conducted - Number of Mobile Devices - Number of IT related Security Violations and Infractions - Number of Systems Involved in a Classified Spill - Median Number of Days to Receive a Classified ATO - Results of Accreditation/Oversight Inspections - Results of Customer Satisfaction Surveys 22

23 23 S E L E C T I N G M E T R I C S T H AT B E S T F I T YO U R S E C U R I T Y O P E R AT I O N S ( V 3 ) Volume Numbers (counts) to track and assess level of security activity Easiest to collect Useful in defending, adjusting and seeking additional resources Velocity Data to capture and assess speed of delivering a security product or service Useful in evaluating process efficiencies and identifying opportunities for improvement Helpful in communicating expectations to customers, partners and stakeholders 23

24 24 Value S E L E C T I N G M E T R I C S T H AT B E S T F I T YO U R S E C U R I T Y O P E R AT I O N S ( V 3 ) Metrics to demonstrate the importance of Security to the overall health and productivity of an organization, capturing key care-abouts of senior management Harder to identify, develop and measure Highlights Return on Investment (ROI) by answering so what questions May include Volume and Velocity data but will be outcome oriented Helpful in providing high-level situational awareness of threats, vulnerabilities and success of mitigating countermeasures Assists Senior Management in making decisions to accept risk, or to take action to lower risks Displays/dashboards are useful, and anecdotes (stories, narrative example and explanations) are important to accompany numbers 24

25 25 Volume S E L E C T I N G M E T R I C S T H AT B E S T F I T YO U R S E C U R I T Y O P E R AT I O N S ( V 3 ) : S O M E E X A M P L E S Visit requests and clearance certifications processed; badges/tokens fabricated and issued; internal access control transactions; security incidents reported; foreign travel and other Security/CI awareness briefings administered Velocity Personnel Security clearance cycle time (nomination to indoctrination); IT accreditations (timelines associated with submission of plans to ATO); response time to alarm annunciations and other emergency circumstances Value Corporate savings (cost avoidance) attributable to security actions taken; security systems reliability; compliance inspection, audit and red team assessment results; elimination or reduction of undesirable events 25

26 26 D ATA TO D A S H B O A R D O N E WAY I T S D O N E : Data is collected both in real time and on a periodic basis, depending on customer and senior management requirements, and on intended use in security operations, to include adjustment of resources Data is collected from a variety of sources: Excel spreadsheets, external SASS (Service Now) and other databases; subsequently using Microsoft SSIS, data is loaded and transformed into Microsoft SQL Server database Once data is collected and aggregated in a local data mart, then metrics are calculated and displayed via a SharePoint portal utilizing Power BI 26

27 27 27

28 28 28

29 29 29

30 30 30

31 31 31

32 32 32

33 33 33

34 34 34

35 35 35

36 36 36

37 37 37

38 38 P R E S E N T I N G P I C T O R I A L D I S P L AY S O F M E T R I C S D ATA T O S E N I O R M A N A G E M E N T : T H E G O L D E N R U L E CHARTS, GRAPHS, DASHBOARDS, DIAGRAMS, TABLES AND ILLUSTRATIONS SHOULD BE USED ONLY SELECTIVELY AS A TOOL TO MAKE KEY POINTS 38