Optimal filter and Cost-Benefit Analysis. Outline. Information security risk management. Risk management terminology overview. Notes. Notes.

Transcription

1 Optimal filter and Cost-Benefit Analysis Tyler Moore CSE 7338 Computer Science & Engineering Department, SMU, Dallas, TX Lecture 3 Outline / 53 Just as it can be useful to translate infosec risks and defenses into the language of investment (ROSI, NPV, etc.), one must also be aware of terminology from risk management As IT becomes essential to many businesses, border between information security investment and general risk management has blurred 4 / 53 Risk management terminology overview Risk analysis identification quantification Risk management acceptance mitigation avoidance transfer Cyberinsurance Risk monitoring validation documentation 5 / 53

2 Risk acceptance Risk acceptance After risks are identified and quantified, they must be managed The simplest option is to do nothing Such risk acceptance is prudent when: Worst-case loss is small enough to be paid from proceeds or reserves 2 Probability of occurrence is smaller than other business risks that threaten the organization s survival This is why the security policies for start-ups are often weaker than for entrenched firms 6 / 53 Risk mitigation Risk mitigation If risk is too big and probable to be accepted, risk mitigation aims to reduce the probability and severity of a loss This is where security investment comes in Recall that the optimal level of investment normally leaves residual risk that must be dealt with using acceptance, avoidance, or transfer 7 / 53 Risk avoidance Risk avoidance Aims to reduce the probability and severity of loss, as in risk mitigation However, rather than use technology, here one forgoes risky activities This introduces opportunity costs of lost business opportunities Example: online merchant refusing overseas orders due to high fraud risk Example: company disconnects database with customers personal information online Question: what are the opportunity costs in these cases? 8 / 53 Risk transfer Risk avoidance The final option is to buy an insurance contract to recover any future losses incurred This is only available in limited circumstances Why has the cyber-insurance market remained small? Difficulty in quantifying losses Even when possible, many firms would rather keep quiet than share with an insurance company Externalities mean that the costs of insecurity are often borne by others Correlated risk is prevalent 9 / 53

3 Risk avoidance Risk management example: credit card issuers Credit card issuers regularly manage fraud Risk acceptance: fraud is paid from the payment fees charged to merchants 2 Risk mitigation: install anti-fraud technology (raises costs of security) 3 Risk avoidance: downgrade high-risk cardholders to debit or require online verification (leads to lost business) 4 Risk transfer: structure consumer credit risk and sell it on the market 0 / 53 Domain-specific models Up to now we have modeled security investment at a very high level Map costs to benefits, assume diminishing marginal returns to investment, etc. Useful for when justifying security budgets compared to non-security expenditures Not useful for deciding how best to allocate a given security budget Today, we discuss a model for a tactical security investment decision: configuring a filter to balance false positives and negatives 2 / 53 ROC curves Binary classification is a recurring problem in CS Common task: distill many observations to a binary signal {0, }: communications theory S = {undervalued, overvalued}: stock trading S = {reject, accept}: research hypothesis S = {benign, malicious}: security filter Such simplification inevitably leads to errors compared to reality (aka ground truth) 3 / 53 Filter defense mechanism ROC curves Reality Signal no attack attack benign α β malicious α β α: false positive rate, β: false negative rate 4 / 53

4 Receiver operating characteristic ROC curves 0Detection rate β 45 False positive rate α 5 / 53 Receiver operating characteristic ROC curves 0Detection rate β 45 EER dashed EER solid α = β False positive rate α 5 / 53 Model for optimal filter configuration Binary classifiers are imperfect Finding the optimal trade-off, say for an IDS or spam filter, is hard Can be framed as an economic trade-off between opportunity cost of false positives and losses incurred by false negatives 6 / 53 Model for optimal filter configuration We can see from ROCs that β can be expressed as a function of α. β : [0, ] [0, ] defines the false negative rate as a function of the false positive rate α β(0) =, β() = 0 We assume β (x) < 0 and β (x) 0 7 / 53

5 Model for optimal filter configuration Suppose we rely on a filter to scan incoming attachments for malware a: cost of false positive (blocking a benign ) b: cost of false negative (delivering malicious ) p: probability of containing malware Cost C(α) = p β(α) b + ( p) α a Suppose p = 0., a = $250, b = $500, α = 0., β =.2 C(α) = = $ / 53 : exercise Suppose we rely on a filter to scan incoming attachments for malware. Suppose the cost of dealing with a false negative event is $400, and the cost of dealing with a false positive is $ % of incoming has malware. You can choose between two configurations Config. A: 0% false positive rate and 30% false negative rate Config. B: 25% false positive rate and 5% false negative rate Your task: compute the expected costs for both configurations, and state which configuration you prefer. 9 / 53 Model for optimal filter configuration α = arg min p β(α) b + ( p) α a α which has first-order condition (FOC) after rearranging, we obtain: 0 = δ α ( p β(α ) b + ( p) α a ) β (α ) = p p a b 20 / 53 (continuous ROC curves) 0Detection rate β ( p)a p b α B α A Indifference curves False positive rate α 2 / 53

6 (continuous ROC curves) A B 0Detection rate β 45 EER A = EER B α = β AUC A = AUC B False positive rate α 2 / 53 (continuous ROC curves) A 0Detection rate β ( p)a p b α B 45 α A B False positive rate α 2 / 53 (discrete ROC curves) E F ( p)a p b 0Detection rate β C α D 45 False positive rate α 22 / 53 example (discrete ROC curves) 0.9 E slope /3 0.3 F 0. Detection rate β C 0.2 α D slope 2 ( p)a p b 0.4 slope α = 0.2 if ( p)a p b False positive rate α 23 / 53

7 : exercise 2 Suppose we rely on a filter to scan incoming attachments for malware. Suppose the cost of dealing with a false negative event is $400, and the cost of dealing with a false positive is $ % of incoming has malware. You can choose between two configurations Config. A: 0% false positive rate and 30% false negative rate Config. B: 25% false positive rate and 5% false negative rate Your task Draw the ROC curve for configurations A and B (plus (0% FP, 00% FN) and (00% FP, 0% FN)) 2 Calculate the slope of the indifference curve for the optimal configuration 3 Select the optimal point for the ROC curve 24 / 53 Review of security investment so far Metrics for quantifying security benefits ALE 0 : expected loss without security investment 2 ALE s : expected loss with security investment 3 EBIS s : ALE 0 ALE s 4 ENBIS s : ALE 0 ALE s c High-level investment metrics ROSI 2 NPV 3 IRR 26 / 53 Security investment questions worth answering Q: Should we invest in security? A: Yes, if ENBIS > 0 Q: Should we invest in defense A or B? A: Choose the one with higher ROSI (or NPV if considering longer time horizons) Q: How much should we invest? A: Security investment models (e.g., Gordon-Loeb) say to invest until marginal cost of added security equals marginal benefit Q: Is a security investment cost-effective? A: Yes, if ENBIS > 0 A2: Probably, if the minimum probability of attack required to break even is high enough 27 / 53 (CBA) Used widely in public policy to justify expenditures Quite similar to the security metrics presented earlier, especially ENBIS Emphasis placed on making best-effort estimates of key figures Costs of insecurity (ALE 0 ) 2 Costs of security countermeasures (c) 3 Probability of attack (p 0 ) 4 Risk reduction r = p0 ps p0 In CBA, a security investment is considered cost-effective if ENBIS > 0. CBA exercises estimate the above figures and use the findings as evidence when deciding whether or not to adopt (or continue spending money on) a countermeasure When there is uncertainty over some figures, a range of values is considered 28 / 53

8 ENBIS using risk reduction ENBIS equations from earlier presentations using Bernoulli loss assumptions used p 0 and the improved probability p s We can equivalently express this in terms of reduced risk ENBIS = (p 0 p s ) λ c ENBIS = p 0 p0 p s λ c p 0 ENBIS = p 0 r λ c 29 / 53 ENBIS for multiple sources of loss Up to now, we have assumed that there is a single financial loss λ associated with an attack In fact, losses can take many forms, each with its own magnitude and probability of occurrence Ideally, we would like to account for each type of loss independently and combine into an aggregate measure Suppose there are n loss types. We can calculate the ENBIS as follows: ENBIS = p 0 r λ c n ENBIS = p 0 r (P(λ i attack) λ i ) c i= 30 / 53 tasks Estimate p 0 using available data (sometimes hard) Enumerate the loss types, estimate their cost and conditional probability of occurring once an attack takes place Estimate (or take as input) security costs c Estimate (or take as input) risk-reduction rate r We discuss cost-benefit efforts for two examples: terrorist attacks targeting highway bridges (reading ) and sewer overflows at wastewater facilities (reading 2) 3 / 53 Case : terrorist attacks targeting highway bridges Estimate p 0 using available data (sometimes hard) No known instances in past, so assign small probability (p 0 = 0 4 ) Enumerate the loss types, estimate their cost and conditional probability of occurring once an attack takes place Bridge replacement: $40 million (average of replacement costs for prior collapses), cond. prob. =.0 2 Loss of life: 80 lives with actuarial value $6.3M each, occurring with cond. prob. 0.2 (estimated from prior collapses) Estimate (or take as input) security costs c NPV of 20% of bridge-replacement value amortized over 25 years = $260,000 Estimate (or take as input) risk-reduction rate r Taken to be r = 0.9 High value selected to give benefit the best possible chance of exceeding costs 32 / 53

9 Case : terrorist attacks targeting highway bridges n ENBIS = p 0 r (P(λ i attack) λ i ) c i= ENBIS = Fill in the equation ENBIS = 247K Based on this calculation, the security investment does not seem to be justified. 33 / 53 Case 2: sewage overflows at wastewater facilities Estimate p 0 using available data Original goal: estimate probability of malicious attack triggering large overflows, but there have only been a few publicly reported attacks Revised goal: estimate probability of large sewage overflows triggered by accident or attack, since both can be detected and sometimes prevented by incident detection system California Water Board reported 46 large overflows in one year in state They separately reported that facilities cover 0,593 sewer miles Hence the number of overflows can be expressed as = # miles. Cities with population over 00,000 have an average of,300 sewer miles in their facilities Hence p 0 = 0.54 Note that p 0 is more accurately interpreted here as the expected number of overflows during the time period 34 / 53 Case 2: sewage overflows at wastewater facilities Enumerate the loss types, estimate their cost and conditional probability of occurring once an attack takes place Loss category Data? Direct losses Cleanup costs yes Property damage yes Regulatory costs yes Lost business for victims no Victim health costs no Indirect losses Lost business for non-victims no Broader environmental impact no Psychological distress no We can estimate the costs for the categories we have to arrive at a lower bound for the total cost 35 / 53 Case 2: sewage overflows at wastewater facilities Enumerate the loss types, estimate their cost and conditional probability of occurring once an attack takes place i Loss category λ i P(λ i SO) Comments Cleanup costs 22K Likely underestimate 2 Property damage.4m 0.25 no data for cond. prob. 3 EPA fine 2.89M violations SOs in CA in yrs 2.% of US pop. in CA 36 / 53

10 Case 2: sewage overflows at wastewater facilities Estimate (or take as input) security costs c City Cost factor Cost/year Sewer miles Reference 20K 300 Atlanta 2 39K 225 DC 3 59K 800 San Francisco 6 8K 993 New Orleans 8 57K 600 Estimate (or take as input) risk-reduction rate r Taken to be r = 0.4 Argued that some overflows couldn t be prevented, but some should be 37 / 53 Case 2: sewage overflows at wastewater facilities n ENBIS = p 0 r (P(λ i attack) λ i ) c i= ENBIS = Fill in the equation ENBIS = 67K Based on this calculation, the security investment is justified for the average city. 38 / 53 Case 2: sewage overflows at wastewater facilities Recall that security investment costs and the expected number of large overflows vary by city City Cost/year Sewer miles ENBIS Reference 20K K Atlanta 39K K DC 59K K San Francisco 8K 993-5K New Orleans 57K K 39 / 53 Case 2: sewage overflows at wastewater facilities 40 / 53

11 What if we are uncertain about the accuracy of estimates? When we are uncertain about one or more of the estimated parameters, we can do a breakeven analysis to identify the value a parameter must take for ENBIS = 0. The best parameter to vary is the one that is most uncertain Often, this is p 0, the probability of attack without security investment 4 / 53 Cybersecurity is not the only discipline where estimating probabilities of rare events is difficult The assessment of the probabilities that adversaries will choose courses of action should be the outputs of analysis, not required input parameters Quote is from National Academies of Science report on bioterrorism risks What does this mean for cost-benefit analysis? 42 / 53 Breakeven analysis with probability of attack as output ENBIS = (p 0 p s ) λ c ENBIS = p 0 p0 p s λ c p 0 ENBIS = p 0 r λ c Setting ENBIS to 0 and solving for p 0 : p 0 = c r λ We can then see for a range of parameter values what the corresponding breakeven probability of attack must be to justify security investment 43 / 53 Breakeven analysis for case p 0 = c r λ p 0 = Fill in the equation p 0 = / 53

12 Breakeven probabilities (as percentages) for case Source: 45 / 53 Breakeven analysis for case 2 p 0 = c r λ c p 0 = r (22K +.4M M 0.00) c p 0 = r 40K 46 / 53 Breakeven probability of sewage overflow for case 2 p_0 (Expected # overflows) c=20k c=50k c=00k Risk reduction probability 47 / 53 Breakeven analysis with risk reduction as output ENBIS = (p 0 p s ) λ c ENBIS = p 0 p0 p s λ c p 0 ENBIS = p 0 r λ c Setting ENBIS to 0 and solving for r: r = c p 0 λ We can then see for a range of parameter values what the corresponding breakeven risk reduction must be to justify security investment 48 / 53

13 Breakeven risk reduction for case 2 Breakeven risk reduction probability sewer miles 500 sewer miles 3000 sewer miles Cost ($K) 49 / 53 R code to generate plot br < f u n c t i o n ( c, l, p ) c /( l p ) c o s t s < seq (0,500, by=) p o v e r < f u n c t i o n ( m i l e s =300) 46/0593 m i l e s pdf ( c b r r sewer. pdf ) p l o t ( x=c o s t s, y=br ( c o s t s, 4 0, p o v e r ( ) ), type = l, y l a b = Breakeven r i s k r e d u c t i o n p r o b a b i l i t y, x l a b = Cost ($K ), lwd =2, y l i m=c ( 0, ) ) l i n e s ( x=c o s t s, y=br ( c o s t s, 4 0, p o v e r ( ) ), l t y = dashed, lwd=2) l i n e s ( x=c o s t s, y=br ( c o s t s, 4 0, p o v e r ( ) ), l t y = dotted, lwd=2) l e g e n d ( b o t t o m r i g h t, l e g e n d=c ( 300 sewer m i l e s, 500 sewer m i l e s, 3000 sewer m i l e s ), l t y=c ( s o l i d, dashed, d o t t e d ), lwd=2) 50 / 53 dev. o f f ( ) Exercise: CBA for patient data breaches Suppose that the Acme hospital chain is considering investing in controls to reduce the likelihood of suffering a breach of personal health records Security improvements will cost $2 million per year, and Acme estimates it would lose $50 million from a successful breach of its records Acmes risk management team estimates that protection would reduce its risk to suffering a breach by 40% Problem : Calculate the break-even annual probability of a breach occurring. 5 / 53 Exercise: for patient data breaches Problem : Calculate the break-even annual probability of a breach occurring. Solution: Set ENBIS to 0 and solve for p 0, we get the following: 52 / 53

14 Exercise: for patient data breaches Suppose instead that it is determined that the breach probability is 5%. Problem 2: Based on this updated information, calculate the risk reduction that would be required of security mechnismsm in order to break even. Solution: set ENBIS to 0 and solve for r, we get the following: 53 / 53