Comparison of Risk Analysis Methods: Mehari, Magerit, NIST800-30 and Microsoft s Security Management Guide Amril Syalim Graduate School of Information Science and Electrical Engineering Kyushu University, Fukuoka, Japan Email: amr@itslab.csce.kyushu-u.ac.jp Yoshiaki Hori and Kouichi Sakurai Faculty of of Information Science and Electrical Engineering Kyushu University, Fukuoka, Japan Email: {hori, sakurai}@csce.kyushu-u.ac.jp Abstract In this paper we compare four risk analysis methods: Mehari, Magerit, NIST800-30 and Microsoft s Security Management Guide. Mehari is a method for risk analysis and risk management developed by CLUSIF (Club de la Sécurité de l Information Français). Magerit is a risk analysis and management methodology for information systems developed by CSAE (Consejo Superior de Administración Electrónica). NIST800-30 is a risk management guide for information technology systems recommended by the National Institute of Standard and Technology (NIST) in NIST Special Publication 800-30. Microsoft s Security Management Guide is a security risk management guide developed by Microsoft. In this paper, we compare those methods based on two main criteria: the first criterion is the steps that are used by the methods to conduct the risk assessment, the second one is the contents of the methods and supplementary documents provided with them. We found that all methods follow the first three general steps of risk analysis. However, the Mehari method, the Magerit method and the Microsoft Security Management Guide do not include control recommendations. Control recommendations in these methods are proposed as the next step to security management (i.e. after risk analysis). All methods provide a detailed guide for risk analysis. However, only three methods Mehari, Magerit and the one proposed in the Microsoft Security Management Guide provide supplementary documents for risk assessment. I. INTRODUCTION Risk assessment or risk analysis is the process of identifying the security risks to a system and determining their probability of occurrence, their impact, and the safeguards that would mitigate that impact. Risk assessment is one step in the process of risk management. The main problem in risk assessment is how to assess all risks in a system/organization so that by using the output of risk assessment, these organizations could define appropriate controls for reducing or eliminating those risks. The method to assess risks is generally composed of the four following steps: thread identification, vulnerability identification, risk determination and control recommendation. These four steps of risk assessment are based on practical experiences in security assessment. These steps come from best practices that have been applied by many organizations for security assessment. There is no standard for risk assessment. Standards like ISO/IEC 27001 and 27002 [12], [13] do not define detailed steps of risk assessment, so if we want to use such standards we have to define our own security assessment method or we can use methods that have been developed by other organizations. There is no formal proof that by following those four steps one can get a list of all the security risks threatening a system and get all appropriate control measures. Actually, formal methods can be used to formally prove the results of risk assessment. However, using formal methods for risk assessment is too complicated, time consuming and needs high expertise in formal programming. Furthermore, formal methods do not show a convincing progress for proofing security of a complex system because too many objects and aspects that should be analyzed. There are many methods that have been developed by many organizations for risk analysis. In this paper we choose four of them and compare the four methods. We choose the methods based on our experience in doing risk analysis/assessment. It does not imply that the four methods are better than the other methods. The four risk assessment methods are: Mehari, Magerit, NIST800-30 and Microsoft s Security Management Guide. Mehari is a method for risk analysis and risk management developed by CLUSIF (Club de la Sécurité de l Information Français), France. Magerit is a risk analysis and management methodology for information systems developed by CSAE (Consejo Superior de Administración Electrónica), Spain. NIST800-30 is a risk management guide for information technology systems recommended by National Institute of Standard and Technology (NIST) in NIST Special Publication 800-30. Microsoft s Security Management Guide is the security risk management guide developed by Microsoft. We compare those methods based on two main criteria: the first criterion is the steps that are used by the methods to conduct the risk assessment, the second one is the contents of the methods and supplementary documents provided with them. We found that all methods follow the first three general steps of risk analysis. However, the Mehari method, the Magerit
method and the Microsoft Security Management Guide do not include control recommendations. Control recommendations in these methods are proposed as the next step to security management (i.e. after risk analysis). All methods provide a detailed guide for risk analysis. However, only three methods Mehari, Magerit and the one proposed in the Microsoft Security Management Guide provide supplementary documents for risk assessment. Organization of this paper is as follow: first we discuss background of risk assessment. After that we discuss each of the risk assessment method: Mehari, Magerit, NIST800-300 and Microsoft Security Management Guide. Then we give our comparison results and conclude the paper. II. BACKGROUND Risk assessment is one process of the risk management methodology. Organizations use risk assessment to determine the extent of the potential threat and the risk associated with an IT system. The output of this process helps organizations to identify appropriate measures for reducing or eliminating risk during the risk mitigation process. Risk is a function of the likelihood of a given threatsource exercising a particular potential vulnerability, and the resulting impact of that adverse event on the organization. A threat-source is any circumstance or event with can or has the potential to cause harm to a system. The threat sources can be natural, human, or environmental. A threat-source does not present a risk when there is no vulnerability that can be exploited. A threat is the potential for a particular threat-source to successfully exercise a particular vulnerability. A vulnerability is a weakness or flaw in system/organization security procedures, design, implementation, or internal controls that could be exploited (accidentally or intentionally) and result in a security breach or a violation of the system s security policy. To determine the likelihood of a future adverse event, threats to an IT system must be analyzed in conjunction with the potential vulnerabilities and the controls for the IT system. Impact refers to the magnitude of harm that could be caused by a threat s exercise of a vulnerability. The impact level is governed by the potential impacts and a relative value for the IT assets and resources affected. Generally there are four steps of risk assessment. The four steps are as follow: 1) Threat Identification This steps identifies all potential threats to the system. Threat identification identifies the potential threatsources and develop a list of a threat statement that is potential threat-sources that are applicable to the system. 2) Vulnerability Identification The goal of vulnerability identification is to develop a list of system vulnerabilities (flaws or weaknesses) that could be exploited by the potential threat-sources. 3) Risk Determination The purpose of this step is to assess the level of risk to the system. The determination of risk for a particular threat/vulnerability pair can be expressed as a function of: The likelihood of a given threat-source s attempting to exercise a given vulnerability The magnitude of the impact should a threat-source successfully exercise the vulnerability The adequacy of planned or existing security controls for reducing or eliminating risk. 4) Control Recommendation In this step, controls that could mitigate or eliminate the identified risks, as appropriate to the system/organization s operations, are provided. The goal of the recommended controls is to reduce the level of risk to the system and its data to an acceptable level. The results of risk assessment are typically a set of list of threats, list of vulnerabilities, list of level of risks and list of control determinations. There are no standards in how to develop the lists. Usually the methods to develop the lists are based on best practices and experiences. III. MEHARI, MAGERIT, NIST800-30 AND MICROSOFT S SECURITY MANAGEMENT GUIDE In this section we introduce the four methods of risk analysis that we compare in this paper: Mehari, Magerit, NIST800-30 and Microsoft s Security Management Guide. A. Mehari Mehari was originally designed to assist Chief Information Security Officers (CISOs) in their information system security management tasks [2]. Mehari aims to provide a set of tools specifically designed for security management, which comprises a set of managerial actions, each of which has a specific goal. Some examples of these are: Developing security plans, or strategic plans, Implementing security policies or rules, Running light or detailed assessments of the state of security, Risk evaluation and management Ensuring the inclusion of security in the management of development projects, Security awareness and training sessions, Operational security management and the control/monitoring of committed actions. Mehari also gives a guideline in security assessment. The steps to conduct risk assessment using Mehari is as follow [3]: 1) Identification of a risk situation There are two main ways to identify risks: A direct approach, using the malfunction value scale. Identification of malfunctions or potential events starts with the activities of the organization and consists in identifying possible malfunctions in the operational processes. It will result in: A description of the possible types of malfunction,
A definition of the parameters that influence the seriousness of each malfunction, An evaluation of the critical thresholds of these parameters that change the level of seriousness of the malfunction. An organized and systematic approach with an automated evaluation using the scenario base provided by Mehari. Mehari provides an extensive knowledge base for doing automated evaluation. We can use the audit forms provided by Mehari knowledge base for doing security risk assessment. 2) Evaluation of natural exposure The natural exposure is risk that comes from the environment. In Mehari, natural exposure to risk can be classified on scale from 1 to 4: Level 1: Very low exposure. Independently of any security measures, the probability that such a scenario will occur is very low and practically negligible. Level 2: Low exposure (hardly exposed). Even without any security measures at all, the combination of the environment (cultural, human, geographic or other) and the context (strategic, competitive, social) make the probability that such a scenario will occur, in the short or medium term, low. Level 3: Medium exposure (not particularly exposed). The environment and context of the enterprise are such that, if nothing is done to avoid it, such a scenario is bound to happen in the more or less short term. Level 4: High exposure (particularly exposed). The environment and context of the enterprise are such that, if nothing is done to avoid it, such a scenario is inevitable in the very short term. 3) Evaluation of dissuasive and preventive factors In this step we should audit the dissuasive and preventive factors that can prevent the risk to occur. 4) Evaluation of protective, palliative and recuperative factors In this step we evaluate the protective, palliative and recuperative factors that can be done after the risks occur. 5) Evaluation of Potentiality In this step we should evaluate the potentiality of risk to occur. We have to answer the question: How likely is the occurrence of the risk being analyzed. Is that scenario completes and creates real damage?. There are five scale of potentiality: Level 0: Not considered. These are scenarios that are so impossible that they are not included in the set of scenarios to be analyzed. Level 1: very unlikely. The occurrence of the risk is totally improbable. Level 2: Unlikely. These are scenarios that, reasonably, could be considered never to happen. Level 3: Likely. These are scenarios that could easily occur, in the more or less short term. Level 4: Very likely. At this level, the scenario can be considered to certainly occur, and in the relatively short term. 6) Evaluation of intrinsic impact The definition of intrinsic impact of a scenario is the evaluation of the consequences of the risk event actually happening, independently of any security measures. In Mehari, the approach used to evaluate intrinsic impact consists of filling in an intrinsic impact table. Mehari knowledge base also give us an intrinsic impact table that can be used for evaluation process. 7) Evaluation of impact and impact reduction Mehari provides an automated evaluation of impact, starting from the intrinsic impact of the scenario and the levels of protective, palliative and recuperative measures. The evaluation is made in two steps: Evaluation of an impact reduction indicator Impact evaluation Mehari also provides an evaluation of impact reduction. This impact reduction factor measures the attenuation of the consequences of the risk, compared to the intrinsic impact previously evaluated. 8) Global risk evaluation After doing the previous steps Mehari define the global risks to the organization. 9) Decision on whether risk is acceptable In this step we should define whether the risk is acceptable or not. If the risk is unacceptable we have to develop the control mechanism to prevent the risk to occur. B. Magerit Magerit was prepared and promoted by CSAE in response to the perception that the government (and, in general, the whole society) increasingly depends on information technologies for achieving its service objectives [7]. Risk analysis using Magerit is following these steps: 1) Determine the relevant assets for the organization, their inter-relationships and their value i.e. what cost would be caused by their degradation The assets are the resources in the information system or related to it that are necessary for the system/organization to operate correctly and achieve the objectives proposed by its management. The essential asset is the information handled by the system, that is the data. Other relevant assets can be identified around these data, for example [7]: The services that can be provided to these data and the services needed to be able to manage these data The computer applications (software) that allow these data to be handled. The computer equipment (hardware) that hosts the data, applications and services. The information media, which are data storage devices.
The auxiliary equipment that complements the computer equipment. The communications networks that allow the exchange of data. The installations that house the computer and communications equipment. The persons who use or operate all the above elements. 2) Determine the threats to which those assets are exposed Threats are things that could happen to the assets and cause damage. There are threats from natural disasters (earthquakes, floods, etc) and industrial accidents (pollution, electrical failures, etc). There are threats caused by persons, either through errors or intentional attacks. 3) Determine what safeguards are available and how effective they are against the risk Safeguards or counter-measures are procedures or technological mechanisms that reduce the risk. There are threats that can be removed simply by suitable organizational mechanism; others require technical devices (programs or equipment). Others need physical security and the personnel policy. 4) Estimate the impact, defined as the damage to the asset arising from the occurrence of the threat Impact is the measurement of the damage to an asset arising from the appearance of a threat. By knowing the value of the assets and the damage caused by the threats, their impact on the system can be derived. 5) Estimate the risk, defined as the weighted impact on the rate of occurrence (or the expectation of appearance) of the threat Risk is the measurement of the probable damage to the system. Knowing the impact of the threats to the assets, the risk can be derived by taking into account the frequency of occurrence. The risk increases with the impact and with the frequency. C. NIST800-30 NIST800-30 has been developed by NIST in furtherance of its statutory responsibilities under the Computer Security Act of 1987 and the Information Technology Management Reform Act of 1996 [10]. The document are for use by Federal organizations which process sensitive information. The steps of risk analysis using NIST800-30 are as follow: 1) System Characterization In this step, the boundaries of the IT system are identified, along with the resources and the information that constitute the system. Characterizing an IT system establishes the scope of the risk assessment effort, delineates the operational authorization boundaries, and provides information (e.g., hardware, software, system connectivity, and responsible division or support personnel) essential to defining the risk. 2) Threat Identification In determining the likelihood of a threat, one must consider threat-sources, potential vulnerabilities, and existing controls. 3) Vulnerability Identification The goal of this step is to develop a list of system vulnerabilities (flaws or weaknesses) that could be exploited by the potential threat-sources. 4) Control Analysis The goal of this step is to analyze the controls that have been implemented, or are planned for implementation, by the organization to minimize or eliminate the likelihood (or probability) of a threat s exercising a system vulnerability. 5) Likelihood Determination This step derives an overall likelihood rating that indicates the probability that a potential vulnerability may be exercised within the construct of the associated threat environment. The governing factors that must be considered are: Threat-source motivation and capability Nature of the vulnerability Existence and effectiveness of current controls 6) Impact Analysis This step determines the adverse impact resulting from a successful threat exercise of a vulnerability. 7) Risk Determination The purpose of this step is to assess the level of risk to the IT system. 8) Control Recommendations This step provide controls that could mitigate or eliminate the identified risks. The recommended controls should reduce the level of risk to the IT system and its data to an acceptable level. 9) Results Documentation This step develop report of the risk assessment result (threat-sources, vulnerabilities, risks assessed, and recommended controls). D. Microsoft s Security Management Guide Microsoft s Security Management Guide is a guide that Microsoft has published that focuses entirely on security risk management [11]. In the Microsoft s Security Management Guide, the Assessing Risk phase represents a formal process to identify and prioritize risks across the organization [11]. The Assessing Risk phase in the Microsoft s Security Management Guide is divided into the following three steps: 1) Planning, building the foundation for a successful risk assessment In the Microsoft s Security Management Guide, the planning step is important to ensure stakeholder acceptance and support throughout the risk assessment process. Stakeholder acceptance is critical, because the Security Risk Management Team requires active participation from other stakeholders. Support is also critical because the assessment results may influence stakeholder budgeting activities if new controls are required
to reduce risk. The primary tasks in the planning step are to properly align the Assessing Risk phase to business processes, accurately scope the assessment, and gain stakeholder acceptance [11]. There are three main tasks in the planning step: a) Alignment of the timing of the assessment b) Scoping c) Stakeholder Acceptance 2) Facilitated data gathering, collecting risk information through facilitated risk discussions The purpose of this step is to gather risk related information from stakeholders across the organization. The primary data elements collected during the facilitated data gathering step are: Organizational assets, that is anything of value to the business. Asset description, that is brief explanation of each asset, its worth, and ownership. Security threats. Causes or events that may negatively impact an asset, represented by loss of confidentiality, integrity, or availability of the asset. Vulnerabilities. Vulnerabilities are weaknesses or lack of controls that may be exploited to impact an asset. Current control environment. Description of current controls and their effectiveness across the system/organization. Proposed controls. Initial ideas to reduce risk. The steps in facilitated data gathering are: a) Risk discussion with stakeholders b) Identifying and classifying assets c) Organizing the risk information: assets, threats, vulnerabilities, and controls d) Data gathering summary 3) Risk prioritization, ranking identified risks in a consistent and repeatable process The prioritization process adds the element of probability to the impact statement. A well formed risk statement requires both the impact to the organization and the probability of that impact occurring. The steps in risk prioritization are: a) Conduct summary level of risk prioritization b) Review with stakeholder c) Conduct detailed level of risk prioritization Summary level of risk prioritization determine impact value from impact statements collected in the data gathering process and estimate the probability of the impact for the summary level list. A complete summary level list is developed by combining the impact and probability values for each risk statement. The review with stakeholder is needed to update stakeholders knowledge about the risk assessment process and solicit their input to help select which risks to conduct in a detailed level analysis. The detailed level of risk view is more specific in its impact and probability descriptions. IV. COMPARISON OF THE FOUR METHODS In this section we compare the four methods. There are two main points that we compare: 1) the steps which are used by the methods for doing risk assessment, and 2) the contents of the methods and supplementary documents included in them A. The steps which are used by the methods for doing risk assessment Below is summary of the steps which are used by the four methods for risk assessment. There are 9 steps for risk analysis in the Mehari. The steps are: (1) identification of a risk situation, (2) evaluation of natural exposure, (3) evaluation of dissuasive and preventive factors, (4) evaluation of protective, palliative and recuperative factors, (5) evaluation of potentiality, (6) evaluation of intrinsic impact, (7) evaluation of impact and impact reduction, (8) global risk evaluation, and (9) decision on whether risk is acceptable. The steps in Mehari follows the first three of the general steps of risk analysis: threat identification, vulnerability identification and risk determination. However the Mehari does not include control recommendation in the steps of risk analysis. Control recommendation is included in the next step of security management after risk analysis. There are 5 steps for risk analysis in the Magerit. The steps are: (1) determine the relevant assets for the organization, their inter-relationships and their value i.e. what cost would be caused by their degradation, (2) determine the threats to which those assets are exposed, (3) determine what safeguards are available and how effective they are against the risk, (4) estimate the impact, defined as the damage to the asset arising from the appearance of the threat, (5) estimate the risk, defined as the weighted impact on the rate of occurrence (or the expectation of appearance) of the threat. The steps in Magerit includes the first three of the general steps of risk analysis: threat identification, vulnerability identification and risk determination. However the Magerit does not include control recommendation in the steps of risk analysis. Control recommendation is included in the next step of security management after risk analysis. There are 9 steps for risk analysis in the NIST800-30: (1) system characterization, (2) threat identification, (3) vulnerability identification, (4) control analysis, (5) likelihood determination, (6) impact analysis, (7) risk determination, (8) control recommendations, and (9) results documentation. The nine steps of risk analysis in the NIST800-30 includes all of the general steps of risk analysis: threat identification, vulnerability identification and risk determination and control recommendation. There are 3 steps of risk analysis in the Microsoft s Security Management Guide: (1) planning, building the foundation for a successful risk assessment, (2) data gathering, collecting risk information through facilitated risk discussions, (3) risk prioritization, ranking identified risks in a consistent process.
The steps in the Microsoft s Security Management Guide includes the first three of the general steps of risk analysis: threat identification, vulnerability identification and risk determination. However the the Microsoft s Security Management Guide does not include control recommendation in the steps of risk analysis. Control recommendation is included in the next step of security management after risk analysis. In brief, all of the methods follow the first three of the general steps of risk analysis: (1) threat identification, (2) vulnerability identification, and (3) risk determination. Only NIST-800-30 includes control recommendation in the step of risk analysis. Mehari, Magerit and Microsoft Security Management Guide do not include control recommendation. Control recommendation in these three methods are included in the next step of security management after risk analysis. B. The contents of the methods and supplementary documents included in them Below is the contents of the four methods and supplementary documents included in them. Mehari consists of 5 documents [2], [3], [4], [5], [6] and one extensive knowledge base in Microsoft Excel format. These documents provide a detailed guide for risk analysis. The knowledge base is a supplementary documents that help in doing risk assessment using Mehari. Magerit consists of 3 documents [7], [8], [9]. The first document [7] is the main document that provide a detailed guide for doing risk analysis using Magerit. The second document is a supplementary document which provide template and framework for doing risk assessment. The third document is a supplementary document that discuss a more detailed and formal techniques for risk analysis. NIST800-30 only consists of 1 document [10]. However, this document provide a detailed guide for risk assessment although with no supplementary document for helping activity of risk assessment using the method. Microsoft s Security Management Guide consists of 1 main document and 4 supplementary documents in Microsoft Word and Excel format. This guide give a detailed guide for risk analysis and the supplementary documents help the activity of risk assessment using this method. VI. ACKNOWLEDGMENTS The authors would like to thank Erwan Le Malécot and Yi Han for their valuable comments. REFERENCES [1] TR Peltier, Information Security Risk Analysis, Auerbach, 1995. [2] Mehari 2007 - Overview, Club de la Sécurité de l Information Français (CLUSIF), 2007. [3] Mehari 2007 - Concepts and Mechanisms, Club de la Sécurité de l Information Français (CLUSIF), 2007. [4] Mehari 2007 - Risk Analysis Guide, Club de la Sécurité de l Information Français (CLUSIF), 2007. [5] Mehari 2007 - Security Stakes Analysis and Classification Guide, Club de la Sécurité de l Information Français (CLUSIF), 2007. [6] Mehari 2007 - Evaluation Guide for security services, Club de la Sécurité de l Information Français (CLUSIF), 2007. [7] Magerit - version 2 - Methodology for Information Systems Risk Analysis and Management - Book I - The Method, Ministerio de Administraciones Publicas, Madrid, 20 June 2006. [8] Magerit - version 2 - Methodology for Information Systems Risk Analysis and Management - Book I - Catalogue of Elements, Ministerio de Administraciones Publicas, Madrid, 20 June 2006. [9] Magerit - version 2 - Methodology for Information Systems Risk Analysis and Management - Book I - Techniques, Ministerio de Administraciones Publicas, Madrid, 20 June 2006. [10] Gary Stoneburner, Alice Goguen, and Alexis Feringa, Risk Management Guide for Information Technology Systems, NIST Special Publication 800-30, July 2002. [11] The Security Risk Management Guide, Microsoft Solutions for Security and Compliance and Microsoft Security Center of Excellence. 2006. [12] BS ISO/IEC 27001:2005, Information technology - Security techniques - Information security management systems - Requirements. BSI, 2007. [13] BS ISO/IEC 27002:2005, Information technology - Security techniques - Code of practice for information security management. BSI, 2007. V. CONCLUSION In this paper we have showed and compare four risk analysis methods: Mehari, Magerit, NIST800-30 and the Microsoft s Security Management Guide. We found that all of the methods follow the first three of the general steps of risk analysis: (1) threat identification, (2) vulnerability identification, and (3) risk determination. Mehari, Magerit and Microsoft Security Management Guide does not include control recommendation. Control recommendation in these methods are included in the next step of security management after risk analysis. We also found that all methods provide a detailed guide for risk assessment, but only Mehari, Magerit and Microsoft Security Management Guide provide supplementary documents for helping risk assessment.