PLANNING AND BUILDING QUALIFIABLE EMBEDDED SYSTEMS: SAFETY AND RISK PROPERTIES ASSESSMENT FOR A LARGE AND COMPLEX SYSTEM WITH EMBEDDED SUBSYSTEMS

Size: px

Start display at page:

Download "PLANNING AND BUILDING QUALIFIABLE EMBEDDED SYSTEMS: SAFETY AND RISK PROPERTIES ASSESSMENT FOR A LARGE AND COMPLEX SYSTEM WITH EMBEDDED SUBSYSTEMS"

Ethelbert Porter
5 years ago
Views:

1 PLANNING AND BUILDING QUALIFIABLE EMBEDDED SYSTEMS: SAFETY AND RISK PROPERTIES ASSESSMENT FOR A LARGE AND COMPLEX SYSTEM WITH EMBEDDED SUBSYSTEMS Nuno Silva (1), Rui Lopes (1), Ricardo Barbosa (1) (1) Critical Software, S.A., Parque Industrial de Taveiro, Lote 48, Coimbra, Portugal, {nsilva, rmlopes, rbarbosa}@criticalsoftware.com ABSTRACT Systems based on embedded components and applications are today used in all markets. They are planned and developed by all types of institutions with different types of background experience, multidisciplinary teams and all types of capability and maturity levels. Organisational/engineering maturity has an impact on all aspects of the engineering of large and complex systems. An embedded system is a specific computer system designed to perform one or more dedicated functions, usually with real-time constraints. It is generally integrated as part of a more complex device typically composed of specific hardware such as sensors and actuators. This article presents an experimented technique to evaluate the organisation, processes, system and software engineering practices, methods, tools and the planned/produced artefacts themselves, leading towards certification/qualification. The safety and risk assessment of such core and complex systems is explained, described on a step-bystep manner, while presenting the main results and conclusions of the application of the technique to a real case study. 1. INTRODUCTION Systems based on embedded components and applications are today used in all markets. They are planned and developed by all types of institutions with different types of background experience, multidisciplinary teams and all types of capability and maturity levels. Organisational/engineering maturity has an impact on all aspects of the engineering of large and complex systems. An embedded system is a specific computer system designed to perform one or more dedicated functions, usually with real-time constraints. It is generally integrated as part of a more complex device typically composed of specific hardware such as sensors and actuators. This article presents an experimented technique to evaluate the organisation, processes, system and software engineering practices, methods, tools and the planned/produced artefacts themselves, leading towards certification/qualification. The safety and risk assessment of such core and complex systems is explained, described on a step-by-step manner, while presenting the main results and conclusions of the application of the technique to a real case study. The implementation of a proper risk and safety management strategy is essential to guarantee the success in any project. Moreover, quantification, monitoring, control and acting upon the dynamic behavior of risks and the dynamicity of projects shall always be accomplished for the whole project duration. According to good practices of Project Management Institute (PMI), about 10% of the overall project budget should be accounted for risks (or will somehow be related to it). Also, it is quite common to use around 2% for non-quantified risks (either not registered or with low probability and impact). This work has been made possible with support from the CRITICAL STEP project [1]. 2. SAFETY AND RISK MANAGEMENT There are several types of safety concerns, and innumerous types of risks, the current work will focus on embedded systems safety concerns and related projects risks. They are tightly related, since the project risks and safety properties shall be identified, monitored and controlled all along the project duration and during operations and maintenance of the system, being it a simple embedded system or a complex system with embedded components. From this moment on we include the safety management in the risk management process and the safety properties assessment in the risk assessment process. 2.1 Risk Management Plan The Project Manager (PM) shall undertake risk management jointly with key project personnel on a periodic and consistent basis. The main steps for risk management include risk identification, risk analysis and risk response planning. Risk identification involves determining which risks might affect the project or the system safety and documenting their characteristics. The PM shall carry out these activities during the regular project meetings

2 and shall record the potential risks in the Risk Record Database. But the risks and hazards shall also be identified during the project planning and construction. Risk analysis evaluates the impact and likelihood of identified risks. The PM, with the assistance of the project members who can best contribute to the evaluation, assess the impact of the risks on cost, schedule, scope, and quality, the probability of its occurrence and where, in the project structure, they might have impact. This process prioritizes risks according to their potential effect on project objectives. The Risk Record Database shall be updated with the analysis information collected during this process. Risk response planning is the process of determining actions or decisions that will reduce the probability or the impact of each risk event. The PM shall propose such actions/decisions which shall be included in the AIL (Action Item List) when approved. 2.2 Risk Management for the case study Large and complex systems composed by embedded systems or relying on embedded components have specific characteristics that highlight the importance of proper Safety and Risk Management. For the specific case of this study we identify: Past similar projects (lessons learned, unexpected delays, extra effort associated, etc); The case study will not be feasible if similar delays and extra effort is required; The case study will be several times more complex than previous similar projects, using new technologies in which the project team has limited experience; Early identification of project's risks allows planning of preventive actions and implementation of proper management activities in order to mitigate the probability of delays and extra effort. 2.3 Advantages of having Risk Management process in place The main advantages of having a Safety and Risk Management process in place for the case study are identified hereafter: Preventive actions: By identifying the projects risks, the project management is able to plan, and execute if necessary, preventive actions in order to avoid or mitigate the impact of risks consequences in the project; Timely adaptation of project management: Management activities may be created or modified in order to address identified risks, increasing the ability to: o Increase monitoring and control: Risks monitoring allow to request and access information that provide increased monitoring and control over the projects activities. The access to this information may result in further risks identification, contributing to mitigate unexpected events that have impact on projects activities. Monitoring and control allows to: Track the evolution of events that lead to the identified risks; Avoid the events leading to the identified risks; o Mitigation of the probability of project delays and extra effort. Risk Management leads to the implementation of several changes in project management practices and execution of preventive actions with impact in other areas (e.g. engineering, validation, quality assurance). By implementing new activities for risk mitigation, there is always the possibility of new risks being identified by the implementation of these activities. This possibility highlights the importance of classifying the risks according to probability and criticality. A mitigation action shall only be implemented if the associated risks of implementing it have a lower probability/impact than the risk that is being mitigated. 3. RISK REGISTER In order to perform a proper risk identification and assessment, risks shall be correctly defined and classified according to their specific characteristics. Each identified risk shall be defined according to fields such as the ones described in Table 1, used for the current case study.

3 Field Risk ID Risk Statement Probability Impact Impact On Priority Cost Impact Category Table 1: Risk definition fields Description Unique ID for risk identification (e.g. RSK001) Description of the risk. This field shall contain: Risk title Risk description Consequence if risk takes place Likelihood of occurrence. Categorised as: Low (1) Medium (2) High (4) Impact that the consequences if risk takes place has in the project. Categorised as: Low (1) Medium (2) High (4) Indicates the variable impacted by the risk consequence. Categorised as: Budget Schedule Effort Quality Priority ranking of the risk. Calculated based on the "Probability" and "Impact". The highest priority risk shall have priority 16 (4x4, high probability and high impact) and the lowest 1 (1x1, low probability and low impact). Represents the perceived cost of the risk at the instant of evaluation. Can be estimated in: Effort Monetary value Identify which area is risk impacting if it occurs. Hereafter is presented as an example a set of areas that can be used. This shall be tailored according to the project needs. Requirements Risk Response Mitigation strategy Mitigation risks Risk status Status date Assigned To Closing rationale Date identification Design Code and Unit Test Integration and Test Eng. Specialties Develop. Process Develop. System Management Process Management Methods Work Environment Resources Contract Program Interfaces ID of the actions that have been identified to mitigate the risk. The selected strategy for mitigating the risk. IDs of the risks that have been created by the implementation of the "Mitigation strategy" (if any). Running status that provides a history of what is being done for the risk and changes in the risk. Categorised as: Identified Dismissed Monitored Occurred Identify the date corresponding to the latest change in the risk status. The person responsible for mitigating the risk. Rationale for risk closure, when risk becomes Dismissed or Occurred. Date in which the risk has been identified. 4. RESULTS SUMMARY AND CONCLUSIONS This section presents the results summary of the risk assessment performed in the scope of the case study. Table 2 provides the risk classification according to a specific set of categories while Table 3 presents an overview of the risks source.

Table 2: Risks categories results Category Score Requirements 5 Design 1 Code and Unit Test 2 Integration and Test 4 Eng. Specialties 4 Develop. Process 3 Develop.

4 Table 2: Risks categories results Category Score Requirements 5 Design 1 Code and Unit Test 2 Integration and Test 4 Eng. Specialties 4 Develop. Process 3 Develop. System 2 The risks identified part of the main areas of concern represent almost 60% (21 of 36) of the total number of risks for this case study. Management Process 4 Management Methods 2 Work Environment 2 Resources 1 Contract 4 Program Interfaces 2 Total 36 Source Table 3: Risks source results Internal 24 External 12 Total 36 Score For the case study where the safety and risk assessment activities have been performed, risks have been identified in all possible categories. Nonetheless, there are some categories that can be highlighted as main areas of concern regarding the specific case study, namely: Requirements Integration and Test Engineering specialties Management process Contract Figure 1: Risks by criticality Figure 1 shows that 49% of the risks have been classified within the "red zone", which indicates that almost half of the risks should be dealt as soon as possible. This classification is based on the risk probability of becoming real and the impact of its consequences. The percentage of "red zone" risks indicates that the case study responsible should act immediately upon the identified risks and maintain a detailed Risk Register throughout the entire project lifecycle. From all the risks with high priority (i.e. inside de red zone ), 12 out of a total of 17 are part of the referred main areas of concern. Therefore by focusing, at an initial stage, on the identified main areas of concern, one is mitigating risks that are both high priority (i.e. high probability and high impact) and affecting the most vulnerable areas of the project. One shall highlight that, by implementing mitigating actions, new risks might be identified related to this implementation. A careful analysis shall be performed regarding the potential risks of implementing each mitigating action. In order to implement an action, the new associated risks shall have less impact and probability than the risk being mitigated. 5. RISK COST/BENEFIT ANALYSIS This section presents a cost/benefit analysis between the identified risks and the associated mitigating actions. In order to perform this analysis, risks cost impact has been estimated, as well as the cost of mitigating actions.

5 5.1 Cost Impact It is necessary to quantify the impact of the identified risks if not mitigated against the cost of mitigation activities. Quantifying the risks impact on the project before the start of the project is always an academic exercise and shall be evaluated as such. The quantification performed has been based on Critical Software experience on past projects and do not reflect any current impact on the project or case study. The following process has been used to reach the rough estimation for risk cost impact (time impact has not been analyzed): all risks have been identified and detailed; a few of the smallest risks (easier to quantify) have been quantified according to experience; a composition of probability versus impact has been used to grade all the risks; based on this composition all risks have been proportionally quantified; a meeting has been held to discuss the value reached by each risk and to make adaptations according to expert judgment, both from Critical Software and the customer. As expected, some of the risks were very hard to quantify since its impact may vary according to several factors. Nonetheless, the figures reached have been estimated through a conservative approach. This means that the real value would probably be higher than the one presented to the customer. The total cost impact will only become real if all risks would occur. Although this appears to be highly improbable, one shall underline that the majority of the risks identified have become real in past projects of that customer. This indicates that the probability of a considerable number of the risks becoming real is high. 5.2 Mitigation activities All the risks have been analyzed and mitigating actions have been identified. One shall highlight that the identified actions are not strictly related to V&V activities. There were several risks with more than one mitigating action identified, not all related to V&V activities and there were also risks not mitigated by V&V activities. Two thirds (24 risks, 67%) of the identified risks could be mitigated by V&V related actions, which demonstrates the impact that V&V activities can have naturally in risk mitigation. Therefore, V&V activities cost impact has been analyzed for the cost/benefit analysis for this case study (and compared to the sum of risk cost of all risks that are mitigated by V&V activities). Besides the high number or risks addressed by the V&V activities, one shall highlight that the same V&V activity is able to address several risks at the same time. By performing a single V&V activity, several risks are mitigated which highlights the activities effectiveness and added value. The mitigation capability of the V&V activities is also different from risk to risk. 5.3 Costs and effectiveness V&V costs are mainly driven by the size of the SW to be analyzed. The project estimates are still high-level, at this stage of the case study. Nonetheless, the project development is planned to be executed in a large amount of years due to the embedded systems complexity and number of components. Since the considered V&V activities follow the entire SW development process, V&V costs have been estimated taking in consideration the following assumptions: V&V activities shall last for the full duration of the project (following the entire SW development process) On average, one engineer shall be allocated to the V&V activities at customer premises Another important metric is the effectiveness of V&V activities in mitigating risks. According to industry assumptions, V&V activities are able to identify and solve between 80% and 90% of the SW issues. Taking a conservative approach, the V&V activities have been considered to have an effectiveness of 80% in this analysis. 5.4 Cost/Benefits Conclusions Hereafter are presented the results of the cost/benefit analysis, taking in consideration the cost estimates deducted from the activities of the previous subsections. The following assumptions shall be taken into account: No other mitigating actions besides V&V related actions are being considered for this analysis;

6 V&V related mitigating action have the same effectiveness in all risks (reduce 80% of the risk cost); Not all risks are mitigated by V&V related actions. Finally, by taking the estimation of the identified risks and their possible impact in terms of extra cost, delays, or safety issues and by comparing with the simple V&V mitigation actions the project concluded that a reduction of 40% of costs would be possible. The estimation for the effectiveness of the V&V activities was estimated at around 50%. 6. ACKNOWLEDGMENT This work has been partially supported by the project CRITICAL Software Technology for an Evolutionary Partnership (CRITICAL-STEP, Marie Curie Industry-Academia Partnerships and Pathways (IAPP) number , within the context of the EU Seventh Framework Programme (FP7). 7. REFERENCES 1. Critical Step website,

Project Theft Management,

Project Theft Management, by applying best practises of Project Risk Management Philip Rosslee, BEng. PrEng. MBA PMP PMO Projects South Africa PMO Projects Group www.pmo-projects.co.za philip.rosslee@pmo-projects.com