Lecture 3: Project Management, Part 2: Verification and Validation, Project Tracking, and Post Performance Analysis

Transcription

1 Lecture 3: Project Management, Part 2: Verification and Validation, Project Tracking, and Post Performance Analysis Prof. Shervin Shirmohammadi SITE, University of Ottawa Prof. Shervin Shirmohammadi ELG

2 V & V Objective: to find problems before the customer does. Verification: verifies whether the product of a given phase satisfies the requirements set at the beginning of that phase Reviews & walkthroughs of many things: plans, design, specifications, Validation: actual testing of the system module testing, user acceptance testing, regression testing. Prof. Shervin Shirmohammadi ELG

3 Reviews It is estimated that 30 to 50% of all project costs are consumed by rework. Rework is caused by either change or problems/mistakes. We need to catch these problems as early as possible. Reviews are one way to catch problems up front. Requires substantial initial investment, but returns very well at the end. A.k.a static testing Prof. Shervin Shirmohammadi ELG

4 Some figures (different studies): Finds 60 90% of all defects Between 2 to 10 times more efficient than actual testing Catches errors in 1/4 th of the time it takes to catch them with dynamic testing ROI = Savings/Costs = 4.48 (1997 data of 112 projects) Why Review? Prof. Shervin Shirmohammadi ELG

5 The Ripple Effect Ripple: affects multiple artifacts Scenario: no Reviews. Module Test SRS SDD Imp Prof. Shervin Shirmohammadi ELG

6 The Ripple Effect ( ) Scenario: with Reviews. SRS Module Test SDD Imp Prof. Shervin Shirmohammadi ELG

7 What to Review? Everything: Proposal Contract Schedule Budget SPMP SRS SDD Test Plan Component Specs Compatibility Prof. Shervin Shirmohammadi ELG

8 The Review Process Conduct reviews before exiting each phase, and also after completion of deliverables (milestone). Gather stakeholders: customer, architects, analysts, modelers, designers, Assign roles: Author Moderator Presenter Scriber Inspector (reviewer) Prof. Shervin Shirmohammadi ELG

9 Roles Author: must update the work product, usually the person who originally constructed it Moderator: ensures others perform their roles to the best of their abilities; should have no vested interest in the outcome Presenter: explains the functionality of each section of the work product Scriber: records all found problems, including suggestions, and assigns severity. Completes the report. Reviewer: each member of the team, including the above Why have both author and presenter? Prof. Shervin Shirmohammadi ELG

10 Review Attitude Review the product, not the person! Find problems, don t solve them. Concentrate on finding deviations from the requirements. Don t review more than 1-2 hours straight. Limit debate. Take responsibility for your review (also credit). Prepare at least a day in advance. Occasionally, review the review process itself. Prof. Shervin Shirmohammadi ELG

11 Dynamic Testing Dynamic as opposed to static testing (review), does actual hardware/software testing. Purpose: to fix as many errors/problems as possible. Can we test everything? It is important to have a test plan. Testing not to be confused with debugging Debugging: the activity of finding and fixing the error which is discovered by the test. Acceptance testing should not be done by developers Cognitive Dissonance: subconscious resistance to admitting one s own mistake Prof. Shervin Shirmohammadi ELG

12 Types of Tests Acceptance: is the final (not always) test based on the customer s specification. Alpha: when development is nearing completion. Beta: when development and testing are nearing completion. Black-box: based on requirements and functionality, not on internal knowledge of the design and code. White-box: based on internal logic (conditions, branches, ). Integration: combining all components. Regression: retesting after fixes and patches. Stress: to test load and performance. Prof. Shervin Shirmohammadi ELG

13 Test Plan A document describing the testing activities and approach. Describes: Objective Scope Approach Focus Identifies: items to be tested the actual test to be done (test cases) schedules personnel the level of acceptable risk Can start right away, by specifying how each requirement can be tested. Prof. Shervin Shirmohammadi ELG

14 Test Plan ( ) PM s responsibility is to ensure: validation criteria has been set correctly (with traceability) Major test phases have been identified correctly Main functions (specially the riskier ones) are tested first Test plan maps well to the project plan Resources are identified and available All required types of tests are included (stress, regression, integration, ) Prof. Shervin Shirmohammadi ELG

15 Example Test Case Table Test Case ID Test Case Description Setup Instructions Execution Instructions Expected Results Observed Results Pass/Fail/ Conditional Pass HW-1 HW-n SW-1 SW-2 Sensor Powers Up Sensor Initializes Sensor Sends Data SW-100 Mainline Program Initializes SW-101 Interrupt Received And Processed SW-200 Executable Loaded to the Target SW-250 Blue-Tooth module is Tx/Rx -ing Sys-1 Sys-2 Sys-3 Sys-20 SW-30 GUI comes up Data I/O is correct Command accepted, Actuator 1 moves Aircraft Sends GPS Position Heart Rate waveform is displayed Prof. Shervin Shirmohammadi ELG

16 When to Stop Testing? We already saw that we can t afford to test everything. So when to stop? Defect detection follows an S curve (also called Logistic Curve) as 1/DefectDensity approaches C, there is diminishing return: large number of tests are need to find smaller number of remaining defects Stop testing when: 1/DefectDensity reaches a certain value we can live with; or set up a testing budget and stop testing when that budget is exhausted Prof. Shervin Shirmohammadi ELG

17 Project Tracking Project management doesn t finish when the plan is ready it has just begun! PM must monitor the progress of the project, and control it. At any given point in time, PM must have an estimation of two sets of numbers: 1. how much time, cost (effort + others) and how what functions have been done up till now; 2. how much time and cost (effort + others) are needed and what fnctions are left to finish the project. TIME COST SCOPE Prof. Shervin Shirmohammadi ELG

18 PM Process: Typical Iteration Prof. Shervin Shirmohammadi ELG

19 Main Control Components Throughout the project, PM should monitor and control: Scope and functions Quality Risks Cost Schedule Prof. Shervin Shirmohammadi ELG

20 Scope Control Mainly controlled through Change Management Change is not free! Proposed changes are first reviewed! Impact of the change is analyzed: function, budget, duration. New test cases according to new changes Verification & validation of new changes A decision is made on whether or not to proceed with the change. Specially important during release time! After approval (by who?) the changes are applied. This avoids changes to affect the system in an unaccountable manner. If you re not managing changes, you re not managing at all! Prof. Shervin Shirmohammadi ELG

21 Change Request Change the look and feel of the user interface according to attached design. The current UI has given a bad score on the usability test. Prof. Shervin Shirmohammadi ELG

22 Risk Monitoring Maintain and update the top 10 list of risks, at least on a weekly basis. Take appropriate action accordingly. Prof. Shervin Shirmohammadi ELG

23 Cost Monitoring Budget is determined from the WBS, and the estimates. Using accounting data, PM can create a Cost Baseline. This is not just salaries; it also includes equipment, material, supplies, software, hardware, Incorporate change into the Cost. cost Cost Baseline Budget At Completion (BAC) time Prof. Shervin Shirmohammadi ELG

24 Schedule Monitoring Update estimates periodically. Developers give the PM new estimates periodically. Estimates should be in the form of time spent, time remaining. % complete is not a useful monitoring metric! Why? Incorporate changes into the schedule. Can use Earned Value Management techniques for forecasting and controlling the schedule and the cost. Prof. Shervin Shirmohammadi ELG

25 Earned Value Management A technique to measure real progress, not just the effort spent. Given the effort spent, it calculates what was accomplished with that effort, and how that translates into progress for the project as a whole. Dates back to the 1800s in industrial engineering. Prof. Shervin Shirmohammadi ELG

26 3 major components: EVM Components BCWS - Budgeted Cost of Work Scheduled Yearned value: it is the cost baseline. BCWP Budgeted Cost of Work performed Earned value: what the project really earned at a given point in time (how much work is done?) ACWP Actual Cost of Work Performed Burned value: what the project actually burned to get to this point (how much did the work done cost?) In the above, typically considers milestones that are 100% done, partial work is not counted. Prof. Shervin Shirmohammadi ELG

27 Measures CV: Cost Variance = BCWP ACWP SV: Schedule Variance = BCWP BCWS CPI: Cost Performance Index = BCWP/ACWP What does it mean? SPI: Schedule Performance Index = BCWP/BCWS What does it mean? CR: Critical Ratio = SPI CPI ECAC: Estimated Cost At Completion = BAC/CPI ETAC: Estimated Time At Completion = Original Time /SPI Prof. Shervin Shirmohammadi ELG

28 CR Controlling Prof. Shervin Shirmohammadi ELG

29 Example Assume today is May 1 st. Prof. Shervin Shirmohammadi ELG

30 Example ( ) Assume all cost figures in $K BCWS = = 80 BCWP = = 60 ACWP = = 50 CV = BCWP ACWP = = 10 good SV = BCWP BCWS = = -20 bad CPI = BCWP / ACWP = 60/50 = 1.2 good SPI = BCWP / BCWS = 60/80 = 0.75 bad CR = CPI SPI = = 0.9 OK Analysis? Prof. Shervin Shirmohammadi ELG

31 Post Performance Analysis One of the most important indications of maturity is Continuous Process Improvement. After a project is finished, we need to: measure the process, collecting process metrics analyze them devise strategies for improving the process feed them back into the next project These activities are known as Post-Performance Analysis (PPA), a.k.a. post-project analysis, or postmortem analysis. Note that this is can be done both during the project (iteration PPA) and/or at the end of the project (project PPA) In fact, the interim observations are the main metrics used in the project PPA as well. Why not wait until the very end? PPA activities will lead to a PPA Report, containing metrics summary and process improvement recommendations. Prof. Shervin Shirmohammadi ELG

32 Process Indicate, at the beginning of the project, what data should be collected, such as: Work products data: how many components, size, usage frequency, power consumption, Risks, their impact, how they were resolved, lessons learned Number and frequency of Changes to specs, plan, estimates, Original estimates vs. actual time for various components These should be gathered by individuals during the project, while memories are still fresh and reasons for decisions can be clearly remembered. At the end, assemble a team for the PPA meeting. Do the analysis and recommendations during the meeting In some cases more than one meeting might be required Publish the PPA Report. Prof. Shervin Shirmohammadi ELG

33 PPA Meeting Give people enough advanced notice to prepare. For the project PPA (final one), send out questionnaires for people to fill in. Keep the meeting short. Start off by mentioning the purpose. Very important for everyone to understand that this is not a finger-pointing session, but a process analysis and improvement session. Go over everyone s collected data, metrics, and suggestions. Categorize the data as needed Watch out for and avoid finger-pointing! It is not a personnel review meeting. Analyze and recommend improvements Publish the PPA report. Prof. Shervin Shirmohammadi ELG

34 PPA Questionnaire In addition to project specific questions, the following questions can be very valuable when asked from the individual s perspective: Identify things that were done right. Identify things that were done wrong. Identify any exceptional situations. How would you do things differently, if you were to do it all over again? Describe the most important 3 things that you could personally have done to improve the process. Other questions: Are you proud of the accomplishments of this project? Was the communication level effective? Was the project well-managed? Did we do the best that we can? Prof. Shervin Shirmohammadi ELG