Software Quality Overview: Costs and Benefits

Transcription

1 Software Quality Overview: Costs and Benefits Duvan Luong Ph.D. Operational Excellence Networks 4/2/2012 Contents copyright Operational Excellence Networks

2 A view of the business profits 2

3 What are Quality Costs? Prevention Costs: Costs incurred to prevent poor quality (help to minimize failure and appraisal costs New product review Quality planning Quality practices training Suppliers survey Process review Quality improvement team Etc. Appraisal Costs: Costs incurred to determine the degree of conformance to quality requirements/standards. Design review Code inspection Engineering testing Independent testing (PV and other testing) Test automation Process assessment/audits Etc. Internal Failure Costs: Costs associated with defects found before product ship to customers. Bug fixes Regression testing Wasted due to missing code delivery time Engineers time Testers time Tech Pubs time Marketing time Cost of late shipment Opportunity lost due to late shipment Etc. External Failure Costs: Costs associated with defects found after customer receives the product Support Investigation and process of complaints Recall Fix/retest for bug-fix releases Shipping of update product Etc. 3

4 Software Quality Costs Software Quality Cost = Cost of defects prevention (Informal reviews/inspections, documentation,...) + Cost of defects detection (Formal reviews/inspections, test planning, preparation, execution, analysis, problems identification, classification...) + Cost of defects correction (Problems replication, correction, re-validation,...) + Cost of lost opportunities (Customer dis-satisfaction, time, sales,...) 4

5 Cost of Software Test & Maintenance Cost of Process Standards and Improvement (in % of R&D budget) Cost of Testing including Review (in % of R&D budget) Cost of Engineering Rework/Maintenance (in % of R&D budget) Product Quality (Customer critical defects/kloc) Defect detection efficiency (% of critical defect escaped to customer) Low Quality Company Industry typical Quality Company Better Quality Company (Space Shuttle) 0% 1-2% 3-5% (5%) 10-13% (No Review) 15-17% (Some Review) >20% (200%) (Full Review) 40-70% 30-40% 20-30% (<20%) >5 defect/kloc 2-5 defect/kloc <2 (.01) defect/kloc > 10% 3-10% <3% (.001) Author private data 5

6 6 Industry Defect Statistics Design 45 % 20 % Requirements 35 % Code & Bad Fixes % of defects introduced by activity Reference: Caper Jones, "Applied Software Measurements", McGraw Hill, 1991 Reference: "CASE Make Strides toward Automated Software Development", Computer Design, January 1987

7 7 The High Cost of Defects 100 Relative Cost to Repair 1 Specif. /Req Design Coding Unit Test 20 Acceptance Test 50 Maintenance Phase in which Defect Is Discovered/Repaired

8 Sample Cost of Quality Metrics Costs for Analyzing, Fixing, Testing and follow-through to resolve a defect: 80 hours for a customer found problem, 20 hours for a system test found problem, 10 hours for an Engineering test found problem Company Critical and Serious Defects Found by Quarter Customer Found System Test Found R&D Found Total Incoming Q04 2Q04 3Q04 4Q04 1Q05 2Q05 3Q05 4Q05 1Q06 Engineer Year Company Efforts for Resolving Defects by Quarter Customer Found Defect Costs System Test Found Defect Costs R&D Found Defect Costs Q04 2Q04 3Q04 4Q04 1Q05 2Q05 3Q05 4Q05 1Q06

9 Defect Removal Efficiency Errors/kloc Remaining Errors/ Kloc Removal Inspections Unit Test Integration System Test Op Demo Operations Efficiency 50% 20% 50% 40% 20% Typical Error Removal Profile Reference: M.Dyer, "The Cleanroom Approach to Quality Software development", John Wiley & Sons, Inc.,

10 Defect Removal Efficiency Removal Steps Personal Checking of design or documents Informal group reviews Formal design inspections Formal code inspections Modeling or prototyping Personal desk checking of code Unit Testing (single module) Function Testing (related module) Integration Testing (complete system) Field Testing (live data) Cumulative efficiency of complete series Lowest Efficiency Average Efficiency Highest Efficiency 15% 35% 70% 30% 40% 60% 35% 55% 75% 30% 60% 70% 35% 65% 80% 20% 40% 60% 10% 25% 50% 20% 35% 55% 25% 45% 60% 35% 50% 65% 95% 99% 99% Reference: Caper Jones, "Applied Software Measurements", McGraw Hill,

11 Defect Detection Costs Joint ApplicationDevelopment Prototyping Requirement review Design inspection Code inspection Unit test Function test System Test Field Test Preparation hours per Test execution hours per KLOC KLOC Reference: Caper Jones, "Applied Software Measurements", McGraw Hill, 1991 Note: normalized with 1Function point = 100 line of C code 11

12 Device Driver Firmware Defect fixing Costs Design defect caught in an Inspection/Review < $50 Implementation defect caught in Unit Test $50 - $100 Design defect caught in Unit Test $100 - $1000 Design/implementation defect caught in system (Lab) test $500 - $8K Design/implementation defect caught in system (QA) test $2K - $16K Design/implementation defect caught in Alpha/Beta test $10K - $1M Design/implementation defect caught after Ship Release $100K - $10M Major defect found on mature flagship product > $200K 12

13 Industry Software Defect fixing Costs Defect caught in JAD $100 (1-2 hr = $100) Prototyping $100 (1-2 hr) Requirement review $100 (1-2 hr) Design inspection $150 (1.5-4 hr) Code inspection $150 (1.5-4 hr) Unit test $250 (2.5-8 hr) Function test $500 ( hr) System test $1000 (10-20 hr) Field test $1000 (10-20 hr) Customer Use $2000 ( hr) Reference: Caper Jones, "Applied Software Measurements", McGraw Hill, 1991 Private data: IBM customer defect fixing cost 82 hours, Hewlett packard 80 hours 13

14 Software Development Efforts % of Dev. Effort System Test Integration Test Unit Test Code and Debug Detailed Design Architecture 2K 8K 32K 128K 512K Project size in Lines of Code Reference: Steve McConnell, "Code complete", Microsoft Press,

15 Calculating the I in the ROI for software 15

16 Data for the Calculation of Software Quality Cost per KLOC Number of defects/kloc found throughout the life of the release Defect introduction rate by lifecycle phase Defect removal rate by lifecycle phase Cost of Quality effort/kloc by activity (review, test, etc.) Cost of defect fixing by lifecycle phase 16

17 Data for the Calculation of Development Quality Cost per KLOC Number of defects/kloc found throughout the life of the release: 30 defects/kloc Defect introduction rate by lifecycle phase: Reqs Design Code QA Field 20% 45% 35% Defect removal efficiency by lifecycle quality activity: Req review Design Review Unit test QA test Field use and test 50% 50% 30% 60% 100% Defect detection costs/kloc by lifecycle phase: Req Design Code QA 4.0 hrs 6.5 hrs 7.5 hrs 15 hrs 10 hrs Fixing costs/defect by lifecycle phase: Req Design Code QA Field Field 1.0 hrs 1.5 hrs 2.5 hrs 10 hrs 20 hrs 17

18 Development Quality Cost per KLOC w/o Unit Testing and Review Defects from previous phase Defects introduced in this phase Defects removed in this phase Defects remained at phase end Detection costs Fixing costs Total Req Design Code QA Field hrs 10 hrs hrs (18X10) 240 hrs (12x20) 195 hrs 250 hrs Total 25 hrs 420 hrs 445 hrs Note: Detection cost is only a small fraction of Fixing cost 18

19 Development Quality Cost per KLOC with Unit Testing, no Review Req Design Code QA Field Total Defects from previous phase Defects introduced in this phase Defects removed in this phase Defects remained at phase end Detection costs Fixing costs hrs 15 hrs 10 hrs hrs (9X2.5) 130 hrs (13X10) 160 hrs (8x20) 32.5 hrs hrs Total 30.0 hrs 145 hrs 170 hrs 345 hrs Note: 7% investment in Unit Testing results in 22% improvement in overall costs per KLOC: 345 hrs vs. 445 hrs 19

20 Development Quality Cost per KLOC with Unit Testing & Design Review Defects from previous phase Defects introduced in this phase Defects removed in this phase Defects remained at phase end Detection costs Fixing costs Req Design Code QA hrs 7.5 hrs 15 hrs 10 hrs 0 15 hrs (10X1.5) 15.0 hrs (6X2.5) 80 hrs (8X10) Field 120 hrs (6x20) Total 39 hrs 230 hrs Total 21.5 hrs 22.5 hrs 95 hrs 130 hrs 269 hrs Note: 10% investment in Design Review and Unit Testing results in 40% improvement in overall costs per KLOC: 269 hrs vs. 445 hrs 20

21 Development Quality Cost per KLOC with Unit Testing & Reqs+Design Review Defects from previous phase Defects introduced in this phase Defects removed in this phase Defects remained at phase end Detection costs Fixing costs Req Design Code QA hrs 6.5 hrs 7.5 hrs 15 hrs 10 hrs 3.0 hrs (3x1.0) 12 hrs (8X1.5) 15 hrs (6X2.5) 80 hrs (8X10) Field 100 hrs (5x20) Total 43 hrs 210 hrs Total 7.0 hrs 18.5 hrs 22.5 hrs 95 hrs 110 hrs 253 hrs Note: 11% investment in Reqs & Design Review and Unit Testing results in 43% improvement in overall costs per KLOC: 253 hrs vs. 445 hrs 21

22 The values of early focus on Quality Engineer's ownership of product code Saving in defect fixing cost Early visibility into development process effectiveness and product quality High potential for "cycle time reduction" Reduction in the cost of lost opportunities 22