Agile Testing Survival Guide

Transcription

1 AN Agile Survival Guide How to build in quality & efficiency right from the start? Ingo Philipp.

2 Cycle Time Years Months Requirements Design Implementation Acceptance Deployment Waterfall

3 Cycle Time Years Months Months Weeks Requirements Design Implementation Deployment Inception Elaboration Construction Transition Rational Unified Process

4 Cycle Time Where we are right now just sucks. Patrick Debois, 2009 Years Months Dev Dev Dev Dev Dev Months Weeks Weeks Days Ops Ops OPS OOPS! Siloisation Dev. It compiles, it works on my machine and therefore it works. Ops. I don't care if it works on your machine! We are not shipping your machine! Agile

5 Cycle Time Years Months Collaborative Development Continuous Release & Deployment Months Weeks Continuous Integration Dev Ops Continuous Feedback Weeks Days Days Minutes Continuous Continuous Monitoring DevOps

6 Cycle Time delivery cycle time 0 Years Months Big Bang Months Weeks Incremental Weeks Days Iterative Agile Days Minutes DevOps Today

7 Cycle Time Future Years Months Months Weeks Fast Lane Weeks Days Days Minutes Businesses must continuously exploit digital technologies to both create new sources of customer value and increase operational agility in service of customers. Forrester, March 2014, The Future Of Business Is Digital delivery cycle time Digital Disruption 0 Internet of Things The Age of Robotics 0 complexity level Today

8 Start software delivery cycle End Present 80% 90% 67% 40% manual testing of all test automation is UI test automation average level of redundancy in enterprise test portfolios average risk coverage achieved in enterprise test portfolios Slow Lane It s me, a problem! 55% 56% 30% 50% of systems only partially accessible by Dev/Test of overall test effort goes into test case maintenance of bugs found in acceptance & production stage of manual testing goes into test data preparation & organization The first step to solve a problem is to accept that you have one. Albert Einstein, 1921

9 Future Start software delivery cycle End Fast Lane By compressing the delivery cycle, do problems just move closer to each other?

10 Future Start The devil is in the combination! End Fast Lane No, they mutually reinforce each other! Hence, the biggest strength of DevOps is not solving problems, but rather exposing buried problems.

11 Future harder isn t the answer, testing smarter is. Wolfgang Platz, 2016 Fast Lane Automation continuous testing is a must The Silver Bullet Is it really the only solution to all these problems?

12 No, it s about being efficient & effective. Is it just about speed? Future Optimize Find the shortest possible path. Automate doing right things Effectiveness B Best Way Leave your shoes behind and drive. Manage doing things right Keep the traffic lights green. A Efficiency Without data you re just another person with an opinion. W. Edwards Deming

13 Optimize Critical Limit Know what matters most. Do the right things. 40 % average risk coverage in enterprise test case portfolios Low Risk Right Way 20% Medium Risk 80% Test Cases High Risk Business Risk Coverage The time needed for testing is infinitely larger than the time available. Risk-Based

14 Optimize 1 N Securities Trading Sprint #N Right Way Capture Order Client Side Validation Check Eligibility Check Suitability Check Availability Market Side Validation Rectify Order Cancel Order N.1 As a role, I want N.2 As a role, I want N.3 As a role, I want N.4 As a role, I want N.5 As a role, I want N.6 As a role, I want N.7 As a role, I want N.8 As a role, I want Epics User Stories Product Backlog Sprint Planning User Stories tasks, spikes, chores Sprint Backlogs

15 Optimize 1 N Main Goal Intermediate Goals Securities Trading Sprint #N Right Way Capture Order Client Side Validation Check Eligibility Check Suitability Check Availability Market Side Validation Rectify Order Cancel Order N.1 As a role, I want N.2 As a role, I want N.3 As a role, I want N.4 As a role, I want N.5 As a role, I want N.6 As a role, I want N.7 As a role, I want N.8 As a role, I want Epics Product View Sprint Planning User Stories testable, measurable items Sprint Views

16 Optimize 1 N Main Goal Intermediate Goals Securities Trading Sprint #N Right Way Capture Order Client Side Validation Check Eligibility Check Suitability Check Availability Market Side Validation Rectify Order Cancel Order N.1 As a role, I want N.2 As a role, I want N.3 As a role, I want N.4 As a role, I want N.5 As a role, I want N.6 As a role, I want N.7 As a role, I want N.8 As a role, I want Governance Epics Product View Traceability Relationships User Stories testable, measurable items Sprint Views

17 Optimize 1 N Main Goal Intermediate Goals Securities Trading Sprint #N Right Way Capture Order Client Side Validation Check Eligibility Check Suitability Check Availability Market Side Validation Rectify Order Cancel Order N.1 As a role, I want N.2 As a role, I want N.3 As a role, I want N.4 As a role, I want N.5 As a role, I want N.6 As a role, I want N.7 As a role, I want N.8 As a role, I want Governance Product View Traceability Relationships Agile Team #1 Sprint Views

18 Optimize 1 N K Main Goal Securities Trading Capture Order Agile Team #K Right Way Client Side Validation Check Eligibility Check Suitability Check Availability Market Side Validation Rectify Order Cancel Order Agile Team #3 Agile Team #2 Governance Agile Team #1 Product View Sprint Views

19 Optimize Right Way Main Goal Securities Trading Capture Order Client Side Validation Check Eligibility Check Suitability Check Availability Market Side Validation Rectify Order Cancel Order Governance 80% 27% 24% 1.5% 1.5% 53% 10% 10% Business Risk Contribution Business Risk Coverage % Product Increment Theme Epic User Story Test Case Product View

20 Optimize Right Way Critical Limit Have the right test cases. Do the right things. % average redundancy level in 67 enterprise test case portfolios Don t get drown in the number of test cases. Test Case Design

21 Optimize It s time to get insured. Right Way Combinatorial Approaches

22 Create Quote 1 Age 2 Gender Maximum risk coverage. Equals exhaustive testing, and so leads to combinatorial explosion. Good 3 4 Payload [kg] Vehicle Type Exponential growth in test cases. Time, resource & cost intensive. Bad 5 Country 6 7 Fuel Type List Price Antieconomical approach. Test case count grows exponentially faster than risk coverage. Bad Mileage Per Year Engine Performance Start Date Test Objective is unknown. Practical significance only for small scale testing missions. Bad Insurance Sum Payment Option Damage Insurance Root cause analysis is a herculean task. Test objective is the entire test case portfolio. Bad Euro Protection Defense Insurance All Possible Combinations

23 Create Quote 1 Age 2 Gender Masters combinatorial explosion. Logarithmic growth in attributes & quadratic growth in instances. Good 3 4 Payload [kg] Vehicle Type Manifold test objective. Multiple pairs covered in a test case, i.e. no unique test goal. Bad 5 Country 6 7 Fuel Type List Price Numerous meaningless test cases. Manual clean-up without decrease in risk coverage is virtually impossible. Bad Mileage Per Year Engine Performance Start Date All pairs are equally important. not focused around most important criteria. Bad Insurance Sum Payment Option Damage Insurance Root cause analysis is a herculean task. Test cases are highly condensed, and so hard to maintain. Bad Euro Protection Defense Insurance All Pairwise Combinations

24 Create Quote 1 Age 2 Gender Minimal number of test cases. Guaranteed that each instance is used at least once. Good 3 4 Payload [kg] Vehicle Type Test Objective is unknown. Test case goals are most versatile and out of all reason. Bad 5 Country 6 7 Fuel Type List Price Enormous maintenance problems. Doing a lot with a little is just elusive. Bad Mileage Per Year Engine Performance Start Date False statements about risk contribution. Useless for risk-based approach on test case level. Bad Insurance Sum Payment Option Damage Insurance Root cause analysis is a herculean task. Maintainability, changeability & understandability are enemies. Bad Euro Protection Defense Insurance Each Choice Coverage Criterion

25 Nothing is perfect, life is messy, outcomes are uncertain, people are irrational, relationships are complex Create Quote 1 Age 2 Gender Creates only slightly more test cases. Linear increase in test case count up to about 95% risk coverage. Good 3 4 Payload [kg] Vehicle Type Assigns a unique & well-defined test objective. Strongly supports changeability, maintainability & understandability. Good 5 Country 6 7 Fuel Type List Price Enables to derive risk contribution. Best to apply risk-based approach on test case level. Good Mileage Per Year Engine Performance Start Date Makes root cause analysis an easy task. Test smarter, not harder and keep it simplistic instead of complex. Good Insurance Sum Payment Option Damage Insurance Assumes attribute independence. About 20% interdependent attributes, and so the logic partly breaks down. Bad Euro Protection Defense Insurance Linear Expansion

26 Nothing is perfect, life is messy, outcomes are uncertain, people are irrational, relationships are complex Create Quote Case #1 Case #2 Case #3 Case #4 Case #5 Case #6 Case #7 1 Age (24,59) >59 (24,59) (24,59) (24,59) (24,59) (24,59) 2 Gender Male Male Female Male Male Male Male 3 Payload [kg] [1,1000] [1,1000] [1,1000] >1000 [1,1000] [1,1000] [1,1000] 4 Vehicle Type Truck Truck Truck Truck Car Motorbike Trailer 5 Country 6 7 Fuel Type List Price Meaningless instance combinations Mileage Per Year Engine Performance Start Date Test Case Design Relations Missing instance combinations Insurance Sum Payment Option Damage Insurance Business Rules unlinked from test cases Euro Protection Defense Insurance Customized Linear Expansion

27 Optimize Right Way Would you believe in what I have just presented? Risk Coverage Optimization

28 Optimize Right Way Projects Analyzed 208 Project Lifetime On Average 8 Months Test Cases On Average 1712 Automation Level On Average 86% Distinct Sectors 7 Financial Consumer Energy Telecommunications Industrials Healthcare Materials *last status update: January 2016 Risk Coverage Optimization

29 Optimize Project #1 Right Way Critical Defects Risk Coverage Optimization

30 Optimize Project #1 #2 Right Way Critical Defects Risk Coverage Optimization

31 Optimize is like playing the lottery Project #1 #2 #3 #208 Randomness rules. Critical defects are just chance hits. Right Way No characteristics. Selective testing is impossible. Nothing to learn. No chance to improve. Critical Defects Risk Coverage Optimization

32 w 1 w 2 w 3 w 4 w i w N w i w i+1 Optimize High Risk Test Cases Low Risk 20% Test Cases 80% everything without really testing everything. Right Way Enable continuous delivery through continuous testing without drowning in test cases. Critical Defects Quod Erat Demonstrandum Risk Coverage Optimization

33 w 1 w 2 w 3 w 4 w i w N w i w i+1 Optimize High Risk Test Cases Low Risk 20% Test Cases 80% Right Way If it seems too good to be true, it probably isn t. Exploratory gets us out of that jam. Critical Defects Risk Coverage Optimization

34 Optimize application s universe Exploratory increase your testing cross section Risks Right Way Specification Based The Agile Future of Manual. Exploratory

35 Optimize Right Way Does this approach pay off? Risk Coverage Optimization

36 Testers Optimize Automate Integrate 11 Tests Value Based Test Cases Model Based Automation Continuous Integration Redundancy 75% Automation Level Distributed Smoke Automation Level Smoke Unknown 34 37% Risk 89% Remodeling traditional approaches 92% Minutes for greater 100% agility. Coverage 10 Weeks Effort Test Data 50% Weeks 48 Hours 8 Hours Test Case Count 7% 53% Execution Test Data Management Single Agent Regression Distributed Regression Risk Coverage Smoke