Black-Box Testing Techniques II

Transcription

1 Black-Box Testing Techniques II Software Testing and Verification Lecture 5 Prepared by Stephen M. Thebaut, Ph.D. University of Florida

2 Cause-Effect Analysis Cause-Effect Analysis is a combinatorial approach that can be viewed as a logical extension of partition testing. It extends the idea of partitioning a multi-dimensional input space by providing a systematic means for generating test case templates to cover different combinations of input Causes resulting in output Effects.

3 Causes and Effects A CAUSE may be thought of as a distinct input condition, or an equivalence class of input conditions. An EFFECT may be thought of as a distinct output condition or change in program state.

4 Causes and Effects Causes and Effects are represented as Boolean variables. The logical relationships among them CAN (but need not) be represented as one or more Boolean graphs. Causes Л V Effects

5 C-E Analysis Process Steps 1. Identify Causes and Effects The most critical and usually the most difficult step Choose an appropriate level of abstraction. Divide and conquer as necessary. Effects may or may not be mutually exclusive.

6 C-E Analysis Process Steps (cont d) 2. Deduce Logical Relationships and Constraints Relationships take the form of conditionals and utilize the logical operators AND, OR, and NOT. Constraints describe relationships among Causes that allow for the identification of impossible combinations.

7 C-E Analysis Process Steps (cont d) 2. Deduce Logical Relationships and Constraints (cont d) Boolean graphs provide a convenient and economical way to visualize relationships and constraints.

8 C-E Analysis Process Steps (cont d) 3. Identify an appropriate Test Case Selection Strategy Determines the number and nature of Cause-combinations to be considered. Strategies can be designed to meet a variety of coverage requirements/ cost constraints.

9 C-E Analysis Process Steps (cont d) 4. Construct a Test Case Coverage Matrix Typically involves tracing through the Cause-Effect relationships to identify combinations of Causes resulting in each Effect according to the selection strategy chosen. This can be extremely tedious...

10 Question To what extent do you think CASE support might be applicable to each step in the process? For which steps do you think it might be most important?

11 Illustration of Step 1 (Identify Causes and Effects) The first input is a yes/no response to the question Do you reside within the city? The second input is gross pay for the year in question. A non-resident will pay 1% of the gross pay in city tax. Residents pay on the following scale: - If gross pay is no more than $30,000, the tax is 1%. - If gross pay is more than $30,000, but no more than $50,000, the tax is 5%. - If gross pay is more than $50,000, the tax is 15%.

12 Guidelines for identifying Causes and Effects Underline words or phrases in the specification that correspond to input/output conditions or changes in state. List each Cause and Effect. Assign a unique number to each (use different number ranges to differentiate Causes from Effects).

13 Guidelines for identifying Causes and Effects (cont d) The first input is a yes/no response to the question Do you reside within the city? The second input is gross pay for the year in question. A non-resident will pay 1% of the gross pay in city tax. Residents pay on the following scale: - If gross pay is no more than $30,000, the tax is 1%. - If gross pay is more than $30,000, but no more than $50,000, the tax is 5%. - If gross pay is more than $50,000, the tax is 15%.

14 Illustration of Step 1 (cont d) Ignoring, again, the unspecified responses to invalid inputs, we have: Causes: Effects: (1) Non-Resident (11) 1% tax (2) Resident (12) 5% tax (3) $0 Gross Pay $30K (13) 15% tax (4) $30K Gross Pay $50K (5) Gross Pay $50K

15 Illustration of Step 2 (Deduce Logical Relationships and Constraints) The first input is a yes/no response to the question Do you reside within the city? The second input is gross pay for the year in question. A non-resident will pay 1% of the gross pay in city tax. Residents pay on the following scale: - If gross pay is no more than $30,000, the tax is 1%. - If gross pay is more than $30,000, but no more than $50,000, the tax is 5%. - If gross pay is more than $50,000, the tax is 15%.

16 What are the constraints? Causes: Effects: (1) Non-Resident (11) 1% tax (2) Resident (12) 5% tax (3) $0 Gross Pay $30K (13) 15% tax (4) $30K Gross Pay $50K (5) Gross Pay $50K

17 Constraints deducible from spec, problem domain knowledge, etc. A. [(1) Л (2)] V [ (1) Л (2)] (i.e., one, and only one of (1) and (2) must be true.) B. [(3) Л (4) Л (5)] V [ (3) Л (4) Л (5)] V [ (3) Л (4) Л (5)] C. [(11) Л (12) Л (13)] V [ (11) Л (12) Л (13)] V [ (11) Л (12) Л (13)]

18 What are the logical relationships? The first input is a yes/no response to the question Do you reside within the city? The second input is gross pay for the year in question. A non-resident will pay 1% of the gross pay in city tax. Residents pay on the following scale: - If gross pay is no more than $30,000, the tax is 1%. - If gross pay is more than $30,000, but no more than $50,000, the tax is 5%. - If gross pay is more than $50,000, the tax is 15%.

19 Conditionals deducible from specification and constraints From the specification we have: (1) => (11) [(2) Л (3)] => (11) [(2) Л (4)] => (12) [(2) Л (5)] => (13)

20 Conditionals deducible from specification and constraints (cont d) Which, in light of the identified constraints, simplify to: [(1) V (3) => (11) [(2) Л (4)] => (12) [(2) Л (5)] => (13)

21 Boolean Graph Representation Non-Res (1) V (11) 1% tax O O [0,30K] (3) (30K,50K] (4) Res (2) Л (12) 5% tax O Л (13) 15% tax >50K (5)

22 Cause/Effect Constraints Exclusive Inclusive E at most one I at least one One & Only One Requires A O one and only one A => B B

23 Illustration of Step 3 (Identify Test Case Selection Strategy) Simple (but extreme) strategies: All Feasible Combinations of Cause Values (AFCCV) All Effects (AE) For the relationships depicted in our graph, how many test cases would be required to achieve AFCCV coverage? AE coverage?

24 AFCCV and AE Coverage Non-Res (1) V (11) 1% tax O O [0,30K] (3) (30K,50K] (4) Res (2) Л (12) 5% tax O Л (13) 15% tax >50K (5)

25 AFCCV and AE Coverage (cont d) AFCCV: There are 2 5 = 32 possible value combinations for all 5 Causes. For Causes (1) and (2), there are 2 feasible value combination pairs (due to the one and only one constraint): TF and FT. (cont d)

26 AFCCV and AE Coverage (cont d) AFCCV: (cont d) Similarly, for Causes (3), (4), and (5), there are 3 feasible value combination triples: TFF, FTF, and FFT. Thus, there are 2 X 3 = 6 feasible combinations of values for all 5 Causes, requiring a total of 6 test cases.

27 AFCCV and AE Coverage (cont d) AE: There are 3 mutually exclusive Effects. Thus, a different combination of Cause values is required for each Effect to evaluate to True. Therefore, 3 test cases are required. (cont d)

28 AFCCV and AE Coverage (cont d) AE: (cont d) In general, when there are N Effects, N or fewer test cases are required for AE Coverage. When the N Effects are mutually exclusive, all N test cases are required.

29 AFCCV and AE Coverage (cont d) Note that AE is analogous to partitioning an input space based solely on the specified outputs. AFCCV is analogous to associating a separate equivalence class with every (feasible) combination of the individual input classes (i.e., the brute-force approach). Question: do these strategies depend on the Cause-Effect relationships?

30 Another Test Case Selection Strategy REPEAT Select the next (initially, the first) Effect. Tracing back through the graph (right to left), find all feasible combinations of connected Cause values that result in the Effect being True. For each new such combination found: Determine values of all other Effects, and Enter values for each Cause and Effect in a new column of the test case coverage matrix. UNTIL each Effect has been selected.

31 What Should We Call this Strategy? How about: All Feasible Combinations of Connected Cause Values that Result in Each Effect being True (AFCCCVREET)?

32 Illustration of Step 4 (Construct a Test Case Coverage Matrix) For Strategy #3, this involves tracing through the Cause-Effect relationships to identify all feasible combinations of connected Causes resulting in each Effect being true

33 Applying Strategy #3 Non-Res (1) V (11) 1% tax O O [0,30K] (3) (30K,50K] (4) Res (2) Л (12) 5% tax O Л (13) 15% tax >50K (5)

34 Applying Strategy #3 Cause Value Combinations for Effect 11: (1) V (3) 1, 3 or 1, 3 or 1, 3 Cause Value Combinations for Effect 12: Cause Value Combinations for Effect 13:

35 Coverage Matrix TEST CASES CAUSES Non-Resident (1) T T F Resident (2) F F T $0 Gross Pay $30K (3) T F T $30K Gross Pay $50 (4) F F Gross Pay $50K (5) F F EFFECTS 1% tax (11) T T T 5% tax (12) F F F 15% tax (13) F F F don t care, subject to Cause constraint B

36 Applying Strategy #3 Non-Res (1) V (11) 1% tax O O [0,30K] (3) (30K,50K] (4) Res (2) Л (12) 5% tax O Л (13) 15% tax >50K (5)

37 Applying Strategy #3 Cause Value Combinations for Effect 11: (1) V (3) 1, 3 or 1, 3 or 1, 3 Cause Value Combinations for Effect 12: (4) Л (2) 2, 4 Cause Value Combinations for Effect 13:

38 Coverage Matrix (cont d) TEST CASES CAUSES Non-Resident (1) T T F F Resident (2) F F T T $0 Gross Pay $30K (3) T F T F $30K Gross Pay $50K (4) F F T Gross Pay $50K (5) F F F EFFECTS 1% tax (11) T T T F 5% tax (12) F F F T 15% tax (13) F F F F don t care, subject to Cause constraint B

39 Applying Strategy #3 Non-Res (1) V (11) 1% tax O O [0,30K] (3) (30K,50K] (4) Res (2) Л (12) 5% tax O Л (13) 15% tax >50K (5)

40 Applying Strategy #3 Cause Value Combinations for Effect 11: (1) V (3) 1, 3 or 1, 3 or 1, 3 Cause Value Combinations for Effect 12: (4) Л (2) 2, 4 Cause Value Combinations for Effect 13: (5) Л (2) 2, 5

41 Coverage Matrix (cont d) TEST CASES CAUSES Non-Resident (1) T T F F F Resident (2) F F T T T $0 Gross Pay $30K (3) T F T F F $30K Gross Pay $50K (4) F F T F Gross Pay $50K (5) F F F T EFFECTS 1% tax (11) T T T F F 5% tax (12) F F F T F 15% tax (13) F F F F T don t care, subject to Cause constraint B

42 Complete Coverage Matrix TEST CASES CAUSES Non-Resident (1) T T F F F Resident (2) F F T T T $0 Gross Pay $30K (3) T F T F F $30K Gross Pay $50K (4) F F T F Gross Pay $50K (5) F F F T EFFECTS 1% tax (11) T T T F F 5% tax (12) F F F T F 15% tax (13) F F F F T don t care, subject to Cause constraint B

43 Comparing Strategy #3 to AFCCV and AE How does Strategy #3 differ from AFCCV? For each Effect, only the connected Causes are considered. It is less conservative: Does not ensure that every feasible combination of Cause values will be covered. And thus does not ensure that every feasible combination of Effect values will be covered. (Relevant when Effects are not mutually exclusive.)

44 Comparing Strategy #3 to AFCCV and AE How does it differ from AE? It is more conservative (ALL feasible combinations of connected Cause values must be covered for each Effect).

45 Coming up in Black-Box Testing Techniques III We step-through another (somewhat more complex) example of Cause-Effect Analysis, Describe a test case design technique for exploring boundary conditions, and Consider a test case design strategy based on intuition and experience.

46 Black-Box Testing Techniques II Software Testing and Verification Lecture 5 Prepared by Stephen M. Thebaut, Ph.D. University of Florida