Code Coverage Analysis

Transcription

1 Code Coverage Analysis Submitted in partial fulfillment of the requirements for the award of the degree of Bachelor of Technology in Computer Science and Engineering Submitted by Akhil Saraswat 112CS0058 Under the guidance of Dr.D.P. Mohapatra Department of Computer Science and Engineering National Institute of Technology Rourkela Rourkela, Odisha, India May 2016

2 Department of Computer Science and Engineering National Institute of Technology Rourkela Rourkela, Odisha, India Supervisor s Certificate This is to certify that the work presented in the dissertation entitled Code Coverage Analysis submitted by Akhil Saraswat, Roll Number 112CS0058, is a record of original research carried out by him under my supervision and guidance in partial fulfillment of the requirements of the degree of Bachelor of Technology in Computer Science and Engineering. Neither this thesis nor any part of it has been submitted earlier for any degree or diploma to any institute or university in India or abroad. Date: Dr. D.P. Mohapatra

3 Declaration of Originality I, Akhil Saraswat, Roll Number 112CS0058 hereby declare that this dissertation entitled Code Coverage Analysis presents my original work carried out as a undergraduate student of NIT Rourkela and, to the best of my knowledge, contains no material previously published or written by another person, nor any material presented by me for the award of any degree or diploma of NIT Rourkela or any other institution. Any contribution made to this research by others, with whom I have worked at NIT Rourkela or elsewhere, is explicitly acknowledged in the dissertation. Works of other authors cited in this dissertation have been duly acknowledged under the sections Reference. I have also submitted my original research records to the scrutiny committee for evaluation of my dissertation. I am fully aware that in case of any noncompliance detected in future, the Senate of NIT Rourkela may withdraw the degree awarded to me on the basis of the present dissertation. Date: May 18, 2016 Akhil Saraswat

4 Acknowledgments I wish to express my profound sense of deepest gratitude to our guides and motivators Dr. S. K. Jena, Prof. Head, and Dr. D. P. Mahapatra, Asst Professor, Computer Science Engineering Department, National Institute of Technology, Rourkela for their valuable guidance, sympathy and co-operation and finally help for providing necessary facilities and sources during the entire period of this project. We wish to convey my sincere gratitude to the faculty of Computer Science Engineering Department who have enlightened me during my studies. The facilities and cooperation received from the technical staff of Computer Science Engg. Dept. is thankfully acknowledged. We express our thanks to all those who helped me in one way or other. Last, but not least, we would like to thank the authors of various research articles and book that referred to. Akhil Saraswat May 2016 National Institute of Technology Rourkela 3

5 Abstract Code coverage are tools of much importance for testing of software that helps the developers to understand which part of there software are tested by the given test suite. There are many software testing tools available in the market that provides testing metrics to check the quality of their code. These automatic testing tools have problems like different code coverage metric using different tools for the same program. For that reason we organized an experiment to get the performance of these coverage tools and to measure the significant difference in the code coverage done by these tools with respect to different criteria.our study shows that their is a large difference between the code coverage tools when compared by branch coverage as well as the affect of large code on the processing capability of these coverage tools.

6 Contents Acknowledgements 3 1 Introduction 1 2 Related works 3 3 Code coverage Tools and Metrics Code Coverage tools Metrics of Code Coverage Process of Coverage Analysis Experiments and Results Codes to be analysed Experimental Set-up Analysis of the Reports and Comparison Analysis with respect to Branch coverage Coverage analysis with Respect to Line Coverage Findings Conclusion And Future Works 17 References 19 i

7 List of Figures 3.1 Testing Coverage Analysis Process with JUnit Few Statement of CoffeeMaker(ScreenShot) Exception Class Screenshot( CoffeeMaker Program) ii

8 List of Tables 4.1 Coverage Report Of Coffee Make (Values are in Percentage) Coverage Report Of CrusieControl (Values are in Percentage) Coverage Report Of JfreeChart (Values are in Percentage) Definitions of Code Coverage Metrics iii

9 Chapter 1 Introduction Software testing is used to show the quality of software and to detect defects in the software as well as remove them.software testing is an important part for the development of the software. It gives the assurance to the developer that the software is working fine. The most important part of the software testing is to give the high coverage percentage of the given program.hence, code coverage is the criterion for software testing. Code coverage represents how much of the code has been covered by atleast one test case. As code coverage testing is also been interpreted as a sign of confirmation for developer in the software there are many code coverage tools present in the market to help the practitioners and end-users to know how the testing process works.so, their are many research on different parts of code coverage. Still only few of them are on coverage tools and out of them many have been investigated theoretically. Therefore, this topic of inconsistency of coverage by different code coverage tools did not get proper attention.till we know, no rearch is there that accounts the diffrence in the code coverare metric wise for diffrenet code coverage tools evaluated for same program. So our study is to address this problem to find the difference of the code coverage by different tools for the same coverage metric and explain the effects of these differences. We also try to explain why these differences occur. As the testing process is very costly so the study is to find which coverage tool is better for what type of purpose. Therefore, we use different open 1

10 source java software and find the coverage on each metric using different tools. We also study different metrics of code coverage: statement, branch, method and line coverage. 1. Identifying which metric is need to be collected during the experiment. 2. Using the code coverage tools which can be integrated with Junit and java eclipse IDE as well as support all the coverage metric we want to study. 3. Getting set of java programs having different lines of code and number of test cases. 4. Getting relevant data for analysis and and showing the results. Hence, we want to understand different code coverage tools through a controlled experiment. The results we present by this experiment shows the difference between the code coverage tools with respect to different coverage metrics especially with large programs. The paper is organized in the following order: Related works in this field i.e studies on different code coverage tools. (Chapter 2) Details about the code coverage analysis including the overview of different tools and metrics. (Chapter 3) Discussions about the experiment, results and our findings (Chapter 4) Future Works (Chapter 5) 2

11 Chapter 2 Related works Code coverage tools are use to generate coverage report which helps the developers to understand the defects in their software. These reports have the details with respect to coverage metrics and some other statistics about the program. The good side is that these tools provide coverage details and help the developer but on the other hand these tools make the testing process more difficult especially when it comes to programs having ten of thousands line of code. Therefore to understand the problem and assist the development by recommending proper testing tool software various studies have been conducted. This chapter shows the related works in the field: Their are several studies that have been conducted to compare between different code coverage tools. Youngbulut and Brykcynski [1][2] have done comprehensive study on different testing tools theoretically. This study have the details about metrics of different tools their reporting format and so other details. Yang [4] compared between 17 coverage tools with respect to coverage metric, test case generation and the coverage tools ability to customize the output file. They also shows which programming language support the instrumentation and have proper reporting formats. they also recommend some tools for better assistance however an actual experiment was not conducted. 3

12 Some researches are experiment based too but the implantation is for the study of individual coverage tools but the experiment s take too much time for large programs. So, Asaf [9] gives a way to show different of views on coverage data increasing the analysis for coverage. Finally, Kim [6] examined, empirically,diffrent ways to perform an experiment on large program. He compare block coverage with decision coverage. In this study he use large programs approx 19000k Line of code. His finding show ways to do analysis on large programs. 4

13 Chapter 3 Code coverage Tools and Metrics 3.1 Code Coverage tools There are different code coverage tools present, all of them have some merits and some demerits. These tools are used to help the developer in testing there software. they tell about which piece of code is covered by the given test cases and which is not as well as they tell about the detail of the line of code which are of no use and help in optimizing the working of the software. Each tool have there particular specification and functionality that differentiate them from others yest we use three tools for study which are used in the current development process and try to find which tool is suitable for which purpose. The tools that we use for analysis are: 1. Cobertura- It is an Eclipse Plugin that used to calculate code coverage by executing test cases. This software uses line and branch coverage to do coverage analysis. The source code that has been covered is shown in green while the red color implies that the part is not covered by the given test cases. Cobertura execute the code through junit framework and shows the coverage results. To use cobertura select coverage code session command under cover as. 2. Clover- This is the most widely used coverage tool for the reason that it function on the source code and do not evaluate those statements that are checked in earlier test run making it faster in execution. It also provide other statics like average complexity, total number of classes etc. To use clover for a project we have to select clover under the option enable for this project. 5

14 3. Jacoco- Java code coverage is a tool that instrument in java byte-code and is capable of find the coverage dynamically i.e on- the fly which make this tool more effective in evaluating code coverage than its earlier version emma. Due to its continuous updates it is used more widely than other coverage tools. Its is add as an coverage mode in Eclipse Ide. To use this tool on Eclipse we have to select command Coverage As. It shows green for fully covered statements, yellow for partially covered and red for not covered Metrics of Code Coverage Coverage analysis can be done in different ways producing coverage with respect to different metrics. these criteria helps in evaluating the total coverage of the software. The following criteria are used to find the coverage: Functional Coverage: Functional coverage involves invoking of all methods atleast once. Statement Coverage: In statement coverage each and every executable statement is invoked at least once. Branch Coverage: It invokes the control structure of for both true and false condition and finds the possible path that will lead for possible conditions. If(condition) { branch-1 } else { branch-2 } Relational operator coverage: Generally the relational operator are used swapped way by mistake, this leads to wrong output. therefore, 6

15 relational coverage analysis check for the incorrect operator and wrong boundary conditions. Path coverage: this include that all possible paths that can be taken within the program should be evaluated atleast once. A full path coverage involves nested conditions to be checked for both true and false.we can also use cyclomatic complexity to find the basic path in order to reduce number of test cases. if(cond1) { path1 } if(cond2) { path2 } In such case full path will be analyzed by setting the values of (cond1,cond2) as (0,0),(0,1),(1,0) and (1,1). Loop Coverage: It involves the percentage of loop that has been covered by the given set of test cases. Class Coverage: It checks whether every class have been invoked while executing the test suite atleast once Process of Coverage Analysis Generally, Analysis process consist of three main functionality: 1. Code instrumentation: Instrumentation can be done in two major ways. Either instrumentation can be done in the source code. This will insert the additional statement in the source code at the time of compilation which is known as static instrumentation or we can instrument the additional statement to get coverage into class files by 7

16 which the testing takes place at the time of execution or on-the-fly. But some of the code coverage tools are based on profiler monitoring which did not involves instrumentation. 2. Recording code coverage data: In this step all the coverage data has been gathered during execution. 3. Analysis of the data recorded: This step includes analysis of the coverage data with respect to different coverage metrics to give feed back to the developer about the software which helps the developer in modifying the software or change the test suite. Figure 3.1: Testing Coverage Analysis Process with JUnit 8

17 Chapter 4 Experiments and Results To Understand the difference between different code coverage tools, we conducted some experiments to find the answers of the following research questions: Q1: Does the coverage of a particular metric value differ when same code is tested by different coverage software? Q2: To understand the effectiveness of code coverage tools when program size increasesl. In this chapter we discuss about the codes used for analysis, their coverage results with respect to different coverage tools and the effectiveness of code coverage tools with large programs. 4.1 Codes to be analysed We use 3 programs of java of different number of line of code: 1. CoffeeMaker: Lines of Code:672 Classes: 9 Test-cases: Cruise Control: 9

18 Lines of Code:25566 Classes: 249 Test-cases: JfreeChart: Lines of Code: Classes: 639 Test-cases: 364 Figure 4.1: Few Statement of CoffeeMaker(ScreenShot) Experimental Set-up Configure the code coverage tool onto the source code. Using instrumentation process instrument the Junit test cases and source code then enable the coverage tool, which make the code coverage statements inserted into the source code to record different coverage metric. 10

19 Figure 4.2: Exception Class Screenshot( CoffeeMaker Program). the code coverage tool gather the data, analyses it and Generate the coverage report. This report consist of coverage details with respect to different coverage metrics like Branch coverage, line coverage, classes coverage. This report on different metrics can be different with respect to coverage tool. For instance jacoco shows line, branch, classes coverage and overall coverage while cobertura shows line and branch coverage To execute with the next coverage tool we need to remove all the instrumentation. Do sameby the next coverage tool. Summing up in total we are running three coverage tools on three different source code in order to get coverage report with respect to each metric Analysis of the Reports and Comparison In this chapter, we will analyses our finding to answer the two research questions. For that reason we are taking two hypothesis: H1: Different code coverage tools showing different coverage with respect to Branch Coverage. 11

20 Criterion Jacoco Cobertura Clover Line Coverage Branch Coverage Table 4.1: Coverage Report Of Coffee Make (Values are in Percentage) Criterion Jacoco Cobertura Clover Line Coverage Branch Coverage Table 4.2: Coverage Report Of CrusieControl (Values are in Percentage) H2: Different code coverage tools showing different values with respect to Line Coverage. Also, for our second research question H3:Effectiveness of code coverage tool decreases as the size of the program increases. These findings are shown below into two subsections Elaborating the results with each coverage metric. the table below shows the details of the code coverage report of different coverage tools with respect to coverage criterion. Criterion Jacoco Cobertura Clover Line Coverage Branch Coverage Table 4.3: Coverage Report Of JfreeChart (Values are in Percentage) 12

21 4.1.3 Analysis with respect to Branch coverage This section is to resolve our first hypothesis, evaluating difference between different coverage tools with respect to branch coverage. Small Program It is clearly visible that the code coverage for cobertura is significantly less as compare to jacoco and clover for the branch coverage. This shows that Cobertura provided least code coverage for the branch coverage in case of coffeemaker program. Medium Program Similar to the small program, in medium sized program like CrusieControl cobertura is showing less coverage as compare to jacoco and clover, while clover showing the highest coverage. Large program In case of large program like JfreeChart it is observed that jacoco is showing significant difference as compare to other code coverage tools, thus supporting our third hypothesis that the consistency of coverage metric of code coverage tool is affected when they calculate coverage value with respect to large coverage tools. Overall these results support our first hypothesis with respect to branch coverage level as well as it also support our third hypothesis on the effect of large programs on coverage value. 13

22 4.1.4 Coverage analysis with Respect to Line Coverage This sub section is related to our hypothesis that the code coverage values with the metric line coverage is different with respect to different Coverage tools. The tools we used for the observation purpose are cobertura clover and jacoco. Small Program In case of small program like coffeemaker we can see that Clover is showing the higest coverage while jacoco is showing the least coverage. Therefore, coverage difference exist between the tools for line coverage. Medium Program In case of CrusieControl program we can see that all the three tools are showing nearly same amount of coverage with slight difference but still clover is showing the highest coverage and the coverage percentage of all the three tools are different. Large Program In case of large program jacoco is showing too much high coverage percentage, thus supporting our third hypothesis as well we can see that clover is still showing percentage coverage more than cobertura. Thus also supporting our first hypothesis Findings By the above evidence we can say that our assumption about the difference between different coverage tools with respect to line coverage and branch coverage are clearly supported. As well as our third assumption about the impact of large programs on code coverage tools is also shown in the above experiment. We also believe 14

23 that the difference occur between the tools are due to the following reason: Instrumentation Different type of code analysis Size and Complexity of source code Definition used for coverage metrics 15

24 Table 4.4: Definitions of Code Coverage Metrics. 16

25 Chapter 5 Conclusion And Future Works In this paper, an experiment has been coducted to show the consistency of code coverage tools with respect to branch coverage and line coverage. According to our findings we have seen that the difference between branch coverage done by three tools are significant. While that of line coverage are slightly different. The purpose of these tools id to mesure the capabilities of diifferent techniques in the software industry and hence some tools might be mislead the devlopers and researchers. Our study makes the contribution towards the understanding of code coverage tools and their specifications. This study shows the difference between the code coverage tools which help the developers to decide which one is suitable for them. This study also shows affect ofsize of program on the the working of coverage tools. Finally we also try to reason out why these code coverage tools shows the variance while doing the same function. We hope our work will help the practitioners and developers toto get a good understanding of different coverage tools and help them choosing the right one for there purpose, so that they can produce better products while this work will enhance the software testing process. In future our study can be extended to evaluate the effectiveness of code coverage by studying it on large scale. One other way is to extend the current study on different object oriented languages with their coverage tools. Finally, Other coverage metrics can also be studied for different code coverage tools such as loop coverage, statement coverage, path coverage and vizual 17

26 effects provided by these tools by these tools. 18

27 References [1] C. Youngblut and B. Brykczynski, œanexaminationofselectedsoftwaretestingtools, IDAPaperP-2769,Inst.forDefenseAnalyses,Alexandria,Va.,Dec [2] C. Youngblut and B. Brykczynski, œanexaminationofselectedsoftwaretestingtools, Supp.IDAPaperP-2925,Inst.forDefenseAnalyses,Alexandria,Va.,Oct [3] M.R. Hoffmann, B. Janiczak, and E. Mandrikov, œeclemmahttp:// [4] Q. Yang, J. J. Li, and D. Weiss, œasurveyofcoveragebasedtestingtools,inast06.acm,2006,pp [5] M. Shahid and S. Ibrahim, œanevaluationoftestcoveragetoolsinsoftwaretesting, 2011InternationalConferenceonTelecommunicationTechnologyandApplicationsProc. ofcsitvol.5,2011. [6] Y. Woo Kim, œefficientuseofcodecoverageinlarge-scalesoftwaredevelopment, IBMCenterforAdvancedStudiesConference, Proceedingsofthe2003ConferenceoftheCentreforAdvancedStudiesonCollaborativeres Toronto,Ontario,Canada,2003,pp [7] Creating code coverage report, petrikainulainen.net/programming/maven/ creating-code-coverage-reports-for-unit-and-integration-tests-with-the-jacoc [8] J. A. Jones, M. J. Harrold, and J. Stasko, œvisualizationoftestinformationtoassistfaultlocalization, inicse02:24thinternationalconf.softwareengineering,2002, [9] S. Asaf, E. Marcus, and A. Ziv, "Definingcoverageviewstoimprovefunctionalcoverageanalysis, 19

28 Proceedingsofthe41stannualConferenceonDesignautomation, SanDiego,CA,USA,2004,pp [10] S. Elbaum, D. Gable, and G. Rothermel, œtheimpactofsoftwareevolutiononcodecoverage, InProceedingsofInternationalConferenceonSoftwareMaintenance, Nov.2001,pp [11] M. Gittens, K. Romanufa, D. Godwin, and J. Racicot, œallcodecoverageisnotcreatedequal: Acasestudyinprioritizedcodecoverage,CASCON2006,pp