Code coverage and postrelease defects: A large-scale study on open source projects

Testing is a pivotal activity in ensuring the quality of software. Code coverage is a common metric used as a yardstick to measure the efficacy and adequacy of testing. However, does higher coverage actually lead to a decline in postrelease bugs? Do files that have higher test coverage actually have...

Full description

Saved in:
Bibliographic Details
Main Authors: KOCHHAR, Pavneet Singh, LO, David, LAWALL, Julia, NAGAPPAN, Nachiappan
Format: text
Language:English
Published: Institutional Knowledge at Singapore Management University 2017
Subjects:
Online Access:https://ink.library.smu.edu.sg/sis_research/3838
https://ink.library.smu.edu.sg/context/sis_research/article/4840/viewcontent/08031982.pdf
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Singapore Management University
Language: English
id sg-smu-ink.sis_research-4840
record_format dspace
spelling sg-smu-ink.sis_research-48402020-01-23T07:24:19Z Code coverage and postrelease defects: A large-scale study on open source projects KOCHHAR, Pavneet Singh LO, David LAWALL, Julia NAGAPPAN, Nachiappan Testing is a pivotal activity in ensuring the quality of software. Code coverage is a common metric used as a yardstick to measure the efficacy and adequacy of testing. However, does higher coverage actually lead to a decline in postrelease bugs? Do files that have higher test coverage actually have fewer bug reports? The direct relationship between code coverage and actual bug reports has not yet been analyzed via a comprehensive empirical study on real bugs. Past studies only involve a few software systems or artificially injected bugs (mutants). In this empirical study, we examine these questions in the context of open-source software projects based on their actual reported bugs. We analyze 100 large open-source Java projects and measure the code coverage of the test cases that come along with these projects. We collect real bugs logged in the issue tracking system after the release of the software and analyze the correlations between code coverage and these bugs. We also collect other metrics such as cyclomatic complexity and lines of code, which are used to normalize the number of bugs and coverage to correlate with other metrics as well as use these metrics in regression analysis. Our results show that coverage has an insignificant correlation with the number of bugs that are found after the release of the software at the project level, and no such correlation at the file level. 2017-09-01T07:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/3838 info:doi/10.1109/TR.2017.2727062 https://ink.library.smu.edu.sg/context/sis_research/article/4840/viewcontent/08031982.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Code coverage Computer bugs Correlation empirical study Measurement Open source software open-source postrelease defects software testing Sonar measurements Codes (symbols) Computer software Correlation methods Defects Information dissemination Measurements Open systems Program debugging Regression analysis Software engineering Software testing Code coverage Computer bugs Empirical studies Open sources Sonar measurements Open source software Programming Languages and Compilers Software Engineering
institution Singapore Management University
building SMU Libraries
continent Asia
country Singapore
Singapore
content_provider SMU Libraries
collection InK@SMU
language English
topic Code coverage
Computer bugs
Correlation empirical study
Measurement
Open source software
open-source postrelease defects
software testing
Sonar measurements
Codes (symbols)
Computer software
Correlation methods
Defects
Information dissemination
Measurements
Open systems
Program debugging
Regression analysis
Software engineering
Software testing
Code coverage
Computer bugs
Empirical studies
Open sources
Sonar measurements
Open source software
Programming Languages and Compilers
Software Engineering
spellingShingle Code coverage
Computer bugs
Correlation empirical study
Measurement
Open source software
open-source postrelease defects
software testing
Sonar measurements
Codes (symbols)
Computer software
Correlation methods
Defects
Information dissemination
Measurements
Open systems
Program debugging
Regression analysis
Software engineering
Software testing
Code coverage
Computer bugs
Empirical studies
Open sources
Sonar measurements
Open source software
Programming Languages and Compilers
Software Engineering
KOCHHAR, Pavneet Singh
LO, David
LAWALL, Julia
NAGAPPAN, Nachiappan
Code coverage and postrelease defects: A large-scale study on open source projects
description Testing is a pivotal activity in ensuring the quality of software. Code coverage is a common metric used as a yardstick to measure the efficacy and adequacy of testing. However, does higher coverage actually lead to a decline in postrelease bugs? Do files that have higher test coverage actually have fewer bug reports? The direct relationship between code coverage and actual bug reports has not yet been analyzed via a comprehensive empirical study on real bugs. Past studies only involve a few software systems or artificially injected bugs (mutants). In this empirical study, we examine these questions in the context of open-source software projects based on their actual reported bugs. We analyze 100 large open-source Java projects and measure the code coverage of the test cases that come along with these projects. We collect real bugs logged in the issue tracking system after the release of the software and analyze the correlations between code coverage and these bugs. We also collect other metrics such as cyclomatic complexity and lines of code, which are used to normalize the number of bugs and coverage to correlate with other metrics as well as use these metrics in regression analysis. Our results show that coverage has an insignificant correlation with the number of bugs that are found after the release of the software at the project level, and no such correlation at the file level.
format text
author KOCHHAR, Pavneet Singh
LO, David
LAWALL, Julia
NAGAPPAN, Nachiappan
author_facet KOCHHAR, Pavneet Singh
LO, David
LAWALL, Julia
NAGAPPAN, Nachiappan
author_sort KOCHHAR, Pavneet Singh
title Code coverage and postrelease defects: A large-scale study on open source projects
title_short Code coverage and postrelease defects: A large-scale study on open source projects
title_full Code coverage and postrelease defects: A large-scale study on open source projects
title_fullStr Code coverage and postrelease defects: A large-scale study on open source projects
title_full_unstemmed Code coverage and postrelease defects: A large-scale study on open source projects
title_sort code coverage and postrelease defects: a large-scale study on open source projects
publisher Institutional Knowledge at Singapore Management University
publishDate 2017
url https://ink.library.smu.edu.sg/sis_research/3838
https://ink.library.smu.edu.sg/context/sis_research/article/4840/viewcontent/08031982.pdf
_version_ 1770573804587712512