Code Coverage and Test Suite Effectiveness: Empirical Study with Real Bugs in Large Systems
During software maintenance, testing is a crucial activity to ensure the quality of program code as it evolves over time. With the increasing size and complexity of software, adequate software testing has become increasingly important. Code coverage is often used as a yardstick to gauge the comprehe...
Saved in:
Main Authors: | , , |
---|---|
Format: | text |
Language: | English |
Published: |
Institutional Knowledge at Singapore Management University
2015
|
Subjects: | |
Online Access: | https://ink.library.smu.edu.sg/sis_research/2974 https://ink.library.smu.edu.sg/context/sis_research/article/3974/viewcontent/P_ID_50992_saner15_coverage.pdf |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Institution: | Singapore Management University |
Language: | English |
id |
sg-smu-ink.sis_research-3974 |
---|---|
record_format |
dspace |
spelling |
sg-smu-ink.sis_research-39742018-07-13T04:33:36Z Code Coverage and Test Suite Effectiveness: Empirical Study with Real Bugs in Large Systems PAVNEET SINGH KOCHHAR, FERDIAN THUNG, David LO, During software maintenance, testing is a crucial activity to ensure the quality of program code as it evolves over time. With the increasing size and complexity of software, adequate software testing has become increasingly important. Code coverage is often used as a yardstick to gauge the comprehensiveness of test cases and the adequacy of testing. A test suite quality is often measured by the number of bugs it can find (aka. kill). Previous studies have analysed the quality of a test suite by its ability to kill mutants, i.e., artificially seeded faults. However, mutants do not necessarily represent real bugs. Moreover, many studies use small programs which increases the threat of the applicability of the results on large real-world systems. In this paper, we analyse two large software systems to measure the relationship of code coverage and its effectiveness in killing real bugs from the software systems. We use Randoop, a random test generation tool to generate test suites with varying levels of coverage and run them to analyse if the test suites can kill each of the real bugs or not. In this preliminary study, we have performed an experiment on 67 and 92 real bugs from Apache HTTPClient and Mozilla Rhino, respectively. Our experiment finds that there is indeed statistically significant correlation between code coverage and bug kill effectiveness. The strengths of the correlation, however, differ for the two software systems. For HTTPClient, the correlation is moderate for both statement and branch coverage. For Rhino, the correlation is strong for both statement and branch coverage. 2015-03-01T08:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/2974 info:doi/10.1109/SANER.2015.7081877 https://ink.library.smu.edu.sg/context/sis_research/article/3974/viewcontent/P_ID_50992_saner15_coverage.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Software Engineering |
institution |
Singapore Management University |
building |
SMU Libraries |
continent |
Asia |
country |
Singapore Singapore |
content_provider |
SMU Libraries |
collection |
InK@SMU |
language |
English |
topic |
Software Engineering |
spellingShingle |
Software Engineering PAVNEET SINGH KOCHHAR, FERDIAN THUNG, David LO, Code Coverage and Test Suite Effectiveness: Empirical Study with Real Bugs in Large Systems |
description |
During software maintenance, testing is a crucial activity to ensure the quality of program code as it evolves over time. With the increasing size and complexity of software, adequate software testing has become increasingly important. Code coverage is often used as a yardstick to gauge the comprehensiveness of test cases and the adequacy of testing. A test suite quality is often measured by the number of bugs it can find (aka. kill). Previous studies have analysed the quality of a test suite by its ability to kill mutants, i.e., artificially seeded faults. However, mutants do not necessarily represent real bugs. Moreover, many studies use small programs which increases the threat of the applicability of the results on large real-world systems. In this paper, we analyse two large software systems to measure the relationship of code coverage and its effectiveness in killing real bugs from the software systems. We use Randoop, a random test generation tool to generate test suites with varying levels of coverage and run them to analyse if the test suites can kill each of the real bugs or not. In this preliminary study, we have performed an experiment on 67 and 92 real bugs from Apache HTTPClient and Mozilla Rhino, respectively. Our experiment finds that there is indeed statistically significant correlation between code coverage and bug kill effectiveness. The strengths of the correlation, however, differ for the two software systems. For HTTPClient, the correlation is moderate for both statement and branch coverage. For Rhino, the correlation is strong for both statement and branch coverage. |
format |
text |
author |
PAVNEET SINGH KOCHHAR, FERDIAN THUNG, David LO, |
author_facet |
PAVNEET SINGH KOCHHAR, FERDIAN THUNG, David LO, |
author_sort |
PAVNEET SINGH KOCHHAR, |
title |
Code Coverage and Test Suite Effectiveness: Empirical Study with Real Bugs in Large Systems |
title_short |
Code Coverage and Test Suite Effectiveness: Empirical Study with Real Bugs in Large Systems |
title_full |
Code Coverage and Test Suite Effectiveness: Empirical Study with Real Bugs in Large Systems |
title_fullStr |
Code Coverage and Test Suite Effectiveness: Empirical Study with Real Bugs in Large Systems |
title_full_unstemmed |
Code Coverage and Test Suite Effectiveness: Empirical Study with Real Bugs in Large Systems |
title_sort |
code coverage and test suite effectiveness: empirical study with real bugs in large systems |
publisher |
Institutional Knowledge at Singapore Management University |
publishDate |
2015 |
url |
https://ink.library.smu.edu.sg/sis_research/2974 https://ink.library.smu.edu.sg/context/sis_research/article/3974/viewcontent/P_ID_50992_saner15_coverage.pdf |
_version_ |
1770572761969721344 |