Automated techniques for improving the quality of existing test suites
Date
2017
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
University of Delaware
Abstract
Testing is playing a crucial and fundamental role in modern software
development. Although software tests are conceptually simple---they are
composed of two primary parts: inputs that are used to execute the
program under test and an oracle that is used to verify that the
execution induced by the inputs produces the expected results---they are often
difficult to write in practice. The software engineering research community
provided many techniques that can help developers determine whether
they have written effective and efficient tests, including various coverage
metrics which have been widely adopted. While they have been proven successful
in practice, many follow-up studies show that there is still a lot to improve
for the test quality measurements, including false negatives in alarming
the lack of tests and constructive suggestions for improvements. ☐ This dissertation focuses on improving the quality of existing test suites
based on interpretations on test inputs and test oracles. If a test oracle
checks the values which the test developers did not or can not control, it
would make the test brittle. I developed a novel technique based on dynamic
tainting which can identify the values that can make tests brittle. An
empirical study on real-world applications shows that the technique can reveal
brittle assertions and the values that cause the brittleness within reasonable
cost. I also developed two techniques that can identify insufficiently tested
code by interpreting traditional coverage information. One is a new approach
based on the concepts of direct coverage and indirect coverage. The other is a
new approach to discover incidentally tested code. Both techniques have shown
efficiency and effectiveness in the empirical studies on real-world
applications.