Write a Blog >>
ICSE 2020
Wed 24 June - Thu 16 July 2020
Wed 8 Jul 2020 16:24 - 16:36 at Goguryeo - A11-Performance and Analysis Chair(s): Pooyan Jamshidi

Performance is one of the important aspects of software quality. Performance issues exist widely in software systems, and the process of fixing the performance issues is an essential step in the release cycle of software systems. Although performance testing is widely adopted in practice, it is still expensive and time-consuming. In particular, the performance testing is usually conducted after the system is built in a dedicated testing environment. The challenges of performance testing make it difficult to fit into the common DevOps process in software development. On the other hand, there exist a large number of tests readily available, that are executed regularly within the release pipeline during software development. In this paper, we perform an exploratory study to determine whether such readily available tests are capable of serving as performance tests. In particular, we would like to see whether the performance of these tests can demonstrate performance improvements obtained from fixing real-life performance issues. We collect 127 performance issues from Hadoop and Cassandra, and evaluate the performance of the readily available tests from the commits before and after the performance issue fixes. We find that most of the improvements from the fixes to performance issues can be demonstrated using the readily available tests in the release pipeline. However, only a very small portion of the tests can be used for demonstrating the improvements. By manually examining the tests, we identify eight reasons that a test cannot demonstrate performance improvements even though it covers the changed source code of the issue fix. Finally, we build random forest classifiers determining the important metrics influencing the readily available tests (not) being able to demonstrate performance improvements from issue fixes. We find that the test code itself and the source code covered by the test are important factors, while the factors related to the code changes in the performance issues fixes have a low importance. Practitioners may focus on designing and improving the tests, instead of fine-tuning tests for different performance issues fixes. Our findings can be used as a guideline for practitioners to reduce the amount of effort spent on leveraging and designing tests that run in the release pipeline for performance assurance activities.

Wed 8 Jul
Times are displayed in time zone: (UTC) Coordinated Universal Time change

16:05 - 17:05: Paper Presentations - A11-Performance and Analysis at Goguryeo
Chair(s): Pooyan JamshidiUniversity of South Carolina
Demonstrations16:05 - 16:08
Robert ChatleyImperial College London, Thomas AllertonStarling Bank
icse-2020-Journal-First16:08 - 16:16
Giovanni GranoUniversity of Zurich, Christoph LaaberUniversity of Zurich, Annibale PanichellaDelft University of Technology, Sebastiano PanichellaZurich University of Applied Sciences
Link to publication DOI Pre-print
icse-2020-Journal-First16:16 - 16:24
Diego CostaConcordia University, Canada, Cor-Paul BezemerUniversity of Alberta, Canada, Philipp LeitnerChalmers University of Technology & University of Gothenburg, Artur AndrzejakHeidelberg University
icse-2020-papers16:24 - 16:36
Zishuo DingConcordia University, Canada, Jinfu ChenConcordia University, Canada, Weiyi ShangConcordia University
icse-2020-Journal-First16:36 - 16:44
Andreas DannPaderborn University, Ben HermannPaderborn University, Eric BoddenHeinz Nixdorf Institut, Paderborn University and Fraunhofer IEM
Link to publication DOI
icse-2020-New-Ideas-and-Emerging-Results16:44 - 16:50
Qi XinGeorgia Institute of Technology, Myeongsoo KimGeorgia Institute of Technology, Qirun ZhangGeorgia Institute of Technology, USA, Alessandro OrsoGeorgia Tech
icse-2020-Journal-First16:50 - 16:58
Marco PaolieriUniversity of Southern California, Marco BiagiUniversity of Florence, Laura CarnevaliUniversity of Florence, Enrico VicarioUniversity of Florence