Write a Blog >>
ICSE 2020
Wed 24 June - Thu 16 July 2020
Wed 8 Jul 2020 16:24 - 16:36 at Goguryeo - A11-Performance and Analysis Chair(s): Pooyan Jamshidi

Performance is one of the important aspects of software quality. Performance issues exist widely in software systems, and the process of fixing the performance issues is an essential step in the release cycle of software systems. Although performance testing is widely adopted in practice, it is still expensive and time-consuming. In particular, the performance testing is usually conducted after the system is built in a dedicated testing environment. The challenges of performance testing make it difficult to fit into the common DevOps process in software development. On the other hand, there exist a large number of tests readily available, that are executed regularly within the release pipeline during software development. In this paper, we perform an exploratory study to determine whether such readily available tests are capable of serving as performance tests. In particular, we would like to see whether the performance of these tests can demonstrate performance improvements obtained from fixing real-life performance issues. We collect 127 performance issues from Hadoop and Cassandra, and evaluate the performance of the readily available tests from the commits before and after the performance issue fixes. We find that most of the improvements from the fixes to performance issues can be demonstrated using the readily available tests in the release pipeline. However, only a very small portion of the tests can be used for demonstrating the improvements. By manually examining the tests, we identify eight reasons that a test cannot demonstrate performance improvements even though it covers the changed source code of the issue fix. Finally, we build random forest classifiers determining the important metrics influencing the readily available tests (not) being able to demonstrate performance improvements from issue fixes. We find that the test code itself and the source code covered by the test are important factors, while the factors related to the code changes in the performance issues fixes have a low importance. Practitioners may focus on designing and improving the tests, instead of fine-tuning tests for different performance issues fixes. Our findings can be used as a guideline for practitioners to reduce the amount of effort spent on leveraging and designing tests that run in the release pipeline for performance assurance activities.

Wed 8 Jul
Times are displayed in time zone: (UTC) Coordinated Universal Time change

16:05 - 17:05: A11-Performance and AnalysisPaper Presentations / New Ideas and Emerging Results / Journal First / Technical Papers / Demonstrations at Goguryeo
Chair(s): Pooyan JamshidiUniversity of South Carolina
16:05 - 16:08
Nimbus: Improving the Developer Experience for Serverless ApplicationsDemo
Robert ChatleyImperial College London, Thomas AllertonStarling Bank
16:08 - 16:16
Testing with Fewer Resources: An Adaptive Approach to Performance-Aware Test Case GenerationJ1
Journal First
Giovanni GranoUniversity of Zurich, Christoph LaaberUniversity of Zurich, Annibale PanichellaDelft University of Technology, Sebastiano PanichellaZurich University of Applied Sciences
Link to publication DOI Pre-print
16:16 - 16:24
What's Wrong with My Benchmark Results? Studying Bad Practices in JMH BenchmarksJ1
Journal First
Diego CostaConcordia University, Canada, Cor-Paul BezemerUniversity of Alberta, Canada, Philipp LeitnerChalmers University of Technology & University of Gothenburg, Artur AndrzejakHeidelberg University
16:24 - 16:36
Towards the Use of the Readily Available Tests from the Release Pipeline as Performance Tests. Are We There Yet?ACM SIGSOFT Distinguished Paper AwardsTechnical
Technical Papers
Zishuo DingConcordia University, Canada, Jinfu ChenConcordia University, Canada, Weiyi ShangConcordia University
16:36 - 16:44
ModGuard: Identifying Integrity & Confidentiality Violations in Java ModulesJ1
Journal First
Andreas DannPaderborn University, Ben HermannPaderborn University, Eric BoddenHeinz Nixdorf Institut, Paderborn University and Fraunhofer IEM
Link to publication DOI
16:44 - 16:50
Program Debloating via Stochastic OptimizationNIER
New Ideas and Emerging Results
Qi XinGeorgia Institute of Technology, Myeongsoo KimGeorgia Institute of Technology, Qirun ZhangGeorgia Institute of Technology, USA, Alessandro OrsoGeorgia Tech
16:50 - 16:58
The ORIS Tool: Quantitative Evaluation of Non-Markovian SystemsJ1
Journal First
Marco PaolieriUniversity of Southern California, Marco BiagiUniversity of Florence, Laura CarnevaliUniversity of Florence, Enrico VicarioUniversity of Florence