Write a Blog >>
ICSE 2020
Wed 24 June - Thu 16 July 2020
Wed 8 Jul 2020 16:24 - 16:36 at Goguryeo - A11-Performance and Analysis Chair(s): Pooyan Jamshidi

Performance is one of the important aspects of software quality. Performance issues exist widely in software systems, and the process of fixing the performance issues is an essential step in the release cycle of software systems. Although performance testing is widely adopted in practice, it is still expensive and time-consuming. In particular, the performance testing is usually conducted after the system is built in a dedicated testing environment. The challenges of performance testing make it difficult to fit into the common DevOps process in software development. On the other hand, there exist a large number of tests readily available, that are executed regularly within the release pipeline during software development. In this paper, we perform an exploratory study to determine whether such readily available tests are capable of serving as performance tests. In particular, we would like to see whether the performance of these tests can demonstrate performance improvements obtained from fixing real-life performance issues. We collect 127 performance issues from Hadoop and Cassandra, and evaluate the performance of the readily available tests from the commits before and after the performance issue fixes. We find that most of the improvements from the fixes to performance issues can be demonstrated using the readily available tests in the release pipeline. However, only a very small portion of the tests can be used for demonstrating the improvements. By manually examining the tests, we identify eight reasons that a test cannot demonstrate performance improvements even though it covers the changed source code of the issue fix. Finally, we build random forest classifiers determining the important metrics influencing the readily available tests (not) being able to demonstrate performance improvements from issue fixes. We find that the test code itself and the source code covered by the test are important factors, while the factors related to the code changes in the performance issues fixes have a low importance. Practitioners may focus on designing and improving the tests, instead of fine-tuning tests for different performance issues fixes. Our findings can be used as a guideline for practitioners to reduce the amount of effort spent on leveraging and designing tests that run in the release pipeline for performance assurance activities.

Wed 8 Jul

Displayed time zone: (UTC) Coordinated Universal Time change

16:05 - 17:05
A11-Performance and AnalysisNew Ideas and Emerging Results / Journal First / Technical Papers / Demonstrations at Goguryeo
Chair(s): Pooyan Jamshidi University of South Carolina
16:05
3m
Talk
Nimbus: Improving the Developer Experience for Serverless ApplicationsDemo
Demonstrations
Robert Chatley Imperial College London, Thomas Allerton Starling Bank
Pre-print
16:08
8m
Talk
Testing with Fewer Resources: An Adaptive Approach to Performance-Aware Test Case GenerationJ1
Journal First
Giovanni Grano University of Zurich, Christoph Laaber University of Zurich, Annibale Panichella Delft University of Technology, Sebastiano Panichella Zurich University of Applied Sciences
Link to publication DOI Pre-print
16:16
8m
Talk
What's Wrong with My Benchmark Results? Studying Bad Practices in JMH BenchmarksJ1
Journal First
Diego Costa Concordia University, Canada, Cor-Paul Bezemer University of Alberta, Canada, Philipp Leitner Chalmers University of Technology & University of Gothenburg, Artur Andrzejak Heidelberg University
16:24
12m
Talk
Towards the Use of the Readily Available Tests from the Release Pipeline as Performance Tests. Are We There Yet?ACM SIGSOFT Distinguished Paper AwardsTechnical
Technical Papers
Zishuo Ding Concordia University, Canada, Jinfu Chen Concordia University, Canada, Weiyi Shang Concordia University
Pre-print
16:36
8m
Talk
ModGuard: Identifying Integrity & Confidentiality Violations in Java ModulesJ1
Journal First
Andreas Dann Paderborn University, Ben Hermann Paderborn University, Eric Bodden Heinz Nixdorf Institut, Paderborn University and Fraunhofer IEM
Link to publication DOI
16:44
6m
Talk
Program Debloating via Stochastic OptimizationNIER
New Ideas and Emerging Results
Qi Xin Georgia Institute of Technology, Myeongsoo Kim Georgia Institute of Technology, Qirun Zhang Georgia Institute of Technology, USA, Alessandro Orso Georgia Tech
16:50
8m
Talk
The ORIS Tool: Quantitative Evaluation of Non-Markovian SystemsJ1
Journal First
Marco Paolieri University of Southern California, Marco Biagi University of Florence, Laura Carnevali University of Florence, Enrico Vicario University of Florence