Write a Blog >>
ICSE 2020
Wed 24 June - Thu 16 July 2020
Tue 7 Jul 2020 16:17 - 16:29 at Goguryeo - A5-Testing and Debugging 2 Chair(s): Marcelo d'Amorim

Developers typically rely on regression testing to know if their recent changes caused any regressions. However, flaky tests, which are tests that may pass or fail even without code changes, can provide misleading signals to developers. In recent years there has been considerable interest from industry and academia on flaky tests. However, there still aren’t many studies on the reproducibility, runtime, causes, and fixes of flaky tests, particularly in industry.

To fill such gap, we study the lifecycle of flaky tests on six large-scale industrial projects within AnonCompany. More specifically, we study the prevalence, reproducibility, characteristics, categories, and resolution of flaky tests. Our study on the prevalence of flaky tests informs us of the impact flaky tests have on the developers at AnonCompany. To understand a major challenge developers have for debugging and fixing flaky tests, we then proceed to study the reproducibility of flaky-test failures. Realizing the challenge developers face when debugging and fixing flaky tests, we proceed to characterize and categorize the flaky tests within AnonCompany. Lastly, we study how long developers take and how effective they are at fixing flaky tests to understand their resolution process of flaky tests. We believe that our study can help reaffirm findings from an industrial setting that previous studies on open-source projects have found and provide many new insights regarding the lifecycle of flaky tests that can help guide future research on the important topic of flaky tests.

Tue 7 Jul

Displayed time zone: (UTC) Coordinated Universal Time change

16:05 - 17:05
A5-Testing and Debugging 2Demonstrations / New Ideas and Emerging Results / Technical Papers at Goguryeo
Chair(s): Marcelo d'Amorim Federal University of Pernambuco
16:05
12m
Talk
Efficient Generation of Error-Inducing Floating-Point Inputs via Symbolic ExecutionArtifact ReusableTechnicalArtifact Available
Technical Papers
Hui Guo University of California, Davis, Cindy Rubio-González University of California, Davis
16:17
12m
Talk
A Study on the Lifecycle of Flaky TestsTechnical
Technical Papers
Wing Lam University of Illinois at Urbana-Champaign, Kıvanç Muşlu Microsoft, Hitesh Sajnani Microsoft , Suresh Thummalapenta Microsoft
16:29
6m
Talk
Is Your Quantum Program Bug-Free?New Ideas and Emerging Results Distinguished Paper AwardsNIER
New Ideas and Emerging Results
Andriy Miranskyy Ryerson University, Lei Zhang Ryerson University, Javad Doliskani Ryerson University, Toronto, Canada
Pre-print
16:35
3m
Talk
DCO Analyzer: Local Controllability and Observability Analysis and Enforcement of Distributed Test ScenariosDemo
Demonstrations
Bruno Lima Faculty of Engineering, University of Porto and INESC TEC, João Pascoal Faria Faculty of Engineering, University of Porto and INESC TEC
16:38
12m
Talk
Ankou: Guiding Grey-box Fuzzing towards Combinatorial DifferenceArtifact ReusableTechnicalArtifact Available
Technical Papers
Valentin Manès CSRC, KAIST, Soomin Kim KAIST, Sang Kil Cha KAIST
16:50
3m
Talk
PG-KLEE: Trading Soundness for CoverageDemo
Demonstrations
Richard Rutledge Georgia Institute of Technology, Alessandro Orso Georgia Tech
16:53
6m
Talk
Blending Kinematic and Software Models for Tighter Reachability AnalysisNIER
New Ideas and Emerging Results
Carl Hildebrandt University of Virginia, Sebastian Elbaum University of Virginia, USA, Nicola Bezzo University of Virginia