Write a Blog >>
ICSE 2020
Wed 24 June - Thu 16 July 2020
Tue 7 Jul 2020 16:17 - 16:29 at Goguryeo - A5-Testing and Debugging 2 Chair(s): Marcelo d'Amorim

Developers typically rely on regression testing to know if their recent changes caused any regressions. However, flaky tests, which are tests that may pass or fail even without code changes, can provide misleading signals to developers. In recent years there has been considerable interest from industry and academia on flaky tests. However, there still aren’t many studies on the reproducibility, runtime, causes, and fixes of flaky tests, particularly in industry.

To fill such gap, we study the lifecycle of flaky tests on six large-scale industrial projects within AnonCompany. More specifically, we study the prevalence, reproducibility, characteristics, categories, and resolution of flaky tests. Our study on the prevalence of flaky tests informs us of the impact flaky tests have on the developers at AnonCompany. To understand a major challenge developers have for debugging and fixing flaky tests, we then proceed to study the reproducibility of flaky-test failures. Realizing the challenge developers face when debugging and fixing flaky tests, we proceed to characterize and categorize the flaky tests within AnonCompany. Lastly, we study how long developers take and how effective they are at fixing flaky tests to understand their resolution process of flaky tests. We believe that our study can help reaffirm findings from an industrial setting that previous studies on open-source projects have found and provide many new insights regarding the lifecycle of flaky tests that can help guide future research on the important topic of flaky tests.

Tue 7 Jul
Times are displayed in time zone: (UTC) Coordinated Universal Time change

16:05 - 17:05: A5-Testing and Debugging 2Paper Presentations / Demonstrations / New Ideas and Emerging Results / Technical Papers at Goguryeo
Chair(s): Marcelo d'AmorimFederal University of Pernambuco
16:05 - 16:17
Talk
Technical Papers
Hui GuoUniversity of California, Davis, Cindy Rubio-GonzálezUniversity of California, Davis
16:17 - 16:29
Talk
Technical Papers
Wing LamUniversity of Illinois at Urbana-Champaign, Kıvanç MuşluMicrosoft, Hitesh SajnaniMicrosoft , Suresh ThummalapentaMicrosoft
16:29 - 16:35
Talk
New Ideas and Emerging Results
Andriy MiranskyyRyerson University, Lei ZhangRyerson University, Javad DoliskaniRyerson University, Toronto, Canada
Pre-print
16:35 - 16:38
Talk
Demonstrations
Bruno LimaFaculty of Engineering, University of Porto and INESC TEC, João Pascoal FariaFaculty of Engineering, University of Porto and INESC TEC
16:38 - 16:50
Talk
Technical Papers
Valentin ManèsCSRC, KAIST, Soomin KimKAIST, Sang Kil ChaKAIST
16:50 - 16:53
Talk
Demonstrations
Richard RutledgeGeorgia Institute of Technology, Alessandro OrsoGeorgia Tech
16:53 - 16:59
Talk
New Ideas and Emerging Results
Carl HildebrandtUniversity of Virginia, Sebastian ElbaumUniversity of Virginia, USA, Nicola BezzoUniversity of Virginia