Write a Blog >>
ICSE 2020
Wed 24 June - Thu 16 July 2020
Thu 9 Jul 2020 00:24 - 00:36 at Goguryeo - P14-Testing Chair(s): Shin Yoo

Test flakiness—inability to reliably repeat a test’s Pass/Fail outcome—continues to be a significant problem in Industry, adversely impacting continuous integration and test pipelines. Completely eliminating flaky tests is not a realistic option as a significant fraction of system tests (typically non-hermetic) for services-based implementations exhibit some level of flakiness. In this paper, we view the flakiness of a test as a rankable value, which we quantify, track and assign a confidence. We develop two ways to model flakiness, capturing the randomness of test results via entropy, and the temporal variation via flipRate, and aggregating these over time. We have implemented our flakiness scoring service and discuss how its adoption has impacted test suites of two large services at Apple. We show how flakiness is distributed across the tests in these services, including typical score ranges and outliers. The flakiness scores are used to monitor and detect changes in flakiness trends. Evaluation results demonstrate near perfect accuracy in ranking, identification and alignment with human interpretation. The scores were used to identify 2 causes of flakiness in the dataset evaluated, which have been confirmed, and where fixes have been implemented or are underway. Our models reduced flakiness by 44% with less than 1% loss in fault detection.

Thu 9 Jul
Times are displayed in time zone: (UTC) Coordinated Universal Time change

icse-2020-paper-presentations
00:00 - 01:00: Paper Presentations - P14-Testing at Goguryeo
Chair(s): Shin YooKorea Advanced Institute of Science and Technology
icse-2020-papers00:00 - 00:12
Talk
Dehai ZhaoAustralian National University, Zhenchang XingAustralia National University, Chunyang ChenMonash University, Xiwei XuData 61, Liming ZhuCSIRO's Data61 and UNSW, Guoqiang LiShanghai Jiao Tong University, Jinshui WangSchool of Information Science and Engineering, Fujian University of Technology, Fuzhou, China
icse-2020-papers00:12 - 00:24
Talk
Xiang GaoNational University of Singapore, Singapore, Ripon SahaFujitsu Laboratories of America, Inc., Mukul PrasadFujitsu Laboratories of America, Inc, Abhik RoychoudhuryNational University of Singapore, Singapore
icse-2020-Software-Engineering-in-Practice00:24 - 00:36
Talk
Emily KowalczykApple Inc., Karan NairApple, Zebao GaoApple, Leopold SilbersteinApple Inc., Teng LongApple, Atif MemonApple Inc.
icse-2020-papers00:36 - 00:48
Talk
Dongjie ChenNanjing University, Yanyan JiangNanjing University, Chang XuNanjing University, Xiaoxing MaNanjing University, Jian LuNanjing University
icse-2020-papers00:48 - 01:00
Talk
Xianhao JinVirginia Tech, USA, Francisco ServantVirginia Tech
Pre-print