Write a Blog >>
ICSE 2020
Wed 24 June - Thu 16 July 2020
Thu 9 Jul 2020 00:24 - 00:36 at Goguryeo - P14-Testing Chair(s): Shin Yoo

Test flakiness—inability to reliably repeat a test’s Pass/Fail outcome—continues to be a significant problem in Industry, adversely impacting continuous integration and test pipelines. Completely eliminating flaky tests is not a realistic option as a significant fraction of system tests (typically non-hermetic) for services-based implementations exhibit some level of flakiness. In this paper, we view the flakiness of a test as a rankable value, which we quantify, track and assign a confidence. We develop two ways to model flakiness, capturing the randomness of test results via entropy, and the temporal variation via flipRate, and aggregating these over time. We have implemented our flakiness scoring service and discuss how its adoption has impacted test suites of two large services at Apple. We show how flakiness is distributed across the tests in these services, including typical score ranges and outliers. The flakiness scores are used to monitor and detect changes in flakiness trends. Evaluation results demonstrate near perfect accuracy in ranking, identification and alignment with human interpretation. The scores were used to identify 2 causes of flakiness in the dataset evaluated, which have been confirmed, and where fixes have been implemented or are underway. Our models reduced flakiness by 44% with less than 1% loss in fault detection.

Thu 9 Jul
Times are displayed in time zone: (UTC) Coordinated Universal Time change

00:00 - 01:00: P14-TestingPaper Presentations / Technical Papers / Software Engineering in Practice at Goguryeo
Chair(s): Shin YooKorea Advanced Institute of Science and Technology
00:00 - 00:12
Talk
Technical Papers
Dehai ZhaoAustralian National University, Zhenchang XingAustralia National University, Chunyang ChenMonash University, Xiwei XuData 61, Liming ZhuCSIRO's Data61 and UNSW, Guoqiang LiShanghai Jiao Tong University, Jinshui WangSchool of Information Science and Engineering, Fujian University of Technology, Fuzhou, China
00:12 - 00:24
Talk
Technical Papers
Xiang GaoNational University of Singapore, Singapore, Ripon SahaFujitsu Laboratories of America, Inc., Mukul R. PrasadFujitsu Laboratories of America, Inc, Abhik RoychoudhuryNational University of Singapore, Singapore
00:24 - 00:36
Talk
Software Engineering in Practice
Emily KowalczykApple Inc., Karan NairApple, Zebao GaoApple, Leopold SilbersteinApple Inc., Teng LongApple, Atif MemonApple Inc.
00:36 - 00:48
Talk
Technical Papers
Dongjie ChenNanjing University, Yanyan JiangNanjing University, Chang XuNanjing University, Xiaoxing MaNanjing University, Jian LuNanjing University
00:48 - 01:00
Talk
Technical Papers
Xianhao JinVirginia Tech, USA, Francisco ServantVirginia Tech
Pre-print