A Container-Based Infrastructure for Fuzzy-Driven Root Causing of Flaky TestsNIER
Intermittent test failures (test flakiness) is common during continuous integration as modern software systems have become inherently non-deterministic. Understanding the root cause of test flakiness is crucial as intermittent test failures might be the result of real non-deterministic defects in the production code, rather than mere errors in the test code. Existing techniques for root causing test flakiness compare the runtime behaviour of passing and failing executions of the same test. They achieve this by repetitively executing a flaky test on an instrumented version of the system under test. This approach has two fundamental limitations: (i) code instrumentation might disrupt the normal behaviour of the test preventing the manifestation of test flakiness; (ii) passively re-executing a test many times could be insufficient for triggering intermittent test outcomes when test flakiness is rare or difficult to manifest. To address these limitations, we propose a new approach for root causing test flakiness that actively explores the non-deterministic space, without instrumenting code. Our novel idea is to repetitively execute a flaky test, under different execution clusters. Each cluster explores a certain non-deterministic dimension (e.g., concurrency, I/O, and networking) with dedicated software containers and fuzzy-driven resource load generators. The execution cluster that manifests the most balanced (or unbalanced) sets of passing and failing executions is likely to explain the broad type of test flakiness.
Wed 8 Jul Times are displayed in time zone: (UTC) Coordinated Universal Time change
16:05 - 17:05: A12-TestingPaper Presentations / Journal First / New Ideas and Emerging Results / Demonstrations / Technical Papers at Silla Chair(s): Sasa MisailovicUniversity of Illinois at Urbana-Champaign | |||
16:05 - 16:17 Talk | Practical Fault Detection in Puppet Programs Technical Papers Thodoris SotiropoulosAthens University of Economics and Business, Dimitris MitropoulosAthens University of Economics and Business, Diomidis SpinellisAthens University of Economics and Business | ||
16:17 - 16:25 Talk | Empirical Assessment of Multimorphic TestingJ1 Journal First Paul TemplePReCISE, NaDi, UNamur, Mathieu Acher(Univ Rennes, Inria, IRISA), Jean-Marc JézéquelUniv Rennes - IRISA | ||
16:25 - 16:28 Talk | RTj: a Java framework for detecting and refactoring rotten green test casesDemo Demonstrations Matias MartinezUniversité Polytechnique Hauts-de-France, Anne EtienUniversité de Lille, CNRS, Inria, Centrale Lille, UMR 9189 –CRIStAL, Stéphane Ducasse INRIA Lille, Christopher FuhrmanÉcole de technologie supérieure Pre-print Media Attached | ||
16:28 - 16:34 Talk | A Container-Based Infrastructure for Fuzzy-Driven Root Causing of Flaky TestsNIER New Ideas and Emerging Results Valerio TerragniUniversità della Svizzera Italiana, Pasquale SalzaUniversity of Zurich, Filomena FerrucciUniversity of Salerno Pre-print Media Attached | ||
16:34 - 16:46 Talk | Learning from, Understanding, and Supporting DevOps Artifacts for Docker Technical Papers Jordan HenkelUniversity of Wisconsin–Madison, Christian BirdMicrosoft Research, Shuvendu LahiriMicrosoft Research, Thomas RepsUniversity of Wisconsin-Madison, USA | ||
16:46 - 16:54 Talk | Improving Change Prediction Models with Code Smell-Related InformationJ1 Journal First Gemma CatolinoDelft University of Technology, Fabio PalombaUniversity of Salerno, Francesca Arcelli FontanaUniversity of Milano-Bicocca, Andrea De LuciaUniversity of Salerno, Andy ZaidmanTU Delft, Filomena FerrucciUniversity of Salerno DOI Pre-print | ||
16:54 - 16:57 Talk | SMRL: A Metamorphic Security Testing Tool for Web SystemsDemo Demonstrations Phu X. MaiUniversity of Luxembourg, Arda GoknilSnT, University of Luxembourg, Fabrizio PastoreUniversity of Luxembourg, Lionel C. BriandSnT Centre/University of Luxembourg |