Methodological Principles for Reproducible Performance Evaluation in Cloud ComputingJ1
The rapid adoption and the diversification of cloud computing technology exacerbate the importance of a sound experimental methodology for this domain. It has been a decade since the first commercial cloud has opened for a general public (Amazon AWS, in 2007). Cloud computing systems are now in much demand, and command significant industrial and scientific interest. Although scientific progress and industry growth depend on using sound principles for measuring and reporting cloud-system performance, this process remains complex. This work investigates how to measure and report performance in the cloud, and how well the cloud research community is already doing it. There are several difficulties, including controlling the time-accuracy trade-off, deciding which data to collect and to report, and using the right summarization statistics in the report. We propose a set of eight important methodological principles that combine best-practices from nearby fields with concepts applicable only to clouds, and with new ideas about the time-accuracy trade-off. We show how these principles are applicable using a practical use-case experiment. To this end, we analyze the ability of the newly released SPEC Cloud IaaS benchmark to follow the principles, and showcase real-world experimental studies in common cloud environments that meet the principles. Last, we report on a systematic literature review including top conferences and journals in the field, from 2012 to 2017, analyzing if the practice of reporting cloud performance measurements follows the proposed eight principles. Worryingly, this systematic survey and the subsequent two-round human reviews, reveal that few of the published studies follow the eight experimental principles. We conclude that, although these important principles are simple and basic, the cloud community is yet to adopt them broadly to deliver sound measurement of cloud environments.
Tue 7 JulDisplayed time zone: (UTC) Coordinated Universal Time change
07:00 - 08:00 | I1-MetastudiesNew Ideas and Emerging Results / Technical Papers / Journal First / Demonstrations at Baekje Chair(s): Michael Vierhauser Johannes Kepler University Linz | ||
07:00 12mTalk | An Evidence-Based Inquiry into the Use of Grey Literature in Software EngineeringTechnical Technical Papers He Zhang Nanjing University, Xin Zhou State Key Laboratory of Novel Software Technology, Software Institute, Nanjing University, Xin Huang State Key Laboratory of Novel Software Technology, Software Institute, Nanjing University, Huang Huang State Key Laboratory of Novel Software Technology, Software Institute, Nanjing University, Muhammad Ali Babar The University of Adelaide | ||
07:12 8mTalk | An Extended Abstract of "Metamorphic Robustness Testing: Exposing Hidden Defects in Citation Statistics and Journal Impact Factors"J1 Journal First Zhi Quan (George) Zhou University of Wollongong, Australia, T.H. Tse The University of Hong Kong, Matt Witheridge University of Wollongong | ||
07:20 8mTalk | A Procedure and Guidelines for Analyzing Groups of Software Engineering ReplicationsJ1 Journal First Adrian Santos Parrilla University of Oulu, Sira Vegas Universidad Politecnica de Madrid, Markku Oivo University of Oulu, Natalia Juristo Universidad Politecnica de Madrid | ||
07:28 3mTalk | An SLR-Tool: Search Process in PracticeDemo Demonstrations Andreas Hinderks University of Seville, Francisco José Domínguez Mayo University of Seville, Jörg Thomaschewski University of Applied Sciences Emden/Leer, María José Escalona University of Seville | ||
07:31 6mTalk | Threats to Validity in Experimenting Mutation-Based Fault LocalizationNIER New Ideas and Emerging Results | ||
07:37 8mTalk | Methodological Principles for Reproducible Performance Evaluation in Cloud ComputingJ1 Journal First Alessandro Vittorio Papadopoulos Mälardalen University, Laurens Versluis Vrije Universiteit Amsterdam, André Bauer University of Würzburg, Nikolas Herbst University of Würzburg, Joakim von Kistowski University of Würzburg, Ahmed Ali-Eldin UMass Amherst, Cristina L. Abad Escuela Superior Politecnica del Litoral, Jose Nelson Amaral University of Alberta, Petr Tuma Charles University, Alexandru Iosup Vrije Universiteit Amsterdam | ||
07:45 8mTalk | Bayesian Data Analysis in Empirical Software Engineering ResearchJ1 Journal First Carlo A. Furia Università della Svizzera italiana (USI), Robert Feldt Chalmers | University of Gothenburg, Blekinge Institute of Technology, Richard Torkar Chalmers and the University of Gothenburg DOI Pre-print |