Write a Blog >>
ICSE 2020
Wed 24 June - Thu 16 July 2020
Tue 7 Jul 2020 07:37 - 07:45 at Baekje - I1-Metastudies Chair(s): Michael Vierhauser

The rapid adoption and the diversification of cloud computing technology exacerbate the importance of a sound experimental methodology for this domain. It has been a decade since the first commercial cloud has opened for a general public (Amazon AWS, in 2007). Cloud computing systems are now in much demand, and command significant industrial and scientific interest. Although scientific progress and industry growth depend on using sound principles for measuring and reporting cloud-system performance, this process remains complex. This work investigates how to measure and report performance in the cloud, and how well the cloud research community is already doing it. There are several difficulties, including controlling the time-accuracy trade-off, deciding which data to collect and to report, and using the right summarization statistics in the report. We propose a set of eight important methodological principles that combine best-practices from nearby fields with concepts applicable only to clouds, and with new ideas about the time-accuracy trade-off. We show how these principles are applicable using a practical use-case experiment. To this end, we analyze the ability of the newly released SPEC Cloud IaaS benchmark to follow the principles, and showcase real-world experimental studies in common cloud environments that meet the principles. Last, we report on a systematic literature review including top conferences and journals in the field, from 2012 to 2017, analyzing if the practice of reporting cloud performance measurements follows the proposed eight principles. Worryingly, this systematic survey and the subsequent two-round human reviews, reveal that few of the published studies follow the eight experimental principles. We conclude that, although these important principles are simple and basic, the cloud community is yet to adopt them broadly to deliver sound measurement of cloud environments.

Tue 7 Jul
Times are displayed in time zone: (UTC) Coordinated Universal Time change

07:00 - 08:00: I1-MetastudiesPaper Presentations / New Ideas and Emerging Results / Technical Papers / Journal First / Demonstrations at Baekje
Chair(s): Michael VierhauserJohannes Kepler University Linz
07:00 - 07:12
Technical Papers
He ZhangNanjing University, Xin ZhouState Key Laboratory of Novel Software Technology, Software Institute, Nanjing University, Xin HuangState Key Laboratory of Novel Software Technology, Software Institute, Nanjing University, Huang HuangState Key Laboratory of Novel Software Technology, Software Institute, Nanjing University, Muhammad Ali BabarThe University of Adelaide
07:12 - 07:20
Journal First
Zhi Quan (George) ZhouUniversity of Wollongong, Australia, T.H. TseThe University of Hong Kong, Matt WitheridgeUniversity of Wollongong
07:20 - 07:28
Journal First
Adrian Santos ParrillaUniversity of Oulu, Sira VegasUniversidad Politecnica de Madrid, Markku OivoUniversity of Oulu, Natalia JuristoUniversidad Politecnica de Madrid
07:28 - 07:31
Andreas HinderksUniversity of Seville, Francisco José Domínguez MayoUniversity of Seville, Jörg ThomaschewskiUniversity of Applied Sciences Emden/Leer, María José EscalonaUniversity of Seville
07:31 - 07:37
New Ideas and Emerging Results
Juyoung JeonHandong Global University, Shin HongHandong Global University
07:37 - 07:45
Journal First
Alessandro Vittorio PapadopoulosMälardalen University, Laurens VersluisVrije Universiteit Amsterdam, André BauerUniversity of Würzburg, Nikolas HerbstUniversity of Würzburg, Joakim von KistowskiUniversity of Würzburg, Ahmed Ali-EldinUMass Amherst, Cristina L. AbadEscuela Superior Politecnica del Litoral, Jose Nelson AmaralUniversity of Alberta, Petr TumaCharles University, Alexandru IosupVrije Universiteit Amsterdam
07:45 - 07:53
Journal First
Carlo A. FuriaUniversità della Svizzera italiana (USI), Robert FeldtChalmers | University of Gothenburg, Blekinge Institute of Technology, Richard TorkarChalmers and the University of Gothenburg
DOI Pre-print