Context-aware In-process Crowdworker RecommendationTechnical
Identifying and optimizing open participation is essential to the success of open software development. Existing studies highlighted the importance of worker recommendation for crowdtesting tasks in order to detect more bugs with fewer workers. However, these studies mainly provide one-time recommendations, i.e., recommending a set of workers for a new task with its fixed, initial context at the beginning of the task. We argue the need for in-process crowdtesting worker recommendation. We motivate this study through a pilot study, revealing the prevalence of long-sized non-yielding windows, i.e., no new bugs are revealed in consecutive test reports during the process of a crowdtesting task. This indicates the potential opportunity for accelerating test cycle by recommending appropriateworkers in a dynamic manner, so that the non-yielding windows could be shortened. This paper proposes a context-aware in-process crowdworker recommendation approach, R3Rec, to dynamically identify and rank a diverse set of capable crowdworkers, in order to detect more bugs earlier and potentially shorten the non-yielding windows. It consists of three main components: 1) the modeling of dynamic testing context in terms of process context and resource context to capture the in-process progress and worker characteristics respectively; 2) the learning-based ranking component to learn the probability of crowdworkers’ bug detection capability; and 3) the diversity-based re-ranking component to adjust the ranked list to potentially reduce the duplicate bugs. The evaluation is conducted on 636 crowdtesting tasks from one of the largest crowdtesting platforms, and the results showthat R3Rec can shorten the non-yielding windows by 50% -58% and reduce the total crowdtesting cost by about 10% on median. This is the first work to propose the in-process worker recommendation solution, and results showed its potential in improving the costeffectiveness of crowdtesting by saving the cost and shortening the testing cycle.
Tue 7 JulDisplayed time zone: (UTC) Coordinated Universal Time change
08:05 - 09:05 | I6-Empirical Studies and RequirementsJournal First / Software Engineering in Practice / Technical Papers at Silla Chair(s): Ita Richardson Lero - The Irish Software Research Centre and University of Limerick | ||
08:05 8mTalk | What do Programmers Discuss about Deep Learning FrameworksJ1 Journal First Junxiao Han Zhejiang University, Emad Shihab Concordia University, Zhiyuan Wan Zhejiang University, Shuiguang Deng Zhejiang University, Xin Xia Monash University | ||
08:13 12mTalk | Detection of Hidden Feature Requests from Massive Chat Messages via Deep Siamese NetworkTechnical Technical Papers Lin Shi ISCAS, Mingzhe Xing ISCAS, Mingyang Li ISCAS, Yawen Wang ISCAS, Shoubin Li ISCAS, Qing Wang Institute of Software, Chinese Academy of Sciences | ||
08:25 8mTalk | Recognizing lines of code violating company-specific coding guidelines using machine learningJ1 Journal First Miroslaw Ochodek Poznan University of Technology, Regina Hebig Chalmers University of Technology & University of Gothenburg, Wilhelm Meding Ericsson, Gert Frost Grundfos, Miroslaw Staron University of Gothenburg | ||
08:33 12mTalk | Context-aware In-process Crowdworker RecommendationTechnical Technical Papers Junjie Wang Institute of Software, Chinese Academy of Sciences, Ye Yang Stevens institute of technology, Song Wang York University, Yuanzhe Hu Institute of Software, Chinese Academy of Sciences, Dandan Wang Institute of Software, Chinese Academy of Sciences, Qing Wang Institute of Software, Chinese Academy of Sciences | ||
08:45 12mTalk | Using a Context-Aware Approach to Recommend Code Reviewers: Findings from an Industrial Case StudySEIP Software Engineering in Practice Anton Strand Ericsson AB, Markus Gunnarsson Ericsson AB, Ricardo Britto Ericsson / Blekinge Institute of Technology, Muhammad Usman Blekinge Institute of Technology |