Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep LearningTechnical
According to the World Health Organization(WHO), it is estimated that approximately 1.3 billion people live with some form of vision impairment globally, of whom 36 million are blind. Due to their disability, engaging these minority into the society is a challenging problem. The recent rise of smart mobile phones provides a new solution by enabling blind users’ convenient access to the information and service for understanding the world. Users with vision impairment can adopt the screen reader embedded in the mobile operating systems to read the content of each screen within the app, and use gestures to interact with the phone. However, the prerequisite of using screen readers is that developers have to add natural-language labels to the image-based components when they are developing the app. Unfortunately, more than 77% apps have issues of missing labels, according to our analysis of 10,408 Android apps. Most of these issues are caused by developers’ lack of awareness and knowledge in considering the minority. And even if developers want to add the labels to UI components, they may not come up with concise and clear description as most of them are of no visual issues. To overcome these challenges, we develop a deep-learning based model to automatically predict the labels of image-based buttons by learning from large-scale commercial apps in Google Play. The experiment results show that our model can make accurate predictions and the generated labels are of higher quality than that from real Android developers. We also submit our predicted labels of buttons of some apps to their development teams, and successfully get some positive feedback.
Sat 11 JulDisplayed time zone: (UTC) Coordinated Universal Time change
01:05 - 02:05 | P29-Android and Web TestingDemonstrations / Technical Papers / Software Engineering in Practice at Goguryeo Chair(s): Hironori Washizaki Waseda University | ||
01:05 12mTalk | SLACC: Simion-based Language Agnostic Code ClonesTechnical Technical Papers George Mathew North Carolina State University, Chris Parnin North Carolina State University, Kathryn Stolee North Carolina State University Pre-print | ||
01:17 8mTalk | Near-Duplicate Detection in Web App Model InferenceTechnical Technical Papers Rahulkrishna Yandrapally University of British Columbia, Canada, Andrea Stocco UniversitĂ della Svizzera italiana, Ali Mesbah University of British Columbia Pre-print | ||
01:25 12mTalk | JSidentify: A Hybrid Framework for Detecting Plagiarism Among JavaScript Code in Online Mini GamesSEIP Software Engineering in Practice Qun Xia Tencent Inc., Zhongzhu Zhou , Zhihao Li Tencent Inc., Bin Xu Tencent Inc., Wei Zou Tencent Inc., Zishun Chen Tencent Inc., Huafeng Ma Tencent Inc., Gangqiang Liang Tencent Inc., Haochuan Lu Fudan University, Shiyu Guo Tencent Inc., Ting Xiong Tencent Inc., Yuetang Deng Tencent, Inc., Tao Xie Peking University | ||
01:37 12mTalk | Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep LearningTechnical Technical Papers Jieshan Chen Australian National University, Chunyang Chen Monash University, Zhenchang Xing Australia National University, Xiwei (Sherry) Xu Data 61, Liming Zhu CSIRO's Data61 and UNSW, Guoqiang Li Shanghai Jiao Tong University, Jinshui Wang School of Information Science and Engineering, Fujian University of Technology, Fuzhou, China | ||
01:49 3mTalk | DroidMutator: An Effective Mutation Analysis Tool for Android ApplicationsDemo Demonstrations Jian Liu East China Normal University, Xusheng Xiao Case Western Reserve University, Lihua Xu New York University Shanghai, Liang Dou East China Normal University, Andy Podgurski Case Western University | ||
01:52 3mTalk | BigTest: Symbolic Execution Based Systematic Test Generation Tool for Apache SparkDemo Demonstrations Muhammad Ali Gulzar University of California, Los Angeles, Madan Musuvathi Microsoft Research, Miryung Kim University of California, Los Angeles |