Automatic Testing and Improvement of Machine TranslationTechnical
This paper presents TransRepair, a fully automatic approach for testing and repairing the consistency of machine translation systems. TransRepair combines mutation with metamorphic testing to detect inconsistency bugs (without access to human oracles). It then adopts probability-reference or cross-reference to post-process the translations, in a grey-box or black-box manner, to repair the inconsistencies. Our evaluation on two state-of-the-art translators, Google Translate and Transformer, indicates that TransRepair has a high precision (99%) on generating input pairs with consistent translations. With these tests, using automatic consistency metrics and manual assessment, we find that Google Translate and Transformer have approximately 38% and 42% inconsistency bugs. Black-box repair fixes 28% and 19% bugs on average for Google Translate and Transformer. Grey-box repair fixes 30% bugs on average for Transformer. Manual inspection indicates that the translations repaired by our approach improve consistency in 87% of cases (degrading it in 2%), and that our repairs have better translation quality in 27% of the cases (worse in 8%).
Sat 11 JulDisplayed time zone: (UTC) Coordinated Universal Time change
00:00 - 01:00 | P26-Deep Learning Testing and DebuggingTechnical Papers at Goguryeo Chair(s): Tim Menzies North Carolina State University | ||
00:00 12mTalk | ReluDiff: Differential Verification of Deep Neural Networks Technical Papers Brandon Paulsen University of Southern California, Jingbo Wang University of Southern California, Chao Wang USC Pre-print | ||
00:12 12mTalk | Structure-Invariant Testing for Machine TranslationTechnical Technical Papers | ||
00:24 12mTalk | Automatic Testing and Improvement of Machine TranslationTechnical Technical Papers Zeyu Sun Peking University, Jie M. Zhang University College London, UK, Mark Harman Facebook and University College London, Mike Papadakis University of Luxembourg, Lu Zhang Peking University, China | ||
00:36 12mTalk | Testing DNN Image Classifier for Confusion & Bias Errors Technical Papers Yuchi Tian Columbia University, Ziyuan Zhong Columbia University, Vicente Ordonez University of Virginia, Gail Kaiser Columbia University, Baishakhi Ray Columbia University, New York | ||
00:48 12mTalk | Repairing Deep Neural Networks: Fix Patterns and Challenges Technical Papers Md Johirul Islam Iowa State University, Rangeet Pan Iowa State University, USA, Giang Nguyen Dept. of Computer Science, Iowa State University, Hridesh Rajan Iowa State University, USA | ||