Write a Blog >>
ICSE 2020
Wed 24 June - Thu 16 July 2020
Sat 11 Jul 2020 00:36 - 00:48 at Goguryeo - P26-Deep Learning Testing and Debugging Chair(s): Tim Menzies

Image classifiers have become an important component of today’s software, from consumer and business applications to safety-critical domains. The advent of Deep Neural Networks (DNNs) is the key catalyst behind such wide-spread success. However, wide adoption comes with serious concerns about the robustness of software systems dependent on image classification DNNs, as several severe erroneous behaviors have been reported under sensitive and critical circumstances. We argue that developers need to rigorously test their software’s image classifiers and delay deployment until acceptable. We present an approach to testing image classifier robustness based on class property violations.

We have found that many of the reported erroneous cases in popular DNN image classifiers occur because the trained models confuse one class with another or show biases towards some classes over others. These bugs usually violate some class properties of one or more of those classes. Most DNN testing techniques focus on per-image violations and thus fail to detect such class-level confusions or biases.

We developed a testing approach to automatically detect class-based confusion and bias errors in DNN-driven image classification software. We evaluated our implementation, DeepInspect, on several popular image classifiers with precision up to 100% (avg. 72.6%) for confusion errors, and up to 84.3% (avg. 66.8%) for bias errors. DeepInspect found hundreds of classification mistakes in widely-used models, many of which expose errors indicating confusion or bias.

Sat 11 Jul
Times are displayed in time zone: (UTC) Coordinated Universal Time change

00:00 - 01:00: P26-Deep Learning Testing and DebuggingPaper Presentations / Technical Papers at Goguryeo
Chair(s): Tim MenziesNorth Carolina State University
00:00 - 00:12
Talk
Technical Papers
Brandon PaulsenUniversity of Southern California, Jingbo WangUniversity of Southern California, Chao WangUSC
Pre-print
00:12 - 00:24
Talk
Technical Papers
Pinjia HeETH Zurich, Clara MeisterETH Zurich, Zhendong SuETH Zurich, Switzerland
00:24 - 00:36
Talk
Technical Papers
Zeyu SunPeking University, Jie M. ZhangUniversity College London, UK, Mark HarmanFacebook and University College London, Mike PapadakisUniversity of Luxembourg, Lu ZhangPeking University, China
00:36 - 00:48
Talk
Technical Papers
Yuchi TianColumbia University, Ziyuan ZhongColumbia University, Vicente OrdonezUniversity of Virginia, Gail KaiserColumbia University, Baishakhi RayColumbia University, New York
00:48 - 01:00
Talk
Technical Papers
Md Johirul IslamIowa State University, Rangeet PanIowa State University, USA, Giang NguyenDept. of Computer Science, Iowa State University, Hridesh RajanIowa State University, USA