Big Code != Big Vocabulary: Open-Vocabulary Models for Source code
Statistical language modeling techniques have successfully been applied to large source code corpora, yielding a variety of new software development tools, such as tools for code suggestion, improving readability, and API migration. A major issue with these techniques is that code introduces new vocabulary at a far higher rate than natural language, as new identifier names proliferate. Both large vocabularies and out-of-vocabulary issues severely affect Neural Language Models (NLMs) of source code, degrading their performance and rendering them unable to scale. In this paper, we address this issue by: 1) studying how various modelling choices impact the resulting vocabulary on a large-scale corpus of 13,362 projects; 2) presenting an open vocabulary source code NLM that can scale to such a corpus, 100 times larger than in previous work; and 3) showing that such models outperform the state of the art on three distinct code corpora (Java, C, Python). To our knowledge, these are the largest NLMs for code that have been reported. All datasets, code, and trained models used in this work are publicly available.
Sat 11 JulDisplayed time zone: (UTC) Coordinated Universal Time change
00:00 - 01:00
P27-ApplicationsSoftware Engineering in Practice / Technical Papers at Silla
Chair(s): Ganesha Upadhyaya Harmony.one
|Big Code != Big Vocabulary: Open-Vocabulary Models for Source codeTechnical|
Rafael-Michael Karampatsis The University of Edinburgh, Hlib Babii Free University of Bozen-Bolzano, Romain Robbes Free University of Bozen-Bolzano, Charles Sutton Google Research, Andrea Janes Free University of Bozen-BolzanoDOI Pre-print
|Engineering for a Science-Centric Experimentation PlatformSEIP|
Software Engineering in Practice
Nikos Diamantopoulos Netflix, Inc., Jeffrey Wong Netflix, Inc., David Issa Mattos Chalmers University of Technology, Ilias Gerostathopoulos Vrije Universiteit Amsterdam, Matthew Wardrop Netflix, Inc., Tobias Mao Netflix, Inc., Colin McFarland Netflix, Inc.
|Managing data constraints in database-backed web applicationsTechnical|
Junwen Yang University of Chicago, Utsav Sethi University of Chicago, Cong Yan University of Washington, Alvin Cheung University of California, Berkeley, Shan Lu University of Chicago
|Improving Data Scientist Efficiency with ProvenanceTechnical|
Jingmei Hu Harvard University, Jiwon Joung Harvard University, Maia Jacobs Harvard University, Margo Seltzer University of British Columbia, Krzysztof Gajos Harvard University