The Definitive ANTLR 4 Reference

by Terence Parr

Programmers run into parsing problems all the time. Whether it’s a data format like JSON, a network protocol like SMTP, a server configuration file for Apache, a PostScript/PDF file, or a simple spreadsheet macro language—ANTLR v4 and this book will demystify the process. ANTLR v4 has been rewritten from scratch to make it easier than ever to build parsers and the language applications built on top. This completely rewritten new edition of the bestselling Definitive ANTLR Reference shows you how to take advantage of these new features.

Watch a lecture by Dr. Parr

“Parr’s clear writing and lighthearted style make it a pleasure to learn the practical details of building language processors.”

Dan Bornstein - Designer of the Dalvik VM for Android

“ANTLR is an exceptionally powerful and flexible tool for parsing formal languages. At Twitter, we use it exclusively for query parsing in our search engine. Our grammars are clean and concise and the generated code is efficient and stable. This book is our go-to reference for ANTLR v4—engaging writing, clear descriptions and practical examples all in one place.”

Samuel Luckenbill - Senior Manager of Search Infrastructure - Twitter, Inc.

Build your own languages with ANTLR v4, using ANTLR’s new advanced parsing technology. In this book, you’ll learn how ANTLR automatically builds a data structure representing the input (parse tree) and generates code that can walk the tree (visitor). You can use that combination to implement data readers, language interpreters, and translators.

You’ll start by learning how to identify grammar patterns in language reference manuals and then slowly start building increasingly complex grammars. Next, you’ll build applications based upon those grammars by walking the automatically generated parse trees. Then you’ll tackle some nasty language problems by parsing files containing more than one language (such as XML, Java, and Javadoc). You’ll also see how to take absolute control over parsing by embedding Java actions into the grammar.

You’ll learn directly from well-known parsing expert Terence Parr, the ANTLR creator and project lead. You’ll master ANTLR grammar construction and learn how to build language tools using the built-in parse tree visitor mechanism. The book teaches using real-world examples and shows you how to use ANTLR to build such things as a data file reader, a JSON to XML translator, an R parser, and a Java class->interface extractor. This book is your ticket to becoming a parsing guru!

You can find out more about ANTLR 4 in this interview with Terence Parr.

What You Need:

ANTLR 4.0 and above. Java development tools. Ant build system optional
(needed for building ANTLR from source)

• Errata, typos, suggestions

• Source Code (zip file)

Releases:

• P2.0 2014/09/16
• P1.0 2013/01/16
• B4.0 2012/12/13
• B3.0 2012/11/12

Preface

• Introducing ANTLR and Computer Languages
  • Meet ANTLR
    • Installing ANTLR
    • Executing ANTLR and Testing Recognizers
  • The Big Picture
    • Let’s Get Meta! excerpt
    • Implementing Parsers
    • You Can’t Put Too Much Water into a Nuclear Reactor
    • Building Language Applications Using Parse Trees
    • Parse Tree Listeners and Visitors
    • A Starter ANTLR Project
  • A Quick Tour
    • Matching an Expression Language
    • Building a Calculator Using a Visitor
    • Building a Translator with a Listener excerpt
    • Making Things Happen During the Parse
    • Cool Lexical Features
• Developing Language Applications With ANTLR Grammars
  • Designing Grammars
    • Deriving Grammars from Language Samples
    • Using Existing Grammars as a Guide
    • Recognizing Common Language Patterns with ANTLR Grammars
    • Dealing with Precedence, Left Recursion, and Associativity
    • Recognizing Common Lexical Structures
    • Drawing the Line Between Lexer and Parser
  • Exploring Some Real Grammars
    • Parsing Comma-Separated Values
    • Parsing JSON
    • Parsing DOT
    • Parsing Cymbol
    • Parsing R
  • Decoupling Grammars and Application-Specific Code
    • Evolving from Embedded Actions to Listeners
    • Implementing Applications with Parse Tree Listeners
    • Implementing Applications with Visitors
    • Labeling Rule Alternatives for Precise Event Methods
    • Sharing Information among Event Methods
  • Building Some Real Language Applications
    • Loading CSV Data
    • Translating JSON to XML
    • Generating a Call Graph
    • Validating Program Symbol Usage
  • Error Reporting and Recovery
    • A Parade of Errors
    • Altering and redirecting ANTLR Error Messages
    • Automatic Error Recovery Strategy
    • Error Alternatives
    • Altering ANTLR’s Error Handling Strategy
• Advanced topics
  • Attributes and Actions
    • Building a Calculator with Grammar Actions
    • Accessing Token and Rule Attributes
    • Recognizing Languages whose Keywords Aren’t Fixed
  • Altering the Parse with Semantic Predicates
    • Recognizing Multiple Language Dialects
    • Deactivating Tokens
    • Recognizing Ambiguous Phrases
    • The Fine Print
  • Wielding Lexical Black Magic
    • Broadcasting Tokens on Different Channels
    • Context-Sensitive Lexical Problems
    • Islands in the Stream excerpt
    • Parsing and Lexing XML
  • Suggestions for building language applications
• ANTLR Reference
  • Exploring the runtime API
    • Library Package Overview
    • Recognizers
    • Input Streams of Characters and Tokens
    • Tokens and Token Factories
    • Parse Trees
    • Errors Listeners and Strategies
  • ANTLR Reference
  • Removing Direct Left-Recursion
    • Direct Left-Recursive Alternative Patterns
    • Left-Recursive Rule Transformations