Historical errata for The Definitive ANTLR 4 Reference

The following:
‘… it doesn’t give damn!’
should be:
‘… it doesn’t give a damn!’

The command:
$ grun Hello r -print
should be:
$ grun Hello r -tree

There should be a space between ‘and’ and ‘Tools’ in:
‘See Compilers: Principles, Techniques, andTools [ALSU06]’

For example, {(}) is ungrammatical because of the token order and (1+2 drives us crazy looking the matching ).
should be
For example, {(}) is ungrammatical because of the token order and (1+2 drives us crazy looking FOR the matching ).

Terence Parr. The Definitive ANTLR 4 Reference, B1.0 (Steve Eckhardt) (Kindle Locations 1965-1966). The Pragmatic Bookshelf (374823).

Just like writing software, we have to figure out: which rules we need,
-omit the colon after “out”

Terence Parr. The Definitive ANTLR 4 Reference, B1.0 (Steve Eckhardt) (Kindle Location 1987). The Pragmatic Bookshelf (374823).

Towards the end of the second paragraph, the sentence is missing couple things: “but it’s helpful have a basic understanding of decision-making so debugging generated parsers is easier.” should be “but it’s helpful to have a basic understanding of the decision-making so debugging generated parsers is easier.”

“To learn more about what how the parser recognized the input” has an extra “what”

The code listing only has “-> skip” but the paragraph right after the listing mentions “-> channel(HIDDEN)”.

The first line in the chapter is missing a “to” in “Now that we know how trigger application code”

The two code listings at the top of the page have their formatting messed up.

The first paragraph is missing either “in” or “and” in “If we recover without consuming a token and get back to the same location ?? the parser, we will recover again without consuming a token.”

The second paragraph has “everything” repeated twice.

The paragraph starting “Once we can translate…” has “the the” towards the end.

The last paragraph has “the the” in the middle.

In the 3rd complete paragraph, grun is defined on p.6, not p. 23.

Suggest deleting “that we defined in Testing the Generated Parser, on page 23” and adding at the end of paragraph 1, p.31, “Make sure you have antlr4 and grun aliases or scripts defined, as explained on pages 5 and 6.”

The paragraph starting “The Java language…” has “was was” in it.

Hello.g4

WS : [ \\t\
\\r]+ -> skip ; // skip spaces, tabs, newlines, \\r needed on Windows to suppress warnings

$ grun Hello r -print
hello parrt
EOF
(r hello parrt)

should be -tree instead of -print

Unfortunately, that assumption is too restrictive because R functions and other expressions to span multiple lines.
should read
Unfortunately, that assumption is too restrictive because R ALLOWS functions and other expressions to span multiple lines.

Terence Parr. The Definitive ANTLR 4 Reference, B1.0 (Steve Eckhardt) (Kindle Locations 3473-3474). The Pragmatic Bookshelf (374823).

Interface ParseTreeListener is in the ANTLR runtime library and dictates that every listener must know response to events visitTerminal,
- did you mean
Interface ParseTreeListener is in the ANTLR runtime library and dictates that every listener must RESPOND to events visitTerminal,

Terence Parr. The Definitive ANTLR 4 Reference, B1.0 (Steve Eckhardt) (Kindle Locations 3662-3663). The Pragmatic Bookshelf (374823).

For example, visitAdd would return the result of adding to subexpressions.
-should read
For example, visitAdd would return the result of adding TWO subexpressions.

Terence Parr. The Definitive ANTLR 4 Reference, B1.0 (Steve Eckhardt) (Kindle Locations 3874-3875). The Pragmatic Bookshelf (374823).

Keywords code example doesn’t work. It fails to generate the lexer class.
beetle:test luke$ ls
Keywords.g4
beetle:test luke$ antlr4 -no-listener Keywords.g4
error(128): Keywords.g4:31:13: attribute references not allowed in lexer actions: $type
beetle:test luke$ ls
Keywords.g4\t\tKeywords.tokens\t\tKeywordsParser.java
beetle:test luke$ javac Keywords*.java
beetle:test luke$ ls
Keywords.g4\t\t\t\tKeywordsParser$StatContext.class
Keywords.tokens\t\t\t\tKeywordsParser.class
KeywordsParser$ExprContext.class\tKeywordsParser.java
beetle:test luke$ grun Keywords stat
Exception in thread “main” java.lang.ClassNotFoundException: Keywords
\tat java.net.URLClassLoader$1.run(URLClassLoader.java:202)
\tat java.security.AccessController.doPrivileged(Native Method)
\tat java.net.URLClassLoader.findClass(URLClassLoader.java:190)
\tat java.lang.ClassLoader.loadClass(ClassLoader.java:306)
\tat sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
\tat java.lang.ClassLoader.loadClass(ClassLoader.java:247)
\tat org.antlr.v4.runtime.misc.TestRig.main(TestRig.java:148)

Missing TestEnum.java in javac command. Also it seems that when actions or predicates are used in a grammar, the parser complains about token. If I remove any predicate or grammar action code from the grammar, the parser doesn’t complain about the .
beetle:test luke$ antlr4 -no-listener Enum.g4
beetle:test luke$ javac Enum*.java TestEnum.java
beetle:test luke$ java TestEnum
enum = 0;
enum=0
line 2:0 no viable alternative at input ‘’

I’m using patch jar antlr4-complete.jar

Here is a much better operator precedence table to start with. See the book R in a Nutshell (2nd edition):

( { Function calls and grouping expressions (respectively)
[ [[ Indexing
:: ::: Access variables in a namespace
$ @ Component / slot extraction
^ Exponentiation (right to left)
- + Unary operators for minus and plus
: Sequence operator
any Special operators

• / Multiply, divide
  + - Binary operators for add, subtract
  < > <= >= == != Ordering and comparison
  ! Negation
  & && And
  | || Or
  ~ As in formulas
  -> ->> Rightward assignment
  = Assignment (right to left)
  <- <<- Assignment (right to left)
  ? Help (unary and binary)

antlr4 unexpected error: attribute references not allowed in lexer actions: $type
beetle:test luke$ antlr4 Java.g4
error(128): Java.g4:670:37: attribute references not allowed in lexer actions: $type
error(128): Java.g4:674:41: attribute references not allowed in lexer actions: $type
The subsequent commands don’t seem to be affected by the error.
javac Java*.java
grun Java tokens -tokens

antlr4 unexpected error: attribute references not allowed in lexer actions: $type
beetle:test luke$ antlr4 Java.g4
error(128): Java.g4:670:37: attribute references not allowed in lexer actions: $type
error(128): Java.g4:674:41: attribute references not allowed in lexer actions: $type
The subsequent commands don’t seem to be affected by the error.
javac Java*.java
grun Java tokens -tokens

grun doesn’t give the parse trees shown on p219.
beetle:test luke$ grun Java statement -gui
i = 1 >> 5;
line 1:2 no viable alternative at input ‘i=’

grun doesn’t give the parse trees shown on p219.
beetle:test luke$ grun Java localVariableDeclarationStatement -gui
List<List> x;

In par beginning “Readers familiar with ANTLR v3…”, in the phrase “(It’s also just right recursive because …)”, the word “just” is an extra word.

In par beginning “As another example …”, “the same” is duplicated in the phrase “… more
or less the same the same lexical construct …”.

The code example problems that Medhat was referring to are the extracts from examples/JSON.g4. In the first, everything following the middle double-quote has a different color and seems to be in italics, "\\\\])* ‘"’ ;

In the second, everything after the first double-quote continuing to the end of the extract has the same problem. You can see that the first line has “fragment” in a different color than the next two lines.

The example about 3/4 of the way down the page that defines the ESC rule has the same problem that Medhat Assaad and I commented about on p.91. Interestingly, the reformatting seems to not be caused by a double-quote inside single-quotes ‘"’, but only by a double-quote appearing as one of a list of characters. Apparently, this is part of an attempt to pretty-print character strings that goes awry when a single double-quote appears. There may be other places in the book where this happens, but searching for a double-quote turned up too many false-positives and I abandoned the search.

The par beginning “Set [btnr”\\\\] matches literally of those character except …" is ungrammatical. Perhaps, “Set [btnr”\\\\] matches EACH of those characterS except …"

Top of page, “Rule rowThe is the same …” should be perhaps “The row rule is the same …”

In par beginning “The token definitions …”, in the sentence “We can use a special escaped \\” in
order to get a double quote inside of the string“, is confusing because
special excaped \\”
indicates that you are using a \\“, just a special kind, but in fact the CSV format uses an alternative to \\”. Recommend changing it from
special escaped \\"
to
special method of escaping a double quote

Or just omit the whole sentence since you fully explain how the CSV format handles double-quotes a few sentences later.

In the extract from CSV.g4 that defines TEXT and STRING, Any STRING would also match the TEXT rule. Apparently the ambiguity is resolved in favor of STRING, as indicated in the test at the bottom of the page, but this contradicts the only rule that we know for ambiguity resolution at this point, which is that the first rule takes precedence. Later we learn about maximal munch, but that doesn’t explain this behavior either.

On the top of the page it says:
“The {!java5?} predicate allows”
but should be:
“The {!java5}? predicate allows”

On the top of the page it says:
“The {!java5?} predicate allows”
but should be:
“The {!java5}? predicate allows”

In the example where we run Test without a curly brace, the
code section reads like the following:

javac ArrayInit*.java Test.java
{1,2
Eof

Instead it should read like:
java Test
{1,2

In the ShortToUnicodeString.java code it would be nice but not necessary to add the @override in front of the enterInit, exitInit, and enterValue methods. This would make it clearer that we are overriding the implemented methods in the ArrayInitBaseListener.

In page 5 of the book where the antlr4 script is described, it would be nice for the readers of the electronic version of the book to know that you can click on the text with the filename (install/antlr4) to grab the script. I didn’t find out about this useful feature until later reading of the book.

In the text:
➾ $ javac ArrayInit*.java Translate.java
➾ $ java Translate
{99, 3, 451}
“\\u0063\\u0003\\u01c3”

I would change it to the following:
➾ $ javac ArrayInit*.java Translate.java
➾ $ java Translate
➾ {99, 3, 451}
➾EOF
“\\u0063\\u0003\\u01c3”

This way this stays consistent with the previous examples.

The link that is on the book for the source code is: http_bypassfilter://pragprog.com/titles/tpantlr2

It would be better if it was:
http_bypassfilter://pragprog.com/titles/tpantlr2/source_code

When using the testrig with the -gui option, if you click OK, the window closes down correctly. However when you press the red X on the top right of the window to close the window, the application is still running (disposed of the window, but does not exit). I am running in windows 7 when bringing up the window.

After running the ExprJoyRide the output I received was:
line 1:4 missing ‘)’ at ‘\
’
versus:
line 1:4 mismatched input ‘\
’ expecting {’)‘, ’+‘, ’*‘, ’-‘, ’/’}
The output text matched though.

I strongly suggest you add a section detailing the differences (the ones relevant to this book) between ANTLR 3 and ANTLR 4. ANTLR grammars tend to be quite hard to debug anyway, and not understanding that something one’s doing might just be wrong for version 3, if that’s what one is using, would be extremely frustrating.

I’m sure people will be using version 3 for quite some time to come, since it’s likely to be a long time until version 4 appears in systems such as CentOS.

The first sentence in the first paragraph of page 53 reads: The key is the TokenStreamRewriter object that knows how to give altered views of a token stream without actually modifying the stream.

New version: The key is that the TokenStreamRewriter object knows how to give altered views of a token stream without actually modifying the stream.

The two alternatives of rule ‘stat’ are for an expression or a function call.

So the descriptions of the two interpretations should be:

f(); as expression f(); as function call

or

f(); as expression alt f(); as function call alt

Rather than:

f(): as expr f(); as stat

Many thanks; a fine follow-on from “The Definitive ANTLR Reference”.

David Turland

missing last ‘u’ in first, comment, line in bigpic/ShortToUnicodeString.java

/ Convert short array inits like {1,2,3} to “\\u0001\\u0002\\0003” */

should be: |
V
/ Convert short array inits like {1,2,3} to “\\u0001\\u0002\\u0003” */

Just to anti-errata(?) the errata: The Author is also introducing the TokenStreamRewriter, so the original text is correct, ie:

“The key is the TokenStreamRewriter object that knows how to give altered views of a token stream…”

hth.

Tiny detail : the full ls output is :

Hello.g4\t\tHelloBaseListener.java\tHelloLexer.tokens\tHelloParser.java
Hello.tokens\t\tHelloLexer.java\t\tHelloListener.java

Yes, the very last line of the page (re-compiles instead of execute).

Not the same output : if

$ cat t.expr
193
a = 5
…

then the output is :

(prog (stat (expr 193) \
) …

To obtain

(prog
(stat \
)
(stat (expr 193) \
)

I had to add a blank line at the beginning of t.expr :

$ cat myt.expr

193
a = 5
…

$ java ExprJoyRide myt.expr
(prog (stat \
) (stat (expr 193) \
) …

Paragrap 4 starting with “The root node”, line 3 :
… (obtained with the -ps option):

Isn’t it rather with the -gui option ?

With
$ grun CSV file -ps data.csv
nothing happens, it waits for input. If you type some, it will erase file data.csv.

The link points to the same page. (You’ll surely change it in the final version, sounds funny.)
In the last paragraph, “As described in Loops versus tail-recursion, on page 88”
should be
In the last paragraph, “As described in Loops versus tail-recursion, below”

Not the same output. The gui tree, left under file, displays “header” instead of “hdr”, and in the second row, third field, “\\”zippo\\“” instead of “”“zippo”“”.

Not the same output. Top of page, graphic tree, right branch under file, should be functionDecl instead of methodDecl.

In the middle of the page :

If we assume, for the moment, that each parse tree node (each rule context object) has a field called value. Then, exitAdd() would look like this:

Expecting a comma after value :

… has a field called value, then exitAdd() would look like this:

First line. Let’s build another a calculator version …
Either “another” or “a” is superfluous.

Last paragraph, 3rd line :
to generate a node definition each function name

-> for <- each ???

Paragraph 4 (counting the code snippet), line 2 :
and b(). functions live in the global scope …

I would expect a new sentence starting with F :
and b(). Functions live in the global scope …

Second paragraph of text, line 4 :

GlobalScope, FunctionScope, and LocalScope

Should be FunctionSymbol instead of FunctionScope :

public class FunctionSymbol in file code/listeners/FunctionSymbol.java.

Bottom, second command (compilation), must be

javac S*.java instead of
javac E*.java

Middle of the page, first execution of grun Simple prog
I have
line 3:0 mismatched input ‘’ expecting ‘}’
instead of
line 3:0 missing ‘}’ at ‘’

Not same output. Top of page, execution of
grun VecMsg vec4
➾ [1,2,3,4,5,6]
line 1:9 rule ints failed predicate: {exceeded max 4}?

instead of

line 1:9 rule ints: exceeded max 4

Compilation errors. Last code snippet errors/TestBail.java

public static class BailELexer extends ELexer { …… must be changed to
public static class BailELexer extends SimpleLexer {

and the last line

javac E*.java TestBail.java …… must be changed to
javac S*.java TestBail.java

Top of page, list of error messages. As a consequence of replacing E by Simple, the last line of the call stack

at EParser.prog(EParser.java:46) ….. becomes
at SimpleParser.prog(SimpleParser.java:68)

Please check.

I have created a MyErrorStrategy.java file with the code snippet shown

public class MyErrorStrategy extends DefaultErrorStrategy {
…

and changed TestBail.java to use it :

parser.setErrorHandler(new MyErrorStrategy());

Compiling MyErrorStrategy.java ==> error cannot find symbol BaseRecognizer

A Spotlight search found it in antlr-3.2. Replaced by Parser :

public class MyErrorStrategy …
… public void reportNoViableAlternative(Parser recognizer, …

Compilation OK, test OK.

Last paragraph, second line

go, we can specify an ANTLRErrorListener as we did in Section 6.2 …

Isn’t rather Section 8.2 Altering and redirecting ANTLR Error Messages ? (first occurrence of ANTLRErrorListener)

Last paragraph (before code snippet) :

Turning to rule e now, let’s see how what it looks like …

what superfluous ???

First paragraph, line 6 :

If T it is a type name …….. should be :
If T is a type name,

Second to last line :

going solve some very difficult recognition

>>> going to solve ???

To be perfect, please align the vertical bars in the code snippet.

Chapter 12 Exploring the runtime API

Please check the link at the bottom of the page. Seams 3.4.

Before section 12.3 : And here’s the build and test sequence:

1) If we compile with

javac *.java

we have errors : cannot find symbol CSVLexer. …
So compile only Simple with :

javac Simple*.java

2) Then we have SimpleLexer.java:17: SimpleLexer is not abstract and does not override abstract method getTokenFactory() …
public class SimpleLexer implements TokenSource {
_^
1 error

Tried with antlr-4.0ea-complete.jar. Pain! Pain! Explosion of errors starting with :

SimpleParser.java:14: cannot find symbol
symbol : class PredictionContextCache

3) Back to 4.0b2. Finally the problem was solved by adding these two lines at the end of SimpleLexer.java :

@Override
public TokenFactory<?> getTokenFactory() {return factory;}

Commands in paragraph 4.

Again, as there are more than one example in the same directory, we cannot compile with javac *.java. Instead use :

javac MyToken*.java TestSimpleMyToken.java

Last paragraph starting with unary prefix.

I would say there is a discrepancy between the sentence :

Any alternative of the form elements expr.

and the use of rule expression in the rule expr :

expr: …
….. | ‘(’ type ‘)’ expression

Seems not to be tail recursive.

Idem for unary suffix on page 249

Third paragraph of text starting with “The second tree illustrates”, line 3

call to expr[4] matches the 2 and then evaluates {4 >= $4}?

As $_p has value 4, it should be {4 >= 4}? (without $), similar to {4>=0}? in the second line of the next paragraph.

Idem for “Predicate {3 >= $5}? deactivates” in the next paragraph, line 6, which should read {3 >= 5}?.

Fourth paragraph of text starting with “The third parse tree”, line 4

inside the call to expr[5], The parser >>>>> (not a new sentence)
inside the call to expr[5], the parser

Among the commands at the bottom of the page :

$ svn add T.g4
A T.g4

Why add to subversion ? —> -bash: svn: command not found.

Please take note that no error is indicated by ANTLR for

@lexer::member {String txt;}

(members without s). “String txt;” is just not generated in the lexer and only the compiler complains.

Paragraph before the code snippet “reference/Nongreedy.g4”

When ANTLR sees an alternatives —> sees alternatives or sees an alternative ???

I don’t understand the third paragraph of text :

Note that this version is looser due to the wildcard. It allows improperly
nested actions such as { { }.

I see no difference between the 2 grammars, both accept improperly nested actions such as { { } :

$ grun Action tokens -tokens
{ { }
[0,0:6='{ { }',<1>,1:0] [1,8:7=‘’,<–1>,2:0]

$ grun ActionWild tokens -tokens
{ { }
[0,0:6='{ { }',<1>,1:0] [1,8:7=‘’,<–1>,2:0]

In the listing for EContext near the bottom of the page, the 4th component, EContext b, has a comment that refers to “label a” instead of “label b”.

Bottom of the page before parser grammar P :

A parser grammar cannot reference literal ‘&’, but it can reference the name the tokens
—> the name -> of <- the tokens ???

Top of the page after the second command antlr4 P.g4
Small detail in the error message : P, not S2 :

error(126): P.g4:3:4: cannot create …… instead of
error(126): S2.g4:4:4:
_

In listing of errors/BailErrorStrategy.java continued from previous page, the first line of the Javadoc comment: recommend deleting word “wrote” and uncapitalizing “Re-throw” yielding “Instead of recovering from exception e, re-throw it wrapped”. Changes need to be made in both the code and the text, of course.

First line :
Everything else in Expr.g4 ……. should be
Everything else in CSV.g4

Not the same output. Paragraph 6 : I have

$ grun CSV file users.csv
values = {}
header: ‘User, Name, Dept’
…

instead of :

$ grun CSV file users.csv
header: ‘User, Name, Dept’

To remove empty values after processing hdr, we can change the @after action to :

if ($values != null && ! $values.isEmpty()) System.out.println(“values = ” + $values);

First paragraph of § 9.3, line 3 :

run to run). This not as unusual as this sounds.

I’m not Anglophone, but I would rather expect :

This is not as unusual as it sounds.

Not the same output. I hate to bother you, but I’m eagle-eyed.

Top of page, first line of output :

line 1:3 extraneous input ‘=’ expecting INT

I have :

line 1:3 extraneous input ‘=’ expecting {CHAR, INT}

Top of page.

$ java TestEnum -java5
enum Temp { HOT, COLD }
enum Temp
line 2:0 no viable alternative at input ‘’
$

Adding EOF to prog removes that ‘’ error :

prog: ( stat
_ _ _ _ | enumDecl
_ _ _ _ )+
_ _ _ _ EOF
_ _ _ _ ;

$ antlr4 -no-listener Enum.g4
$ javac Enum*.java TestEnum.java
$ java TestEnum -java5
enum Temp { HOT, COLD }
enum Temp
$

Second paragraph of text ending with :

token. (More on this inThe reference chapter: Section 14.5, Semantic Predi- cates, on page 278.)
_

Bottom of the page. Oops ! Forgot to compile TestEnum2.java

➾ $ javac Enum2*.java . . . . . . . . . . . . . . . . .instead of
➾ $ javac Enum2*.java TestEnum2.java

Sorry, sorry, not the same output. Top of the page :

$ java TestEnum2
➾ enum Temp { HOT, COLD }
➾EOF
line 1:5 missing ‘=’ at ‘Temp’
line 1:10 mismatched input ‘{’ expecting ‘;’
line 1:15 mismatched input ‘,’ expecting ‘=’
line 1:22 mismatched input ‘}’ expecting ‘=’

instead of

$ java TestEnum2
➾ enum Temp { HOT, COLD }
➾EOF
line 1:5 missing ‘=’ at ‘Temp’
line 1:15 mismatched input ‘,’ expecting ‘=’
line 1:22 mismatched input ‘}’ expecting ‘=’
$ java org.antlr.v4.Tool
ANTLR Parser Generator Version 4.0b2

It is as if it has resynchronized on ID HOT, skipping the {. Is it the case, as explained at the bottom of page 156 ?

“Here is how ANTLR uses those ideas together in a nutshell: parsers perform single-token insertion and single-token deletion upon mismatched token errors if possible. If not, parsers gobble up tokens until they find a token that could reasonably follow the current rule and then return, continuing as if nothing had happened.”

Middle of the page.

ID : [a-zA-Z]+ {if (java5 && getText().equals(“enum”)) setType(ENUM);} ;

Enum2Lexer.java:78: cannot find symbol
symbol : variable ENUM
location: class Enum2Lexer
\t\t\tcase 0 : if (java5 && getText().equals(“enum”)) setType(ENUM); break;
. . . . . . . . . . . . . . . . . . the caret under ENUM . . . . . . . . . . . . . . . . . . . ^
1 error

Solution :

tokens {ENUM} . . . . . . at the top of the grammar, and ENUM qualified by the name of the parser :
ID : [a-zA-Z]+ {if (java5 && getText().equals(“enum”)) setType(Enum2Parser.ENUM);} ;

Link in the middle of the page. Same remark as page 88 :

Language Ambiguities are Bad, Umkay, on page 196.) . . . . >>>>
Language Ambiguities are Bad, Umkay, above

Last sentence :

The following parse trees (created using grun-ps option)

See also page 86. On Mountain Lion, the command

$ grun PredCppStat stat -ps xx

displays an icon in the dock, the same cup of coffee as antlrworks2, which disappears after two seconds. To obtain a visible, stable parse tree, I have to use the -gui option :

$ grun PredCppStat stat -gui

Last line. The command must be
antlr4 IDKeyword.g4 . . . . . instead of
antlr IDKeyword.g4

Lack of definition of the token type.

I have a grammar that recognizes statements written in capitals. I try to adapt it to recognize lower-case letters, as a modern compiler can do. So the grammar must be case insensitive. A statement looked like this :

statement
…..
. . . | stmtPUT
…..
stmtPUT : ‘PUT’ ( ‘PAGE’ | ‘SKIP’ ( ‘(’ expr ‘)’ )? ) ;

I have experimented all sorts of transformations, such as :

statement
…..
. . . | {getCurrentToken().getText().equals(“PUT”)}? stmtPUT

ID : [a-zA-Z]+ {setText(getText().toUpperCase());}

line 4:9 no viable alternative at input ‘PUT SKIP’
Lower-case statements are not recognized. It took me some time to realize that the input “put skip;” creates two tokens of type ID which, even translated to upper-case, will never match “native” ‘PUT’ and ‘SKIP’, because they have their proper token type. At least I have a vague recollection of v3.2 I did two years ago. A search with “token type” in v4 shows that this notion is never clearly explained (just with some examples) : three occurrences in section 4.6, a little more in section 9.3.

It would be good to reproduce part of the section 2.6 of the previous book (starting at page 27, PDF P4.0 page 44) :
page 28 note 9 : In practice, tokens consist of at least two pieces of information: the token type (which lexical structure) and the text matched by the lexer.
page 29 last paragraph : Further, the lexer can group related tokens into token “classes” or token types such as INT (integers), ID (identifiers), FLOAT (floating-point numbers), and so on. The lexer groups vocabulary symbols into types when the parser doesn’t care about the individual symbols, just the type.

It would also be good to emphasize that a parser rule containing ‘token’ strings as in

stat: ‘while’ ‘(’ expr ‘)’ stat
. . . | ‘if’ expr ‘then’ stat (‘else’ stat)?

creates hidden tokens (I suppose), as if we had explicit lexer rules :

IF : ‘if’ ;
THEN : ‘then’ ;
ELSE : ‘else’ ;
WHILE : ‘while’ ;

and, most important, these rules have precedence over the ID rule which is usually near the end of the grammar. (The order is mentioned once on page 194 for explicit lexer rule : ENUM: ‘enum’ {java5}? ; // must be before ID).

And also that the parser matches by token type, not string content … Well, I should know that, I have written a PL/I parser by hand. But it was five years ago.

Top of page. The first command

antlr4 SimplyPy.g4 . . . . . must be
antlr4 SimplePy.g4
……………….^ SimplE not Y

Not the same output.

[0,8:8='\ ',<11>,1:8] [1,9:9=‘f’,<4>,2:0]
[2,10:10='(',<6>,2:1] [3,11:11=‘1’,<5>,2:2]
[4,12:12=',',<1>,2:3] [5,25:25=‘\
’,<11>,2:16]
[6,26:26='\ ',<11>,3:0] [7,29:29=‘2’,<5>,4:2]
[8,30:30=',',<1>,4:3] [9,44:44=‘\
’,<11>,4:17]
[10,77:77='\ ',<11>,5:32] [11,80:80=‘3’,<5>,6:2]
[12,81:81=')',<7>,6:3] [13,94:94=‘\
’,<11>,6:16]
[14,95:95='\ ',<11>,7:0] [15,96:96=‘g’,<4>,8:0]
[16,97:97='(',<6>,8:1] [17,98:98=‘)’,<7>,8:2]
[18,108:108='\ ',<11>,8:12] [19,109:109=‘\
’,<11>,9:0]
[20,110:110='1',<5>,10:0] [21,111:111=‘+’,<2>,10:1]
[22,114:114='2',<5>,11:0] [23,115:115=‘+’,<2>,11:1]
[24,118:118='3',<5>,12:0] [25,119:119=‘\
’,<11>,12:1]
[@26,120:119=‘’,<–1>,13:2]
line 2:16 no viable alternative at input ‘\
’
line 4:3 extraneous input ‘,’ expecting NEWLINE
line 6:3 extraneous input ‘)’ expecting NEWLINE

I did some research :

@lexer::members {
int nesting = 0;
Boolean showNesting(String type) {System.out.println(type + " nesting=" + nesting); return false;}
}

LPAREN : ‘(’ {nesting; System.out.println(“nesting now ” + nesting);} ;

RPAREN : ‘)’ {nesting—; System.out.println(“nesting now ” + nesting);} ;

IGNORE_NEWLINE
: ‘\\r’? ‘\
’ {showNesting(“i”) || nesting > 0}? -> skip
;

NEWLINE
// : ‘\\r’? ‘\
’
// : ‘\\r’? ‘\
’ {showNesting(“n”) || true}?
// : ‘\\r’? ‘\
’ {nesting 0}? : '\\r'? '\ ' {showNesting("n") || nesting 0}?
;

i nesting=0
n nesting=0
nesting now 1
i nesting=1
n nesting=1
i nesting=1
n nesting=1
i nesting=1
n nesting=1
i nesting=1
n nesting=1
nesting now 0
i nesting=0
n nesting=0
i nesting=0
n nesting=0
nesting now 1
nesting now 0
i nesting=0
n nesting=0
i nesting=0
n nesting=0
i nesting=0
n nesting=0
[0,8:8='\ ',<11>,1:8] [1,9:9=‘f’,<4>,2:0]
[2,10:10='(',<6>,2:1] [3,11:11=‘1’,<5>,2:2]
[4,12:12=',',<1>,2:3] [5,29:29=‘2’,<5>,4:2]
[6,30:30=',',<1>,4:3] [7,80:80=‘3’,<5>,6:2]
[8,81:81=')',<7>,6:3] [9,94:94=‘\
’,<11>,6:16]
[10,95:95='\ ',<11>,7:0] [11,96:96=‘g’,<4>,8:0]
[12,97:97='(',<6>,8:1] [13,98:98=‘)’,<7>,8:2]
[14,108:108='\ ',<11>,8:12] [15,109:109=‘\
’,<11>,9:0]
[16,110:110='1',<5>,10:0] [17,111:111=‘+’,<2>,10:1]
[18,114:114='2',<5>,11:0] [19,115:115=‘+’,<2>,11:1]
[20,118:118='3',<5>,12:0] [21,119:119=‘\
’,<11>,12:1]
[@22,120:119=‘’,<–1>,13:2]

There are still two extra \
after f and g.
I don’t understand why the showNesting(“n”) from NEWLINE is executed even if nesting > 0.

During the first reading, I had noted things to revisit. If you put this piece of code directly into a grammar :

grammar Decl;
decl
: decl ‘[’ ‘]’ …

and execute it with an input file t.decl containing *(*a)[][] (only one line, no newline), then :

$ grun Decl decl t.decl
No method for rule decl or it has arguments

A look in DeclParser.java shows the argument :

public final DeclContext decl(int _p) throws RecognitionException {

Introducing a start rule solves the problem :

grammar Decl;
start : decl ;
decl : …..

Now the parser :

public final StartContext start() throws RecognitionException {
…
try {
…
setState(4); decl(0);

Re-reading § 13.2 makes it clear, the parameter _p is added by the transformation :

$ antlr4 -Xlog Decl.g4
wrote ./antlr-2012-10-30-07.00.59.log
$ cat antlr-2012-10-30-07.00.59.log
…
2012-10-30 07:00:59:014 left-recursion LogManager.java:48 decl[int _p]
: ( {} ‘*’ decl[3]
| ‘(’ decl ‘)’
| ID
)
( {4 >= $_p}? ‘[’ ‘]’
)*
;

It also shows that there is already a loop to process the multiple []. I had tried with :

decl
: decl (‘[’ ‘]’)*

which ended up with :

Exception in thread “main” java.lang.reflect.InvocationTargetException
…
Caused by: java.lang.StackOverflowError

How to interpret the diagnostics message ?

Given a simple input :

DECLARE MONTHS (12) CHAR (9) INIT (‘January’, ‘February’, …);

the -gui option (a very powerful tool, indeed) shows that it is correctly parsed. But the -diagnostics option displays various messages :

line 2:45 reportAttemptingFullContext d=123, input=‘’January’,’
line 2:45 reportAmbiguity d=123: ambigAlts={1, 3}, input=‘’January’,’
line 2:57 reportAttemptingFullContext d=123, input=‘’February’,’
line 2:57 reportAmbiguity d=123: ambigAlts={1, 3}, input=‘’February’,’
…
line 7:12 reportAttemptingFullContext d=111, input=’’
line 7:12 reportContextSensitivity d=111, input=’’

On contrary to the example given on this page, the alternative chosen by the parser is not displayed. Is it an indication that something is wrong in the grammar ?

Solved. Found an ambiguity with STRING, keyword of the statement PUT STRING (= C sprint()) and the usual STRING :

factor
: keywordOrID
| INT
| STRING
| ‘(’ expr ‘)’
;

keywordOrID
: keywords
| ID
;

keywords
/* PL/I keywords are not reserved words. They are recognized as keywords
by the compiler only when they appear in their proper context.
In other contexts they may be used as programmers-defined identifiers :

DECLARE SEPARATOR CHAR (10) INIT (‘==’); -> initial attribute
INIT : PROCEDURE; -> identifier INIT, label of the procedure
*/
: statementNames
| statementKeywords
| statementAttributes
;

statementNames
: CALL | CLOSE | DECLARE | IF | PROCEDURE | PUT
;

statementKeywords
: ELSE | ENVIRONMENT | FILE | INIT | … | SKIP | STRING | THEN | UNLOAD
;

Renaming a parser rule from keywordOrID to IDorKeyword, because it sounds better, can have quite brutal effects :

$ antlr4 Pl1.g4
GrammarTreeVisitor.g: node from line 690:9 required (…)+ loop did not match anything at input ‘keywords’
can’t find rule grammarSpec or tree structure error: …
}) (RULES (RULE T_234 (BLOCK (ALT ’ENDFILE’))) (RULE T233 (BLOCK (ALT ’COMPILETIME’))) (RULE T232 (BLOCK (ALT ’ALL’))) (RULE T231 (BLOCK (ALT ’DO’))) (RULE T230 (BLOCK (ALT ’*’))) (RULE T229 (BLOCK (ALT ’LABEL’))) (RULE T228 (BLOCK (ALT ’ACOS’))) (RULE T227 (BLOCK (ALT ’ERFC’))) (RULE T226 (BLOCK (ALT ’SEQUENTIAL’))) (RULE T_225 (BLOCK (ALT ‘OPTIONS’))) (BLOCK (ALT … 30 lines
java.lang.reflect.InvocationTargetException
\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
\tat java.lang.reflect.Method.invoke(Method.java:597)
\tat org.antlr.v4.parse.GrammarTreeVisitor.visit(GrammarTreeVisitor.java:191)
… 3 more screens

Forewarned is forearmed.

Paragraph after the code snippet lexmagic/XMLLexer.g4, line 4.

Please note a tiny discrepancy between :

notation declarations of the form: <>.

and the fact that the second exclamation point does not appear neither in the comment nor in the DTD rule.

/ Scarf all DTD stuff, Entity Declarations like <!ENTITY …>,
* and Notation Declarations <!NOTATION …>
*/
DTD : ‘<!’ .* ‘>’

Command above the graphic :

$ grun XML document -ps /tmp/t.ps XML-inputs/entity.xml

As already noted on pages 86 and 199, you probably use the -ps option to produce the file that will create, with some other tool, the graphic incorporated into the PDF.

For us reader wanting to see the output of the command as in the book, we need the -gui option :

grun XML document -gui XML-inputs/entity.xml

Also note that, in order to obtain the same output, the reader must edit the file and remove the copyright notice.

! Excerpted from “The Definitive ANTLR 4 Reference”,
! published by The Pragmatic Bookshelf.
! Copyrights ….

It would be good not to insert the copyright notice in data files serving as input to run the examples, as it already happens for some of them (e.g. in examples).

3rd Par beginning “Let’s break that …”, next to last line has “Java Stringsplit()” with missing period. Should be “Java String.split”.

Paragraph 7 :

$r and $rl evaluate to ParserRuleContext objects of type RRuleContext for rule name r and rule label rl. $rll evaluates to List for rule list label rll.

Considering that (from CSV.g4)

file
: hdr ( rowsrow[$hdr.text.split(“,”)] {$i;} )

generates

\tpublic static class FileContext extends ParserRuleContext {
\t\tpublic RowContext row;
\t\tpublic List rows = new ArrayList();

If italic R stands for the rule name, RContext is better than RRuleContext, hence :

$r and $rl evaluate to ParserRuleContext objects of type RContext for rule name r and rule label rl. $rll evaluates to List for rule list label rll.

Well, concerning “parser rule containing ‘token’ strings” I must have missed section 4.5 page 73 third paragraph from bottom :

For keywords, operators, and punctuation, we don’t need lexer rules because we can directly reference them in parser rules in single quotes like ‘while’, ‘*’, and ‘’.

Third paragraph from bottom :

For keywords, operators, and punctuation … in single quotes like ‘while’, ‘*’,

Did I miss something ? Or it should be emphasized that such a standalone token must not appear in another token rule.

Given the input file ri.java :

$ cat ri.java
import org.antlr.v4.runtime.tree.*;
import org.antlr.v4.runtime.Token;

public interface RenameListener extends ParseTreeListener {
\tvoid enterStmt(RenameListenerParser.StmtContext ctx);
\tvoid exitStmt(RenameListenerParser.StmtContext ctx);
}

the following grammar does not work because the dot in RenameListenerParser.StmtContext (line 5) is not matched by the one in SPECIAL (type=6) of rule rest, but by the ‘.’ (type=2) appearing first in rule xpackage.

grammar ExtractImport;

file : line* EOF ;

line: ‘import’ xpackage ‘;’ NL
| rest NL
| NL
;

xpackage : ID ( ‘.’ ( ID | ’’ ) ) ;

rest : ( ID | SPECIAL | TAB)+ ;

ID : LETTER ( LETTER | DIGIT | ‘_’ )* ;

fragment LETTER : [a-zA-Z] ;
fragment DIGIT : [0-9] ;

SPECIAL : ( ‘(’ | ‘)’ | ‘-’ | ‘_’ | ‘.’ | ‘>’ | ‘<’ | ‘{’ | ‘}’)+ ;

NL : ‘\\r’? ‘\
’ ;
TAB : ‘\\t’ ;
WS : ’ ’ -> channel(HIDDEN) ;

==

$ grun ExtractImport file -tokens ri.java
[0,0:5='import',<1>,1:0] [1,6:6=’ ‘,<9>,channel=1,1:6]
[2,7:9='org',<5>,1:7] [3,10:10=’.‘,<2>,1:10]
[4,11:15='antlr',<5>,1:11] ... [49,155:174=’RenameListenerParser’,<5>,5:16]
[50,175:175='.',<2>,5:36] [51,176:186=‘StmtContext’,<5>,5:37]
…
line 5:36 extraneous input ‘.’ expecting {ID, SPECIAL, NL, ‘\t’}
line 6:35 extraneous input ‘.’ expecting {ID, SPECIAL, NL, ‘\t’}

==

With :

xpackage : ID ( DOT ( ID | ‘’ ) ) ;
SPECIAL : ( ’(’ | ‘)’ | ‘-’ | ‘_’ | DOT | ‘>’ | ‘<’ | ‘{’ | ‘}’)+ ;
DOT : ‘.’ ;

DOT has type=6 and the dot in RenameListenerParser.StmtContext is matched by SPECIAL (type=5).

[49,155:174='RenameListenerParser',<4>,5:16] [50,175:175=‘.’,<5>,5:36]
[@51,176:186=‘StmtContext’,<4>,5:37]

So the rule rest matches the line. OK, but now :

line 1:10 mismatched input ‘.’ expecting {’;‘, ’.‘}
line 2:10 mismatched input ’.’ expecting {’;‘, ’.’}

The dot in the import statement is captured by SPECIAL (type=5), and the rule xpackage, which needs a DOT of type 6 no longer matches :

[2,7:9='org',<4>,1:7] [3,10:10=‘.’,<5>,1:10]
[@4,11:15=‘antlr’,<4>,1:11]

===

The solution is to have the dot in the form of a standalone token ‘.’ or as a DOT lexer rule, repeated if necessary as in rest, but never embedded in another token rule :

xpackage : ID ( ‘.’ ( ID | ‘’ ) ) ;
rest : ( ID | SPECIAL | ’.’ | ‘;’ | TAB)+ ;
SPECIAL : ( ‘(’ | ‘)’ | ‘-’ | ‘_’ | ‘>’ | ‘<’ | ‘{’ | ‘}’)+ ;

Bottom, Keywords. It seems that Java does not like if a rule is named package :

line: ‘import’ package ‘;’ NL
| rest NL
| NL
;

package : ID ( DOT ( ID | ’’ ) ) ;

(same grammar as my post page 73)

$ antlr4 ExtractImport.g4
$ javac ExtractImport*.java
ExtractImportParser.java:102: expected
\t\tpublic PackageContext package() {
…………………………………………^
ExtractImportParser.java:102: illegal start of type
\t\tpublic PackageContext package() {
…………

First complete par beginning “unary suffix” also refers to rule “expression” but should be “expr”.

As well as any other Java reserved word …

stat : ( classDef | methodDef | if | unless | ‘begin’ )

if : ‘if’ expr ( ‘then’ | ‘:’ ) statement ( ‘else’ statement )?

===>

extract_listenerParser.java:344: expected
\t\tpublic IfContext if() {
\t\t ^
extract_listenerParser.java:344: illegal start of type
\t\tpublic IfContext if() {

George, it has been a mystery for me too. Now I think I have understood. If you change the field rule to :

field
: TEXT {System.out.println($field.text + " is a TEXT“);}
| STRING {System.out.println($field.text + ” is a STRING");}
|
;

you get :

Mid Bonus is a TEXT
June is a TEXT
“$2,000” is a STRING

as you guessed from the token type <4>. Now change data.csv to put a dot instead of the comma in the amount field :

Mid Bonus,June,“$2.000”

and run again :

Mid Bonus is a TEXT
June is a TEXT
“$2.000” is a TEXT

I would add, in page 15, that ambiguity is resolved in favour of the first rule as long as the input flow satisfies both rules until the right edge of the lexer rule / implicit token, as in the case of ‘begin’/ID with a `begin`input, or TEXT/STRING with a `“$2.000”` (with a dot) input. In the case of `“$2,000”` with a comma, the input is ambiguous only in the first three characters `"$2`, the lexer has the choice. Then comes the comma and the TEXT rule is out because it negates the comma. The only remaining choice is STRING, which is greedy at this position and loops until the next quote.

I suggest to add a section regarding large grammars (many keywords and many parser rules) and suggestions on how to create it to avoid exceeding the method limit size in java.

The following sections describe some approaches, but there might be other approaches not listed.
Section 10.3 of the book (Recognizing Languages whose Kewwords aren’t Fixed) mentions grammar with many keywords.
Section 15.2 subsection “Grammar Imports”. I would suggest to add a working example of a grammar build using imports of multiple grammars and lexers.

The approaches should have a note stating whether they result in smaller code (to not exceed the java limit) or they improve readability or both.
An example of large grammar is here: www.antlr.org/pipermail/antlr-interest/2012-November/045698.html, but it has a “code too large” error.

The grammar for Java 6 provided with the book’s example programs, tour/java.g4, incorrectly lists the ^ operator as being right-associative. In Java ^ is a bitwise and logical operator, not an exponent operator. The Java 7 Language Reference says on page 518 (or see docs.oracle.com/javase/specs/jls/se7/html/jls-15.html#jls-15.22), speaking of the bitwise and logical operators &, ^{, and |, “Each of these operators is syntactically left-associative (each groups left-to-right)”. (The java.g4 file correctly makes the}= assignment operator right-associative.)

It doesn’t concern this page, but code/extras/CPPBaseLexer.g4

$ antlr4 CPPBaseLexer.g4
warning(131): CPPBaseLexer.g4:21:14: greedy block ()* contains wildcard; the non-greedy syntax ()*? may be preferred

I couldn’t get rid of that error, even after adding a ? after ~[\\r\
]*.

OTHER_CMD : ‘#’ ~[\\r\
]*? ‘\\r’? ‘\
’ ; // can’t use .*; scarfs \
\
after include

The command “javac *.java” cannot work as is, without specifying a CLASSPATH for the ANTLR jar file.

Extremely minor typo: At the bottom of the page, the “fragment ESC” the “]” in <[btrn“\\\\]> is italicized, when it shouldn’t be. Ditto for the following ”;"

Also, the comment (“// \\b, |t, \
etc…” is in blue. But the comment above for “ID” (“// From C language”) is in green (which is what you seem to use other places). So presumably the ESC comment should also be in green.

See, I told you it was extremely minor.

Sorry for posting this here, rather than some other place. The only other I found was on groups.google.com that required me to register, which I didn’t want to do. So here it is. Ignore the “PDF Page # 1” I had to put in.

Putting “options { language = CSharp2; }” into a .g4 file, and using 4.0b3, blows up on a NullPointerException. The traceback is as follows…

error(31): ANTLR cannot generate CSharp2 code as of version 4.0b3
Exception in thread “main” java.lang.NullPointerException
\tat org.antlr.v4.analysis.LeftRecursiveRuleAnalyzer.getArtificialOpPrecRule(LeftRecursiveRuleAnalyzer.java:244)
\tat org.antlr.v4.analysis.LeftRecursiveRuleTransformer.translateLeftRecursiveRule(LeftRecursiveRuleTransformer.java:132)
\tat org.antlr.v4.analysis.LeftRecursiveRuleTransformer.translateLeftRecursiveRules(LeftRecursiveRuleTransformer.java:89)
\tat org.antlr.v4.semantics.SemanticPipeline.process(SemanticPipeline.java:93)
\tat org.antlr.v4.Tool.processNonCombinedGrammar(Tool.java:393)
\tat org.antlr.v4.Tool.process(Tool.java:381)
\tat org.antlr.v4.Tool.processGrammarsOnCommandLine(Tool.java:345)
\tat org.antlr.v4.Tool.main(Tool.java:192)

“At this point, we’re just trying to get the gist of the development process… ” should be “At this point, we’re just trying to get the list of the development process…”

It seems that all texts, brackets, braces, colons, semi-colons, \
, etc (particularly in the grammars) between single apostrophes are italicized :

page 6 : r : ‘hello’ ID ;
pages 14 : | ID ‘(’ ‘)’ ‘;’ // function call statement ;
expr: ID ‘(’ ‘)’
page 15 : BEGIN : ‘begin’ ;
page 24 : init : ‘{’ value (‘,’ value)* ‘}’ ;
page 35 : | ‘(’ expr ‘)’
NEWLINE:‘\\r’? ‘\
’
page 38 : NEWLINE:‘\\r’? ‘\
’
page 39 : | ‘(’ expr ‘)’
page 41 : idem
page 46 : : ‘class’ Identifier typeParameters? (‘extends’ type)?
: type Identifier formalParameters (‘[’ ‘]’)* methodDeclarationRest
| ‘void’ …
page 49 : NL : ‘\\r’? ‘\
’ ;
page 52 et seq. : COMMENT : ‘<!—’
EntityRef : ‘&’ etc, etc
page 62 : packageSpec : ‘package’ identifier ‘;’ ;
classDefinition :
‘class’ «optional superclassSpec optional implementsClause classBody» ;
superclassSpec : ‘super’ identifier ;
implementsClause :
‘implements’ «one or more identifiers separated by comma» ;
classBody : ‘{’ «zero-or-more members» ‘}’ ;
page 64 : retr : ‘RETR’ INT ‘\
’ ;
page 65 : file : (row ‘\
’)* ; + all ‘:’ and ‘;’
page 66 : type: ‘float’ | ‘int’ | ‘void’ ; // user-defined types
vector : ‘[’ INT+ ‘]’ ;

I stop the inventory on page 67 because there are too many.

Also outside grammars as on page 71 :
expr : ‘(’ expr ‘)’ | ID ;
classDef : ‘class’ ID ‘{’ (classDef|method|field) ‘}’ ;

§ 3.1 second paragraph “The first step”, second line :

compares the input against a language’s syntactic rules.

Not sure about a — > rule_S_ ] Could be :

compares the input against a SET OF language’s syntactic rules.

second line :

of you reading this electronically, you can click on the gray box

Hmm, it’s gray in the printed book, lime green in the PDF.

Middle of the page, paragraph

Running the ANTLR tool on Hello.g4 generates an executable recognizer embodied by HelloParser.java and HelloLexer,

To be coherent —> and HelloLexer.java

First line

1. start the TestRig on grammar Hello had rule r

“had rule r” sounds strange

First line : missing (s) in [input-filename]

$ grun
java org.antlr.v4.runtime.misc.TestRig GrammarName startRuleName
[-tokens] [-tree] [-gui] [-ps file.ps] [-encoding encodingname]
[-trace] [-diagnostics] [-SLL]
[input-filename(s)] <———————-
…

In the directory

code/starter

there is a file

build.properties

that may have been dropped accidentally here.

Line 4 The “and” disturbs me in

natives on lines 11 and 12 that recursively invoke expr and on the left edge.

I would expect either of

recursively invoke expr on the left edge.
recursively invoke expr on the left and right edge.

By the way, what is an edge ? First occurrence here, next (right edge) on page 41, then can be deduced from the explanation on page 73, first paragraph.

Middle of the page :

javac *.java

For people working directly in the tour directory, this will compile everything (all other examples).

It would be preferable to compile only LibExpr :

javac LibExpr*.java

Second execution of java Test, not the same output.

I have double-checked that I have the latest version, downloaded from Github on Monday 3 December and rebuilt.

$ alias
alias antlr4=‘java -jar /usr/local/lib/antlr-4.0b4-3déc-complete.jar’
$ echo $CLASSPATH
.:/usr/local/lib/antlr-4.0b4-3déc-complete.jar

$ antlr4 ArrayInit.g4
$ javac ArrayInit.java T.java
$ java Test
{1,2
➾EOF
line 2:0 missing ‘}’ at ‘’
(init { (value 1) , (value 2) <missing ‘}’>)

Bottom, not the same output.

$ antlr4 Expr.g4
$ antlr4 LibExpr.g4
$ javac Expr.java LibExpr.java
$ java ExprJoyRide
(1+2
3
➾EOF
line 1:4 mismatched input ‘\
’ expecting {’)‘, ’+‘, ’*‘, ’-‘, ’/’}
(prog (stat (expr ( (expr (expr 1) + (expr 2)) <missing ‘)’>) \
) (stat (expr 3) \
))

Small difference between the file code/tour/Expr.g4 and the book :

ID : [a-zA-Z]+ ; // match identifiers
ID : [a-zA-Z]+ ; // match identifiers

Near bottom. Small difference between the file code/tour/ExprJoyRide.java and the book :

ParseTree tree = parser.prog(); // parse; start at prog
System.out.println(tree.toStringTree(parser)); // print tree as text

ParseTree tree = parser.prog(); // parse; start at prog
System.out.println(tree.toStringTree(parser)); // print tree as text

Third paragraph from bottom.

Visually, the internal grammar representation of rule group <——

Isn’t it rather rule sequence ?

Also in the picture.

End of third paragraph :

For example, we might want to turn the enum keyword on and off in a Java grammar to parse different versions of the language.

This may not be noticed by most readers, but the beginning of this chapter
First … Second … Third … Fourth … Finally
describes the content of the quick tour. Turning enum on or off is described somewhere else in the book. In 4.4 Making Things Happen During the Parse, the quick tour selects a column with an external parameter.

File code/tour/XML.g4 contains :

lexer grammar XMLLexer;

Hence :

$ antlr4 XML.g4
error(8): XML.g4:1:14: grammar name XMLLexer and file name XML.g4 differ

Should be lexer grammar XML;

First line :

sequence with Terminator and sequence with separator.

Is there a reason for capital T (erminator) and not S (eparator) ?

Third line from bottom :

STRING: ‘"’ .* ‘"’ ;

Shouldn’t it be

STRING: ‘"’ .*? ‘"’ ;

to avoid the warning

warning(131): Log.g4:6:13: greedy block ()* contains wildcard; the non-greedy syntax ()*? may be preferred

Also in gramar snippet

STRING: ‘"’ .* ‘"’ ;

Chapter 6 finished without using code/examples/Hello.g4 (already used in code/install).

Third line, what is this comment after ?

parrt

In the beginning of file code/listeners/JSON2XML.java :

parrt

and the execution shows nothing :

$ java JSON2XML t.json
…

parrt

Bottom after the grammar snippet :

Given one too many integers, as in the following session, we see an error message and get error recovery that throws out the extra comma and integer.

Is one correct ? (there are two integers and one comma too many).

At the end of the sentence —-> integers in plural ?

End of second to last paragraph “A word of caution” :

it’s not a good idea use a semantic predicate.

My English may be old fashioned, but I would say “to use …”

it’s not a good idea TO use a semantic predicate.

Have finished chapter 9 without using code/errors/Enum.g4 (which will be used in chapter 11 and can be found in code/predicates).

Finished chapter 10 without using code/action/TestFuzzyJava.java.

Middle of the page :

enum Temp { HOT, COLD } . . . . . . . . . // in file Temp.java

There is no such file in code/predicates.

Middle of the page, first enumDecl
: ‘enum’ id ‘{’ id (‘,’ id)* ‘}’

There is no name=, hence {System.out.println(“enum ”+$name.text);} would fail.

Paragraph 4, truncated sentence :

In this case, ANTLR’s error recovery realizes it doesn’t have a valid assignment and scarfs tokens until it finds something that can

Messages have disappeared.

It works in 4.0b3 :

$ alias
alias antlr43=‘java -jar /usr/local/lib/antlr-4.0b3-complete.jar’
$ antlr43 PredCppStat.g4
$ javac PredCppStat*.java
$ grun PredCppStat stat -diagnostics
f(i);
line 1:4 reportAttemptingFullContext d=0, input=‘f(i);’
line 1:1 reportContextSensitivity d=0, input=‘f(’
expr f(i)

I have just downloaded and rebuilt, copied dist/antlr-4.0b4-complete.jar to /usr/local/lib, renamed to antlr-4.0b4-5dec-complete.jar :

$ echo $CLASSPATH
.:/usr/local/lib/antlr-4.0b4-5dec-complete.jar
$ alias
alias antlr4=‘java -jar /usr/local/lib/antlr-4.0b4-5dec-complete.jar’
$ antlr4
ANTLR Parser Generator Version 4.0b4
…
$ antlr4 PredCppStat.g4
$ javac PredCppStat*.java
$ grun PredCppStat stat -diagnostics
f(i);
➾EOF
expr f(i)
$

First execution of grun Java tokens -tokens.

Using code/tour/Java.g4, I have more tokens (spaces) than in the book :

[0,0:0='i',<98>,1:0] [1,1:1=’ ‘,<100>,channel=1,1:1]
[2,2:2='=',<25>,1:2] [3,3:3=’ ‘,<100>,channel=1,1:3]
[4,4:4='1',<91>,1:4] [5,5:5=’ ‘,<100>,channel=1,1:5]
[6,6:6='>',<81>,1:6] [7,7:7=’>’,<81>,1:7]
…

The graphic at the bottom is not up-to-date, there are now two more stat — \
(6 newlines instead of 4 in the previous version).

Second line :

The other three newlines terminate statements.

should be adapted : 3 NL terminate expr, and 2 additional NL are empty lines.

The TestSimple runs in 4.0b3, but no longer with the latest release :

$ alias
alias antlr4=‘java -jar /usr/local/lib/antlr-4.0b4-5dec-complete.jar’
$ echo $CLASSPATH
.:/usr/local/lib/antlr-4.0b4-5dec-complete.jar
$ javac Simple*.java TestSimple.java
SimpleLexer.java:68: cannot find symbol
symbol : method create(SimpleLexer,int,java.lang.String,int,int,int,int,int)
location: interface org.antlr.v4.runtime.TokenFactory<capture#9 of ?>
return factory.create(this, ttype, text, Token.DEFAULT_CHANNEL,
^ [caret under the dot in factory.create]
1 error

Orphaned closing parenthesis at the end of the first sentence :

These unbuffered streams are useful when efficiency is the top concern. You can even combine them with the technique from the previous section).

End of second paragraph “Any other alternative” :

ANTLR collects them and deals with them properly.
Assume that the order of all other alternatives matters.
Here are a few sample primary expression alternatives:

The sentence Assume sounds strange, suspended without link.

properly and assumes ? or Let’s assume?

I have an EOF in the message :

line 3:0 extraneous input ‘;’ expecting {, INT, ID}

instead of

line 3:0 extraneous input ‘;’ expecting {INT, ID}

Bottom, just to keep the same rhythm and music :

Parsers can import parsers.

should be

Parser grammars can import parsers.

or even (more verbose but more accurate, closer to “Not every kind of grammar can import every other kind of grammar:”)

• Lexer grammars can import lexer grammars.
• Parser grammars can import parser grammars.
• Combined grammars can import lexer or parser grammars.

After the grammar reference/foo/Count.g4

The grammar itself then should be

then sounds superfluous.

Last line :

public interface AListener extends ParseTreeListener {

does not correspond to grammar T.
Also next page (AParser).

In b2 it was T :

public interface TListener extends ParseTreeListener {

EDIT

There is a file reference/A.g4 which corresponds to grammar T in the book.

So just change grammar T to grammar A

Bottom. But A.g4 is correct and does not produce these errors :

$ antlr4 A.g4
warning(125): A.g4:2:13: implicit definition of token INT in parser

I suggest file AE.g4

grammar AE;
stat: ‘return’ e ‘;’ # Return
| ‘break’ ‘;’ # Break
;
e : e ‘*’ e # e
| e ‘+’ e # Stat
| INT # Int
;

$ antlr4 AE.g4
error(124): AE.g4:5:23: rule alt label e conflicts with rule e
error(124): AE.g4:6:23: rule alt label Stat conflicts with rule stat
warning(125): AE.g4:2:13: implicit definition of token INT in parser

Paragraph 5 :

ANTLR also provide —-> provides ???

[ Incredible, a few of these typos were already in previous release(s) :) Nobody’s perfect.]

Middle of the page :

You can suffix the ?, , and + subrule operators with the nongreedy operator, which is also a question mark: ??, ?, and +?.

Should be ? instead of ?.

Explanation of FailedPredicateException : after reading several times, I understand that “This predicate” refers to FailedPredicateException, not NoViableAltException.

Confusing because This usually refers to the closest thing talked about.

Suggestion : replace “This predicate” by its name FailedPredicateException.

Middle of the page :

more Match this rule but continue looking for a token. The text for the token that matches next <——-

I suppose it refers to nextToken(). The whole sentence is difficult to understand.

The message now displays a \
(backslash n) instead of a physical newline :

$ grun Actions tokens -tokens
["foo]
line 1:0 token recognition error at: ‘["foo]\
’
[@0,7:6=‘’,<–1>,2:0]

$ grun Actions tokens -tokens
<"foo>
line 1:0 token recognition error at: ‘<"foo>\
’
[@0,7:6=‘’,<–1>,2:0]

Note : FuzzyJava is demonstrated in § 15.6.

There is no Hi.g4 in code/reference.

Created.

Middle of the page :

superClass Set the superclass of the generated parser or lexer <———-

But the lexer does not seam affected :

$ antlr4 -DsuperClass=XX Hi.g4
$ grep ‘public class’ HiParser.java
public class HiParser extends XX {
$ grep ‘public class’ HiLexer.java
public class HiLexer extends Lexer { <———-

To test tokenVocab, I had to create a file code/reference/SomeLexer.g4 and a R.g4 to test instead of P.g4 which already exists and is different.

Chapter 13 finished without using code/api/L.g4 and P.g4.
They also exit in reference and are used on page 282.

Chapter 12 finished apparently without using code/lexmagic/JavaLiterals.tokens, T.java and Tf.java.

Chapter 11 finished apparently without using code/predicates/CDecl.g4 and CExpr.g4.

EDIT page 34

Please ignore next comment on page 34, it’s OK like this (semantic predicate used to make part of the grammar disappear).

Well, T2.g4 does not exist either, but the reader can create it.

The parser does detects an ambiguous phrase —>
The parser does detect an ambiguous phrase

our next goal is to translate not just recognize initializers —>
our next goal is to translate, not just recognize initializers

or
our next goal is to translate, not just recognize them

For those with experience using a previous versions of ANTLR —>
For those with experience using a previous version of ANTLR

replace the lexical rules from original grammar —>
replace the lexical rules from the original grammar

grammar rules comprised of —>
grammar rules comprising

First line includes “we can make parser rules that matches the
records in a log file” - should be “match” rather than “matches”

The token definitions for Expr.g4 should be included in what is printed in the book because ExprParser.MULT is used in the following code snippet of TestEvaluater.java. In that case, it would also make sense to use MULT and ADD in the grammar rules, for consistency.

Maybe use ‘and override’ rather than ‘and implement’ here:

“The process is simply to extend the parser class generated
by ANTLR and implement one or more of the methods triggered by the gram-
mar:”

And similarly here:
“Notice that we don’t have to implement startFile() or finishFile() because of the
default implementations in the PropertyFileParser superclass generated by ANTLR.”

/DT/

A line of Java code “tokens.getText(ctx.type());”… there does not appear to be a TokenStream.getText(Token) method in the current version of v4.

Missing the word “to” in “Finally, we need annotate the root of the parse tree”

You run TestRig with an input file (not the keyboard) for the first time on this page. You might want to mention that the input goes in t.expr.

Change “in” to “and” in “counting from 0 in treating tabs as one character”

This page has the wrong code fragment below the sentence “As a convenience, ANTLR also generates class PropertyFileBaseListener …”. It appears to be a copy-and-paste from 7.3.

In the 3rd example the comments suggest some matches that do not match the “expr” definition given here, or you would have to assume that 2*b is a valid ID.

expr: ID ‘[’ expr ‘]’ // a[1], a[b[1]], a[(2*b[1])]
| ‘(’ expr ‘)’ // (1), (a[1]), (((1))), (2*a[1])
| INT // 1, 94117

“OK. Let’s take grammar Expr out for a joy ride. Download it […]”

It’s a book with paper, not pdf nor epub!
Download the source code for all our samples from (http: pragprog.com/titles/tpantlr2/source_code

There is an error with the rule at the end of the page:
vector : ‘[’ INT+ ‘]’

In the comment beside the rule it is described as recognizing [1] or [1 2] or [1 2 3], and so on. But the space between the numbers is not included in the rule.

The first clause for NUMBER in the JSON grammar seems to have an error because it does not allow a zero directly after the point as in the following number
1.05e+2
I tested it with this object:
{name, age, weight

In the Java pseudo-grammar here is typo in this code:
superclassSpec : ‘super’ identifier ;

The ‘super’ keyword is not allowed to use in this context, there is should be ‘extends’ keyword instead. The following is correct one:
superclassSpec : ‘extends’ identifier ;

Input “exmagic/t.cym” doesn’t have syntax highlighting, whereas the output has it. Not exactly dramatic, but this inconsistency looks a bit weird.

There is a reference to JSON2XML_ST.java source which uses StringTemplate for XML translation. But in the source code itself there is reference to the XML.stg file which absent in the book source archive.

JSON2XML_ST.java (line 45):
STGroup templates = new STGroupFile(“XML.stg”);

It would be nice if you put it to archive, since (IMHO) it’s not very easy to find it elsewhere.
Thanks.

The footnote contains a broken link. Searching mefeedia.com for “terence parr” or “jvm” both find nothing.

Running Translate on {99, 3, 451} as shown in the book works slick. But try {{1}} and Translate blows up with a NullPointerException in ShortToUnicodeString.enterValue. I’m guessing this is because the value in this case is not an integer but is a sublist {1}.

After creating the Hello.g4 sample grammar, then generating and compiling the Java, running grun leads to an error on Windows: “Can’t load Hello as lexer or parser”

In addition, running grun with no parameters, does more than what you say in the book: It prints the usage messages and a NullPointerException stack trace

TestRig.main ->
TestRig.process ->
java.lang.ClassLoader.loadClass ->
sun.misc.Launcher.$AppClassLoader.loadClass

Here are the files in the local directory:

Hello.g4
Hello.tokens
HelloBaseListener.class
HelloBaseListener.java
HelloLexer.class
HelloLexer.java
HelloLexer.tokens
HelloListener.class
HelloListener.java
HelloParser$RContext.class
HelloParser.class
HelloParser.java

Here is the Hello.g4 file:

grammar hello;\t// define a grammar called Hello
r : ‘hello’ ID;\t// match keyword ‘hello’ followed by an identifier
ID : [a-z]+;\t// match lower-case identifiers
WS : [ \\t\\r\
]+ -> skip ;\t//skip spaces, tables, newlines, \\r (Windows)

I’m running on Windows 7 Ultimate x86, 4GB RAM, 8 cores.

expr: ID ‘(’ exprList? ‘)’ // use token ID and not rule expr to build a function call

Both stat examples are missing ‘while’ ‘(’ expr ‘)’ in the second choice.

The text discusses the class PropertyFileBaseListener but instead of presenting the code for PropertyFileBaseListener, the code of the class PropertyFileBaseVisitor is presented.

Number in JSON.g4 grammar is defined wrongly. First alternative is wrong.

It doesn’t accept floating point numbers that contain zero after period char. For e.g. number: 1.01 would be parsed as two number tokens: 1.0 and 1.

It’s very easy to verify this running grun JSON value -tokens.

The parse tree diagram for JSON should have “pair” nodes instead of “element” nodes.

Section 13.7 (“Maximizing Parser Speed”) in the new ANTLR version 4 book says to use the code

parser.getInterpreter().setSLL(true);

to use the SLL parsing strategy, but there seems to be no such method on the class returned by parser.getInterpreter().

What seems to work instead is the following:

parser.getInterpreter().setPredictionMode( PredictionMode.SLL );

Diagram at the bottom of the page:

Not a typo per se, but the tokens in the diagram are not in the gray boxes which can be confusing at first sight.

In grammar fragment for Java, “superclassSpec” should be “extendsClause”, defined as:-

extendsClause : ‘extends’ Identifier ;

In the definition of the “expr” @ array choice I read "| expr ‘[’ expr ‘]’. This will though allow for constructs like 3[1] to be recognized by the parser which are meaningless.
Example:
int bad() {
\ta=7[9];
}
int good() {
\ta=array[9][2+4];
}
Result: (file (functionDecl (type int) bad ( ) (block { (stat (expr a) = (expr (expr 7) [ (expr 9) ]) ;) })) (functionDecl (type int) good ( ) (block { (stat (expr a) = (expr (expr (expr array) [ (expr 9) ]) [ (expr (expr 2) + (expr 4)) ]) ;) })))

I believe the construct should be “| [-]? ID ‘[’ expr ‘]’” which should restrict the parser interpretation.

In which case the grun result is:
line 2:4 mismatched input ‘[’ expecting {’;‘, ’*‘, ’+‘, ’-‘, ’==‘}
(file (functionDecl (type int) bad ( ) (block { (stat (expr a) = (expr 7 [) <missing ’;’>) stat (stat 9 ] ;) })) (functionDecl (type int) good ( ) (block { (stat (expr a) = (expr array [ (expr 9) ] [ (expr (expr 2) + (expr 4)) ]) ;) })))

Catches the error as expected.

locals [int i = 1;] <- semicolon is error

As well as changing
…setSLL(true) to …setPredictionMode(PredictionMode.SLL)
you will need to change
…setSLL(false) to …setPredictionMode(PredictionMode.LL)
near the bottom of the catch-clause.
— George Cowan

In the summary bullet points at the bottom of the page, make it clear that combined grammars can import combined grammars, e.g.,

• Combined grammars can import any kind of grammar (except, for the time being, ones using lexical modes).

The exception that is actually thrown by the SLL parser is org.antlr.v4.runtime.misc.ParseCancellationException. Therefore, in the if statement of the catch clause, we must say
if ( ex.getClass() == ParseCancellationException.class && …)
or even better change the catch clause to
catch (ParseCancellationException ex) {
if ( ! (ex.getCause() instanceof RecognitionException) ) {
throw ex;
}
else { …
- George S. Cowan

Reference 7 for the primary DOT language reference seems to be at h**p://www.graphviz.org/content/dot-language.
Apache at www.graphviz.org returns a 403 response status for the URL in the book.

“Stepping up our processing needs a little bit,” … of foo?

Antlr4
Notes

Page 63, 64, 66 about the | symbol.
It seems to me that in the antlr expression brackets are missing. The first example (‘=’ expr|) seems to me that the choice is expr or blank, instead of equals expr or blank.
Same as on page 64, with | id ‘=’ id | that should be like | ( id ‘=’ id ) |.
And on page 66 where the stat: ‘while’ ‘(’ expr ‘)’ stat | ‘{’ stat* ‘}’ …; is later changed to block, but it seems to me again that it missed the () around ‘{’ stat* ‘}’.

In case I am not correct, please explain better how to use it. In the examples it seems almost that the newline is also involved in grouping the symbols.

Please ignore #52317

The second paragraph from the bottom of the page should begin “The fail option takes either a string literal in single quotes or an action …”, that is, “single quotes” not “double quotes”. Double quotes are only allowed inside action braces.

“TokenStream” is not coloured properly.

Please see this private gist: gist.github.com/mahata/abd583dc67ff40f46e5b

Hello,

thanks for the excellent software and book. It seems that the following example from page 221 does not work

~
BAD_COMMENT1: ‘’
{System.err.println(“Can’t have —-> end comment”);} -> skip ;

BAD_COMMENT2:
‘’
{System.err.println(“Can’t have — in comment”);} -> skip;
~

fails with the following message:

>action in lexer rule ‘BAD_COMMENT1’ must be last element of single outermost alt.

I tried the example after failing to both set a semantic predicate value (as in page 217, lexmagic/SimplePy.g4) and at the same time switch context (as in page 222, lexmagic/ModeTagsLexer.g4).

The EvalVisitor class is defined:

public class EvalVisitor extends LabeledExprBaseVisitor

LabedExprBaseVisitor is an interface, not a class — this cannot be valid Java.

In your JSON.g4 grammar, if I wanted to match the number 2.01 I could not do it. The “INT” allows no leading zero, so the 2nd “INT” can’t match the “01” after the dot.

In your JSON.g4 grammar, if I wanted to match the number 2.01 I could not do it. The “INT” allows no leading zeroes, so the 2nd “INT” can’t match the “01” after the dot.

In the tour/Data.g4 example, there’s the following piece of code:

sequence[int n]
locals [int i = 1;]
: ( {$i<=$n}? INT {$i;} )* // match n integers
;

Doesn’t work for me the way it’s described in the book. The tree is wrong, grouping n+1 characters instead of n. Looks like $i<=$n should be $i<$n - exit condition is 1 character too long in the book/examples.

Hi,
I am using Antlr4.4. I have adapted the example tour/Data.g4 as follows:

sequence[int n]
locals [int i = 42;]
: ( {$i < $n}? INT { System.out.println($i); $i; } )* // match n integers
;

It seems that the initialisation of the local variable i fails. Instead of being initialised to 42, the variable i always starts out at 0.

This seems to be more of a bug in Antlr than a bug of the book.

On page 73 at the bottom of the page, the link to the java grammar
does not work.

On page 102 the rule for `expr` is stated as follows:

~
expr: ID ‘(’ exprList? ‘)’ // func call like f(), f(x), f(1,2)
| ID ‘[’ expr ‘]’ // array index like a[i], a[i][j]
| ‘-’ expr // unary minus
| ‘!’ expr // boolean not
| expr ‘*’ expr
| expr (‘+’|‘-’) expr
| expr ‘==’ expr // equality comparison (lowest priority op)
| ID // variable reference
| INT
| ‘(’ expr ‘)’
;
~

The part about array indexing seems to be wrong since, as given, the rule does not permit
an array index of the form `a[i][j]`. However, the comment states that `a[i][j]` is a legal
array index. Furthermore, the parse tree on the next page suggests that the alternative
responsible for array indexing should be given as

| expr ‘[’ expr ‘]’

Regards,
Karl Stroetmann

Chapter 5.4: the right association examples uses the old syntax:
The book still shows the V4.2 style assoc while v4 has it changed

running the example in the book:
UNRECOGNIZED_ASSOC_OPTION
public static final ErrorType UNRECOGNIZED_ASSOC_OPTION
Compiler Warning 157.
rule rule contains an assoc element option in an unrecognized location

In ANTLR 4.2, the position of the assoc element option was moved from the operator terminal(s) to the alternative itself. This warning is reported when an assoc element option is specified on a grammar element that is not recognized by the current version of ANTLR, and as a result will simply be ignored.

The following rule produces this warning.

x : ‘x’
| x ‘+’<assoc=right> x // warning 157
|<assoc=right> x * x // ok
;

When I run antlr4 on Simple.g4 I get the error message:

error(65): Simple.g4:18:52: unknown attribute text for rule stat in $stat.text
error(65): Simple.g4:20:54: unknown attribute text for rule stat in $stat.text

The examples errors/TestE_Dialog.java and errors/TestE_Listener2.java do not compile because of the import directive

import com.sun.istack.internal.Nullable;

They do compile after this import directive is changed to

import org.antlr.v4.runtime.misc.Nullable;

In the example “examples/CSV.g4”, the syntax highlight seems to be incorrect, i.e. the second line is highlighted as if it was a string, but in fact it is a lexer rule. Similar problems are elsewhere in the book, from the top of my head is in “examples/JSON.g4” on PDF page 93.

This rule (near the top of the page(:

assign : ID (WS|COMMENT)? ‘=’ (WS|COMMENT)? expr (WS|COMMENT)? ;

should be:

assign : ID (WS|COMMENT)* ‘=’ (WS|COMMENT)* expr (WS|COMMENT)* ;

The source for tour/EvalVisitor.java is no longer valid for Antlr 4.4. Where are the modified sources?

“mode” should be purple-boldfaced to represent it as a keyword.

[final sentence of the page, the word “of” is missing, I suspect:] “… on the right side OF rule ‘file’.”

Throughout the book, cross-reference links to other pages are missing a space between the comma and the words “on page NN”. Presumably, this is a mistake in a repeating template of some kind. I’m on page 36 and mention this because I’ve seen it so often it has become an irritant by now. This particular case is: “Chapter 5, Designing Grammars,on page 59.”

In method consume() the assignment charPositionInLine = 0 should be charPositionInLine = –1. This will correct the erroneous output of TestSimpleMyToken on page 241 where the last four tokens have wrong coordinates for the character position: ‘abc’ 2:1 should be 2:0, etc.

The sample errors/TestBail.java contains
extends SimpleLexer
I cannot find the class SimpleLexer in ANTLR4 java download.
There is a class Lexer and there is a discussion in the book about making a simple lexer, which may be are mixed up.

Ultimately, we need the ability
to divine a language’s structure from a set of representative input files.

There is a difference between godlike and splitting a language.

Section 4.5 Cool Lexical Features

Last paragraph on page 68:

>>Each line of that output represents a token and contains the token index, the start and stop character, the token text, the token type, and finally the line and character position within the line.

Should probably be “the start and stop character position”.

Would be clearer to either change “grun XML” to “grun XMLLexer”, or to clarify that the suffix “Lexer” is implicit in TestRig:

>>Here’s how to do a build and launch the test rig:
>>➾ $ antlr4 XMLLexer.g4
>>➾ $ javac XML*.java
>>➾ $ grun XML tokens -tokens t.xml

i.e. state that both “grun XML” and “grun XMLLexer” work since you have the option of omitting “Lexer”.

-RedTailedHawk

Reference to bad URL on antlr.org in footnote #3:
3. h-t-t-p://www.antlr.org/grammar/java

Note: I couldn’t enter the full URL due to spam prevention

There is a code snippet on this page for the section 7.2-Implementing Applications with Parse-Tree Listeners. The method named “PropertyFileBaseVisitor” should probably be named “PropertyFileBaseListener”. The code is also incorrect as it should implement PropertyFileListener, not PropertyFileVisitor.

I’m guessing what happened is that section 7.3-Implementing Applications with Visitors was written first, and the source code snippet was copied to 7.2, and didn’t get corrected.

The Windows commands to run ANTLR on this page are not correct. Below I have some example commands that I’ve tested in Windows. I kept the paths the same to be consistent with the book, but you may have to adjust the minor version number of the ANTLR 4.x *.jar files. There should also be some code to set the CLASSPATH environment variable either from the command line or via Control Panel for the Windows instructions, since the commands assume it is set.

set CLASSPATH=“C:\\jars\\antlr-runtime-4.4.jar;C:\\jars\\antlr-4.4-complete.jar;”

doskey antlr4=java -jar C:\\libraries\\antlr-4.4-complete.jar $*

doskey grun=java org.antlr.v4.runtime.misc.TestRig $*

I found that you cannot have a label on a rule containing a single condition (no alternatives). I found that it’s helpful for debugging purposes to use labels when troubleshooting a grammar when giving source code as input to the generated parser. I have been using them for a reference when validating entry and exit events for grammar unit tests. As a trick/tip to the reader, it might be worth mentioning that you must have at least two alternatives to use these labels.

In DefPhase.java, the statement:
int typeTokenType = ctx.type().start.getType();

There is no explanation of what the “start” field is. I found an explanation at www.antlr.org/api/Java/org/antlr/v4/runtime/ParserRuleContext.html. Reading that web page, it seems like “getStart()” would be a better way to access the initial token in the context.

Suggestion: mention that “getStart()” returns the initial token in the context.

In the subsection “Matching Numbers” there is a parenthetical note “(See Section 6.5, Parsing R, on page 104 for lexical rules that match full floating-point numbers and even complex numbers like 3.2i.)”

Nowhere in section 6.5 are lexical rules to match floating-point numbers discussed. There are rules for matching (complex) floating-point numbers in the source code file code/examples/R.g4

I suggest removing the parenthetical note.

Lexer Rule Elements section:

The ESC rule is documented as ‘\\\\’ . ;
This implies that backslash quoting is needed in character literals, but on p280 it is documented that it is only needed in character classes (‘char sets’).

It took me a while to figure this out because the Antlr4 plugin (latest in IntelliJ) provides a misleading error about ‘no viable input alternative’.

Suggest removing the latter, and putting a separate para below the lexical rules that states how to quote what, and where backslash quoting is needed.

In this example code you’re using the Java Stack class. As per the Java documentation, this is deprecated and the ArrayDeque class should be used instead. It makes no practical difference, but books shouldn’t be recommending APIs that are deprecated for a good reason.

Configuration:
1. Page 6 in PDF.
2. antlr-4.5.1-complete.jar
3. ubuntu 14.04 machine

Scenario:
When trying to run ‘grun Hello r -tokens’ command on Linux with ANTLR version 4.5.1 there is following warning shown
Warning: TestRig moved to org.antlr.v4.gui.TestRig; calling automatically
So there is a need to update the following line in the book.
From
$ alias grun=‘java org.antlr.v4.runtime.misc.TestRig’
to
$ alias grun=‘java org.antlr.v4.gui.TestRig’

In the diagram, under “Good Syntax”, the expression is [i^2]. I believe it should be [1^2].

In source code example for “underlineError”, if the string to be parsed is short enough and just one line long, and an error occurs in that one line, the string placed in “errorLine” is not null-terminated. Then, when “errorLine” is printed, garbage characters are displayed.

In the picture breaking down parse tree into RuleNodes, TerminalNodes, TokenStream and CharStream, the highlighting of the TokenStream was probably intended to line up with the characters in “sp = 100;” The highlighting serves to show what parts of the string belong to each token. However, the highlighting is much wider than the rendered string. It is probably a font-related problem.

if you test the expression “3*(4+5)”, you will get the result 0;
why?
the code " | ‘(’ e ‘)’ {$v = $e.v;}"
should be modefied as " | ‘(’ a=e ‘)’ {$v = $a.v;}"

The sentence beginning “At version 4.0, ANTLR is still written in ANTLR3, so…” caused me a lot of human backtracking to parse. I would suggest this could be rephrased for clarity. Changing the “At” to “As of” might have worked for me.

The book and sample JSON grammar seems to introduce an unnecessary restriction by adding a rule named ‘json’, which not present in either the json.org syntax diagrams, or the formal ECMA standard. The correct starting rule for JSON is ‘value’, this is made clear from paragraph 4 of the ECMA standard, which starts out saying:

A JSON text is a sequence of tokens formed from Unicode code points that conforms to the JSON value grammar.

correct warning when running> antlr4 java.4g

copying solution from stackoverflow since url not allowed here:

The should now be moved left of the “expression”.

It must be placed always right to the surrounding |:

to correct the java.4g file:

| expression ‘&&’ expression
| expression ‘||’ expression
| expression ‘?’ expression ‘:’ expression
|<assoc=right> expression
( ‘=’
| ‘+=’
| ‘-=’
| ‘*=’
| ‘/=’
| ‘&=’
| ‘|=’
| ‘^=’
| ‘>>=’
| ‘>>>=’
| ‘<<=’
| ‘%=’
)
expression

Line 17 says {if ($v.int==0) notifyListeners(“values must be > 0”);}

I think we should change “notifyListeners” into “notifyErrorListeners” b/c in the Parser class, we only have notifyErrorListeners method!

When I run antlr4 on Simple.g4 I get the error message:

error(65): Simple.g4:18:49: unknown attribute text for rule stat in $stat.text
error(65): Simple.g4:20:51: unknown attribute text for rule stat in $stat.text

I guess it is due to ambiguity, as rule stat has two alternatives. How to fix it? Easy. Try the following:

stat: expr ‘;’
{System.out.println(“found expr: ”$expr.text“;”);}
| ID ‘=’ expr ‘;’
{System.out.println(“found assign: ”$ID.text“=”$expr.text“;”);}
;

This will make the code work.

Section 5.6 just after 192.168.209.85 “GET /download/foo.html HTTP/1.0” 200

The rule
file : NL+ ;
should be
file : ( STUFF NL )+ ;
since the current rule can only match a file of empty lines.

The page has the following code
ID : a=NameStartChar NameChar*
{
if ( Character.isUpperCase(getText().charAt(0)) ) setType(TOKEN_REF);
else setType(RULE_REF);
}
;
Problem there is that ANTLR parser rejects the reference a to token NameStartChar. I guess the reference can safely be dropped.

Your UnderlineListener will throw java.lang.ArrayIndexOutOfBoundsException for the following input:

class T {int f(x) { a = 3; }

How to fix it? You need to add another line of code:

if (line <= lines.length)

and put the following code inside this if:

String errorLine = lines[line - 1];
System.err.println(errorLine);
for (int i=0; i<charPositionInLine; i) System.err.print(" “);
int start = offendingToken.getStartIndex();
int stop = offendingToken.getStopIndex();
if ( start>=0 && stop>=0 ) {
\tfor (int i=start; i<=stop; i) System.err.print(”^");
}
System.err.println();

You said we will get the result 0 when we test the expression “3*(4+5)”.

Then you suggested the following change:
the code " | ‘(’ e ‘)’ {$v = $e.v;}" should be modefied as " | ‘(’ a=e ‘)’ {$v = $a.v;}"

But I found this change is NOT necessary, the code just runs fine w/o any change!

examples/CSV.g4
TEXT : ~[,\
\\r"]+ ;
STRING : ‘"’ (‘“”’|~‘“‘)* ’”’ ; // quote-quote is an escaped quote

Should be:
TEXT : ~[,\
\\r]+ ; // no quote
STRING : ‘"’ (‘“”’|~‘“‘)* ’”’ ; // quote-quote is an escaped quote

In Chapter 2. The Big Picture • 14, the comment on the first line for instanciation rule is wrong:
stat: ID ‘=’ expr ‘;’ // match an assignment; can match “f();”

The rule stat can’t match “f();”

By fill in: {99,99,3,{451,99,3},451,451,8} throws an
“main” java.lang.NullPointerException at ShortToUnicodeString.enterValue(ShortToUnicodeString.java:27)

changing from
/ Translate integers to 4-digit hexadecimal strings prefixed with \\\\u */
@Override
public void enterValue(ArrayInitParser.ValueContext ctx) {
// Assumes no nested array initializers
int value = Integer.valueOf(ctx.INT().getText());
System.out.printf(“\\\\u%04x”, value);
}

to

/ Translate integers to 4-digit hexadecimal strings prefixed with \\\\u */
@Override
public void enterValue(ArrayInitParser.ValueContext ctx) {
// Assumes no nested array initializers
if (ctx.INT() != null){
int value = Integer.valueOf(ctx.INT().getText());
System.out.printf(“\\\\u%04x”, value);
}
}

works fine.

The Test.java example in Section 3.3 uses ANTLRInputStream.

Using IdeaJ to compile things, this primitive shows as deprecated.

I’m new to Java so I might have misconfigured stuff, but…

The EXP token rule fragment incorrectly uses INT for the exponent value; leading zeros are actually allowed by the specification, here (e.g., 1.1e009 is valid). The book grammar should use [0-9]+, instead.

The implementation of underlineError() IMHO bears a potential IndexOutOfBoundsException:

/* … */
String[] lines = input.split(“\
”);
String errorLine = lines[line - 1];
/* … */

If the last character of a parsed file is a newline, and the parser issues a parse error past that last newline character, lines[line - 1] will issue an out of bounds exception, since

“Trailing empty strings are therefore not included in the resulting array.” according to the documentation of String.split()

The link in footnote #10, to the R Language Overview (www.stat.wisc.edu/~deepayan/SIBS2005/slides/language-overview.pdf) is broken.

Under Section 3.3 Integrating a Generated Parser into a Java Program,
the program listing “starter/Test.java” uses ANTLRInputStream which has been deprecated. The code should be changed use CharStream instead of ANTRLInputStream as follows:

// import ANTLR’s runtime libraries
import org.antlr.v4.runtime.*;
import org.antlr.v4.runtime.tree.*;

public class Test {
\tpublic static void main(String[] args) throws Exception {
\t\t// create a CharStream that read from standard input
\t\tCharStream input = CharStreams.fromStream(System.in);

\t\t// create a lexer that feeds off of input CharStream
\t\tArrayInitLexer lexer = new ArrayInitLexer(input);

\t\t// create a buffer of tokens pulled from the lexer
\t\tCommonTokenStream tokens = new CommonTokenStream(lexer);

\t\t// create a paerser that feeds off the tokens buffer
\t\tArrayInitParser parser = new ArrayInitParser(tokens);

\t\tParseTree tree = parser.init(); // begin parsing at init rule
\t\tSystem.out.println(tree.toStringTree(parser)); // print LISP-style tree
\t}
}

In the sentence “The parser gobbles up y;;; until it sees the start of another member (set © earlier) and then loops back to member.”, set © should be set(a). This sentence describes the parser behavior as described by set(a) in the previous page and not set© as mentioned within the parentheses in the sentence.

Code:
public class PropertyFileBaseVisitor extends AbstractParseTreeVisitor implements PropertyFileVisitor {
Override public T visitFile(PropertyFileParser.FileContext ctx) { } Override public T visitProp(PropertyFileParser.PropContext ctx) { } }

This subsection is introducing Listener patter, according to the context, i think “PropertyFileBaseVisitor” should be “PropertyFileBaseListener” (and also “AbstractParseTreeVisitor”, “PropertyFileVisitor”).

ctx.op.getType() no longer exists in antlr4.

Use this instead:
if(ctx.ADD() != null )
\t\treturn left + right;
\treturn left - right; //must be SUB

and:

if(ctx.MUL() != null )
\treturn left * right;
return left / right; //must be DIV