Download TOKEN - Department of Computer Science

CMPE 152: Compiler Design
April 6 Class Meeting
Department of Computer Engineering
San Jose State University
Spring 2017
Instructor: Ron Mak
www.cs.sjsu.edu/~mak
1
Syntax Error Handling and JavaCC
1.
Detect the error.

2.
Flag the error.

3.
JavaCC does that based on the
grammar in the .jj file.
JavaCC does that for you with its error messages.
Recover from the error so you can
continue parsing.

You set this up using JavaCC.
Computer Engineering Dept.
Spring 2017: April 6
CMPE 152: Compiler Design
© R. Mak
2
Token Errors

By default, JavaCC throws an exception
whenever it encounters a bad token.

Token errors are considered extremely serious
and will stop the translation unless you take
care to recover from them.

Example LOGO program that moves a cursor
on a screen:
FORWARD 20
RIGHT 120
FORWARD 20
Computer Engineering Dept.
Spring 2017: April 6
CMPE 152: Compiler Design
© R. Mak
3
Token Errors, cont’d
SKIP : {
" " | "\n" | "\r" | "\r\n"
}
logo_tokenizer.jj
TOKEN : {
<FORWARD : "FORWARD">
| <RIGHT : "RIGHT">
| <DIGITS: (["1"-"9"])+ (["0"-"9"])*>

What happens if we feed the tokenizer
bad input?
FORWARD 20
LEFT 120
FORWARD 20
Computer Engineering Dept.
Spring 2017: April 6
CMPE 152: Compiler Design
© R. Mak
Demo
4
Token Errors, cont’d

One way to recover from a token error is to
skip over the erroneous token.
public static void main(String[] args) throws Exception
{
java.io.Reader reader = new java.io.FileReader(args[0]);
SimpleCharStream scs = new SimpleCharStream(reader);
LogoTokenManager mgr = new LogoTokenManager(scs);
while (true) {
try {
if (readAllTokens(mgr).kind == EOF) break;
}
catch (TokenMgrError tme) {
System.out.println("TokenMgrError: " +
tme.getMessage());
skipTo(' ');
}
}
logo_skip_chars.jj
}
Computer Engineering Dept.
Spring 2017: April 6
CMPE 152: Compiler Design
© R. Mak
5
Token Errors, cont’d
private static void skipTo(char delimiter)
throws java.io.IOException
{
String skipped = "";
char ch;
logo_skip_chars.jj
System.out.print("*** SKIPPING ... ");
while ((ch = input_stream.readChar()) != delimiter) {
skipped += ch;
}
System.out.println("skipped '" + skipped + "'");
}
Computer Engineering Dept.
Spring 2017: April 6
CMPE 152: Compiler Design
© R. Mak
Demo
6
Synchronize the Parser

Skipping over a bad token isn’t
a complete solution.

The parser still needs to synchronize at
the next good token and then attempt to
continue parsing. SKIP : {
logo_synchronize.jj
" "

First, add an
error token to
represent any
invalid input
characters:
Computer Engineering Dept.
Spring 2017: April 6
}
TOKEN : {
<FORWARD : "FORWARD">
| <RIGHT
: "RIGHT">
| <DIGITS : (["1"-"9"])+ (["0"-"9"])*>
| <EOL
: "\r" | "\n" | "\r\n">
| <ERROR
: ~["\r", "\n"]>
}
Any character except \r or \n.
CMPE 152: Compiler Design
© R. Mak
7
Synchronize the Parser, cont’d

A program consists of one or more move
(FORWARD) and turn (RIGHT) commands.

Must also allow for an erroneous command.
logo_synchronize.jj
void Program() : {}
{
(
try {
MoveForward() {System.out.println("Processed Move FORWARD");}
| TurnRight()
{System.out.println("Processed Turn RIGHT");}
| Error()
{handleError(token);}
}
catch (ParseException ex) {
handleError(ex.currentToken);
}
)+
}
Computer Engineering Dept.
Spring 2017: April 6
CMPE 152: Compiler Design
© R. Mak
8
Synchronize the Parser, cont’d
void MoveForward() : {}
{
<FORWARD> <DIGITS> <EOL>
}
void TurnRight() : {}
{
<RIGHT> <DIGITS> <EOL>
}
void Error() : {}
{
<ERROR>
logo_synchronize.jj
}

The Error() production rule is invoked for the
<ERROR> token.

The <ERROR> token consumes the bad character.
Computer Engineering Dept.
Spring 2017: April 6
CMPE 152: Compiler Design
© R. Mak
9
Synchronize the Parser, cont’d
logo_synchronize.jj
JAVACODE
String handleError(Token token)
{
System.out.println("*** ERROR: Line " + token.beginLine +
" after \"" + token.image + "\"");
Token t;
do {
t = getNextToken();
} while (t.kind != EOL);
return t.image;
Synchronize the parser to the
next “good” token (EOL).
You can do this better with a
complete synchronization set!
}

The JAVACODE header precedes pure Java
code that’s inserted into the generated parser.
Computer Engineering Dept.
Spring 2017: April 6
CMPE 152: Compiler Design
© R. Mak
Demo
10
Repair the Parse Tree

After the parser recovers from an error, you
may want to remove a partially-built AST node.

The erroneous production must call
jjtree.popNode().
logo_tree_recover.jjt
JAVACODE
String handleError(Token token) #void
{
System.out.println("*** ERROR: Line " + token.beginLine +
" after \"" + token.image + "\"");
Token t;
do {
t = getNextToken();
} while (t.kind != EOL);
}
jjtree.popNode();
return
t.image;
Computer Engineering Dept.
Spring 2017: April 6
CMPE 152: Compiler Design
© R. Mak
Demo
11
Debugging the Parser

Add the option to debug the parser.
options {
DEBUG_PARSER=true;
}


Print production rule method calls and returns.
Print which tokens are consumed.
Computer Engineering Dept.
Spring 2017: April 6
CMPE 152: Compiler Design
© R. Mak
12
Review: Interpreter vs. Compiler

Same front end


Same intermediate tier


parser, scanner, tokens
symbol tables, parse trees
Different back end operations
Computer Engineering Dept.
Spring 2017: April 6
CMPE 152: Compiler Design
© R. Mak
13
Review: Interpreter vs. Compiler, cont’d

Interpreter: Use the symbol tables and parse
trees to execute the source program.


executor
Compiler: Use the symbol tables and parse
trees to generate an object program for the
source program.

code generator
Computer Engineering Dept.
Spring 2017: April 6
CMPE 152: Compiler Design
© R. Mak
14
Target Machines

A compiler’s back end code generator produces
object code for a target machine.

Target machine: the Java Virtual Machine
(JVM)

Object language: the Jasmin assembly
language

The Jasmin assembler translates the assembly
language program into .class files.

Java implements the JVM which
loads and executes .class files.
Computer Engineering Dept.
Spring 2017: April 6
CMPE 152: Compiler Design
© R. Mak
15
Target Machines, cont’d

Instead of using javac to compile a source
program written in Java into a .class file.

Use your compiler to compile a source program
written in your chosen language into a Jasmin
object program.

Then use the Jasmin assembler to create the
.class file.
Computer Engineering Dept.
Spring 2017: April 6
CMPE 152: Compiler Design
© R. Mak
16
Target Machines, cont’d

No matter what language the source program
was originally written in, once it’s been compiled
into a .class file, Java will be able to load and
execute it.

The JVM as implemented by Java runs on a
wide variety of hardware platforms.
Computer Engineering Dept.
Spring 2017: April 6
CMPE 152: Compiler Design
© R. Mak
17
Java Virtual Machine (JVM) Architecture

Java stack


Heap area





CMPE 152: Compiler Design
© R. Mak
code for methods
constants pool
Native method stacks

Computer Engineering Dept.
Spring 2017: April 6
dynamically allocated
objects
automatic garbage
collection
Class area


runtime stack
support native methods,
e.g., written in C
(not shown)
18
Java Virtual Machine Architecture, cont’d

The runtime stack
contains stack frames.


Stack frame =
activation record.
Each stack frame
contains



local variables array
operand stack
program counter
(PC)
What is missing in the JVM
that we had in our
Pascal interpreter?
Computer Engineering Dept.
Spring 2017: April 6
CMPE 152: Compiler Design
© R. Mak
19