Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Parsing Quantum Chemistry Output files Using Jflex and CUP Sudhakar Pamidighantam NCSA, University of Illinois JFlex • Jflex from http://jflex.de This is a lexical analyzer generator for java Lexical analysis is the process of taking an input string of characters (such as the Our Quantum Chemistry Output from an Application such as Gaussian and producing a sequence of symbols called "lexical tokens", or just "tokens", which may be handled more easily by a parser. Tokens are symbols derived from regular expressions which are used by a parser for further action. Typical Tokens from Gaussian Output Strings String Token (Symbol) “Number of steps in this run” “Step number” "NUMERICALLY ESTIMATING GRADIENTS ITERATION“ "CCSD(T)=" "SCF Done: E(RHF) =" “Maximum Force” “RMS Force” Found Iter NSearch NSearch Energy Energy MaxGrad RmsGrad A Lexer Inputfile Taken from examples http://jflex.de/manual.html#ExampleUserCode • Java Specifics • • • import java_cup.runtime.*; • Options and Declarations • • • • • • /* • %unicode ---- • %cup • %% The name of the class JFlex will create will be Lexer. Will write the code to the file GoptfreqLexer.java. */ %class GoptfreqLexer %public defines the set of characters the scanner will work on. For scanning text files, %unicode should always be used. switches to CUP compatibility mode to interface with a CUP generated parser %cupdebug Creates a main function in the generated class that expects the name of an input file on the command line and then runs the scanner on this input file. Prints line, column, matched text, and CUP symbol name for each returned token to standard out. Continued Flex Input Lexical state • • • • • • • • • • • • • • • %state ITER %state INTVALUE %state FLOATVALUE %state ITER2 %state ITER3 %state FLOAT1 %state FLOAT2 %state IGNOREALL %state INPUT %state INPUTA %state INPUTB %state INPUTC %state INPUTD %state INPUTE %state INPUTF State is a lexical state and is identified by a name and it controls how matches can happen/not happen Jflex File Structures • The code included in %{...%} is copied verbatim into the generated lexer class source. Here you can declare member variables and functions that are used inside scanner actions. • • /* Macro Declarations These declarations are regular expressions that will be used latter in the Lexical Rules Section. */ • • • • • • • • • • • • • • • • • • LineTerminator = \r|\n|\r\n InputCharacter = [^\r\n] WhiteSpace = {LineTerminator} | [ \t\f] Comment = {TraditionalComment} | {EndOfLineComment} | {DocumentationComment} TraditionalComment = "/*" [^*] ~"*/" EndOfLineComment = "//" {InputCharacter}* {LineTerminator} DocumentationComment = "/**" {CommentContent} "*"+ "/" CommentContent = ( [^*] | \*+ [^/*] )* /* adjust syntax font-coloring */ Identifier = [:jletter:] [:jletterdigit:]* dec_int_lit = 0 | [1-9][0-9]* dec_int_id = [A-Za-z_][A-Za-z_0-9]* DIGIT = [0-9] FLOAT = [+|-]?{DIGIT}+"."{DIGIT}+ INT = [+|-]?{DIGIT}+ BOOL = [T|F] EQ = "=" STRING = [A-Z]+ GRAB = [^(" "|\r|\n|\r\n| \t\f)]+ • %% Jflex File • • • • • • • • • /* ------------------------Lexical • the start state YYINITIAL. */ Rules Section---------------------- */ /* This section contains regular expressions and actions, i.e. Java code, that will be executed when the scanner matches the associated regular expression. */ /* YYINITIAL is the state at which the lexer begins scanning. So these regular expressions will only be matched if the scanner is in • • <YYINITIAL> { Jflex Symbol Generation • /* Return the token STPT declared in the class sym that was found. */ • "-- Stationary point found" { return symbol(Goptfreqsym.STPT); } • “Standard orientation:” { return symbol(Goptfreqsym.GEOM;} • /* Print the token found that was declared in the class sym and then return it. */ • “Standard orientation" { System.out.print(" + "); return symbol(Goptfreqsym.GEOM); } Scanner Methods and Fields available in Action • • • void yybegin (int lexicalState) /* enters the lexical state lexicalState */ String yytext() /* returns the matched input text region */ ….. (see http://www.jflex.de/manual.html for more methods ) <YYINITIAL>{ "Stationary point found" { yybegin(ITER); return new Symbol(FinalCoordSym.FOUNDITER); } <ITER>{ "X Y Z" { yybegin(INPUTF); "THE_END_OF_FILE" { "Standard orientation:" { .|\n {} } return new Symbol(FinalCoordSym.INPUT1); } yybegin(IGNOREALL); return new Symbol(FinalCoordSym.SCFDONE); } yybegin(IGNOREALL); return new Symbol(FinalCoordSym.SCFDONE); } Parsing for Geometry • • <INPUTF> { "---------------------------------------------------------------------" { yybegin (INPUT); return new Symbol(FinalCoordSym.DASH1); } } • <INPUT> { {INT} { yybegin (INPUTA); • • return new Symbol(FinalCoordSym.INPUT2, new Integer(yytext())); } "---------------------------------------------------------------------" { yybegin (ITER); return new Symbol(FinalCoordSym.DASH2); } } • <INPUTA> { {INT} { yybegin (INPUTB); return new Symbol(FinalCoordSym.INPUT3, new Integer(yytext())); } } • <INPUTB> { {INT} { yybegin (INPUTC); return new Symbol(FinalCoordSym.INPUT4, new Integer(yytext())); } } • <INPUTC> { {FLOAT} {yybegin (INPUTD); return new Symbol(FinalCoordSym.INPUT5, new Float(yytext())); } } • <INPUTD> { {FLOAT} { yybegin (INPUTE); return new Symbol(FinalCoordSym.INPUT6, new Float(yytext())); }} • <INPUTE> { {FLOAT} { yybegin (INPUT); return new Symbol(FinalCoordSym.INPUT7, new Float(yytext())); } } • • <IGNOREALL>{ .|\n {} } .|\n {} • • • • • • • • • • Standard orientation: --------------------------------------------------------------------Center Atomic Atomic Coordinates (Angstroms) Number Number Type X Y Z --------------------------------------------------------------------1 7 0 .000000 .111062 .000000 2 1 0 -.931526 -.259200 .000000 3 1 0 .465763 -.259119 .806530 4 1 0 .465763 -.259119 -.806530 --------------------------------------------------------------------- CUP Parser Generator • CUP is a perser generator for Java • It generates Look Ahead Left to Right parser from simple specifications • This is similar to YACC • These tools are used to construct relationships from basic structures for compilers ( and natural languages) Cup File Structure • 4 Main parts Part 1. preliminary and miscellaneous declarations Imported Code ( classes) Initialization Invoking Scanner Getting Tokens ( lexical tokens) Cup File Structure • Part 2 Declares Terminals and Non Terminals Associate Object classes with above Terminals are of type Notype or Integer Terminals are symbols with Association to Strings ( Non terminals) Cup File structure • Part 3 Specification of Precedence and Associativity of Terminals • Part 4 Grammar Cup Usage If the specification is in a file parser.cup then java java_cup.Main < parser.cup Would result in two java source files Sym.java sym class contains a series of constant declarations, one for each terminal symbol. This is typically used by the scanner to refer to symbols (e.g. with code such as "return new Symbol(sym.SEMI);" ). Parser.java The parser class implements the parser itself. CUP File Structure Note • To calculate and print values of each expression, we must embed Java code within the parser to carry out actions at various points. • In CUP, actions are contained in code strings which are surrounded by delimiters of the form {: and :} In general, the system records all characters within the delimiters, but does not try to check that it contains valid Java code. Example finalcoord.cup Part1. Preliminaries/ Initialization • • • • import java_cup.runtime.*; import javax.swing.*; import java.util.*; import java.io.*; /* comment code • Standard orientation: • --------------------------------------------------------------------• Center Atomic Atomic Coordinates (Angstroms) • Number Number Type X Y Z • --------------------------------------------------------------------• 1 7 0 .000000 .111062 .000000 • 2 1 0 -.931526 -.259200 .000000 • 3 1 0 .465763 -.259119 .806530 • 4 1 0 .465763 -.259119 -.806530 • --------------------------------------------------------------------• • • • • • • • • • OUTPUT FORMAT:____________________________________________________________ 1NSERCH= 0 more text SCF Done: E(RHF) = -7.85284496695 A.U. after 8 cycles more text Maximum Force 0.000000 0.000450 YES RMS Force 0.000000 0.000300 YES more text TO MONITOR:____________________________________________________________ iteration, energy • • • MANUALLY ADD TO CUP-GENERATED CLASS IN SCFaParser.java:________________ //add to CUP$SCFaParser$actions public ParseSCF2 parseSCF; • • */ //add to the constructor of CUP$SCFaParser$actions parseSCF = new ParseSCF2(); Example finalcoord.cup Part1 continued… • • • • • • • • • • • • • • • • action code {: //__________________________________ public static boolean DEBUG = true; private static JTable table; private static final String tableLabel = "SCF Intermediate Results:"; // private static String cycle = "0"; public static JTable getTable() { return table; } public static String getTableLabel() { return tableLabel; } // } :} Example finalcoord.cup Part 2 Terminal and Non terminal Declarations • terminal INPUT1, FOUNDITER, SCFDONE, DASH1, DASH2; • terminal Integer INPUT2, INPUT3, INPUT4, ITERATION; • terminal Float ENERGY, INPUT5, INPUT6, INPUT7; • non terminal startpt, scfintro, scfpat, scfcycle, cycle, grad1, grad2; • non terminal inp2, inp3, inp5, inp6, inp7, cycle1, cycle2, cycle3; Example finalcord.cup Part 3 Precedence and associativity This is optional and is important for ambiguous grammers This is not sued as our parsing is straight forward Example finalcoord.cup Part 4 Grammar // Start with a non-terminal ( symbol/string) ::= action, terminal, nonterminal, precedence and a ; at the end // Java code is inbetween {: … :} // productions separated by | • startpt ::= scfintro scfpat SCFDONE ; • scfintro ::= FOUNDITER {: if (DEBUG) System.out.println("CUP:Input: found the start of Iteration"); :}; • scfpat ::= scfpat scfcycle {: if (DEBUG) System.out.println("CUP:Input: in scfpat"); :} | scfcycle ; • scfcycle ::= INPUT1 DASH1 cycle1 DASH2; • cycle1 ::= cycle1 cycle2 | cycle2 ; • cycle2 ::= inp2 inp3 INPUT4 inp5 inp6 inp7 ; Example finalcoord.cup Grammar Continued • • • inp2 ::= INPUT2:in2 {: //___________________________________________________________________ if (DEBUG) System.out.println("CUP:Input: center number "+in2); :} ; • • • inp3 ::= INPUT3:in3 {: //___________________________________________________________________ if (DEBUG) System.out.println("CUP:Input: atomic number "+in3); :} ; • • • inp5 ::= INPUT5:in5 {: //___________________________________________________________________ if (DEBUG) System.out.println("CUP:Input: x coordinate "+in5); :} ; • • • inp6 ::= INPUT6:in6 {: //___________________________________________________________________ if (DEBUG) System.out.println("CUP:Input: y coordinate "+in6); :} ; • • • inp7 ::= INPUT7:in7 {: //___________________________________________________________________ if (DEBUG) System.out.println("CUP:Input: z coordinate "+in7); :} ; CUP Customization java java_cup.Main options < finalcoord.cup • Options -package GridChem -sym FinalCoordSym.java -parser FinalCoordParser.java More options at http://www.cs.princeton.edu/~appel/modern/java/CUP/manual.html#about Bottom_up Parser Architecture buffer Of states visited State Actio n Go To