Download String Tokenizer

File I/O
Overall Theory by using Scenarios
 much of the ideas will come straight from code
 all scenarios have something added or new
 data file has an invisible marker (EOF) at end that our program uses
File IO Scenario #1
Scenario #1 Code
import java.io.*;
import java.util.*;
public class FileIO2 {
public static void main(String[] args)
{
boolean debug = true;
// Create an array of Objects
int [] scores = new int[10];
Scanner infile = null;
try
{
infile = new Scanner(new FileReader("scores.txt"));
}
catch (FileNotFoundException e)
{
System.out.println("File not found");
e.printStackTrace(); // prints error(s)
System.exit(0); // Exits entire program
}
1
int s; // score
int i = 0; // i is used to place data into the correct array element (index)
while(infile.hasNextLine())
{
String line = infile.nextLine();
StringTokenizer tokenizer = new StringTokenizer(line);
// You should know what you are reading in
s = Integer.parseInt(tokenizer.nextToken());
if(debug)
{
System.out.println("score recevied: " + s);
}
scores[i++] = s;
}
infile.close( );
}
}
Questions:
1. Identify these areas:
a. validation (opening a file)
b. while loop (yes, while loop, not do/while loop)
c. debugging flag
d. where data is being read from the file
e. where we are checking to see if there is more data in the file
f. breaking down and converting the scores from strings to integers
2. What is “i” used for?
3. What does infile mean?? What would outfile mean?? (Take a wild guess)
4. What code will STOP the program if the file is not found?
5. What libraries did you have to include?
6. Where would the EOF be placed at the scores.txt file?
7. Order the following steps correctly:
a. declare temp variables
b. close file
c. open file
d. convert data (originally string) to whatever we need (int in this case)
e. read data from the file
f. check to see if the file was opened correctly
2
File IO Scenario #2
Scenario #2
Questions for Code below:
1. Identify these areas:
a. validation (opening a file)
b. while loop
c. debugging flag
d. where data is being read from the file
e. where we are checking to see if there is more data in the file
f. breaking down and converting the scores from strings to integers
2. What is the name of the file we are reading the data from?
3. Each line read from the file is broken up into how many pieces of data??
4. What Java key word breaks up the line into those data pieces?
3
import java.io.*;
import java.util.*;
public class FileIOTest {
public static void main(String[] args)
{
boolean debug = true;
// Create an array of Objects
Employee [] CCBC = new Employee[10];
for(int i = 0; i < CCBC.length; i++)
{
CCBC[i] = new Employee();
}
//
//
//
//
Scanner infile = null;
try
{
infile = new Scanner(new FileReader("document.txt"));
}
catch (FileNotFoundException e)
{
System.out.println("File not found");
e.printStackTrace(); // prints error(s)
System.exit(0); // Exits entire program
}
String name, department, title; int salary;
int i = 0; // i is used to place data into the correct array element (index)
while(infile.hasNextLine())
{
String line = infile.nextLine();
StringTokenizer tokenizer = new StringTokenizer(line);
// You should know what you are reading in
name = tokenizer.nextToken();
department = tokenizer.nextToken();
title = tokenizer.nextToken();
salary = Integer.parseInt(tokenizer.nextToken());
CCBC[i] = new Employee(name, department, title, salary);
i++;
if(debug)
{
System.out.println("Employees
System.out.println("Employees
System.out.println("Employees
System.out.println("Employees
}
name: " + name);
department: " + department);
title: " + title);
salary: " + salary);
}
infile.close( );
}
}
// Code this!!! If you can’t find your Employee, use mine in Advanced Objects
4
Reading/Writing Thoughts
 When reading in from a file
o YOU ARE THE ONE WHO CREATED THE FILE!!!!
 So you know how it is laid out.
o Or the file will be given to you (like scores.txt)
o Since you are reading values IN, do NOT set any variables in your
program!!! Since you are getting the value from the file
lastname firstname
lastname firstname
lastname firstname
lastname firstname
lastname firstname
lastname firstname
lastname firstname
lastname firstname
PROGRAM
(reads in values)
 When writing OUT to a file
o YOU ARE THE ONE WHO CREATING THE FILE!!!!
 So you design how it is laid out.
o Since you are writing values OUT, your program will set the values, then
write!!!
PROGRAM
(write out values)
lastname firstname
lastname firstname
lastname firstname
lastname firstname
lastname firstname
lastname firstname
lastname firstname
lastname firstname
5
Opening a file (for input or output)
 always within a try/catch
 you could also use the JOptionPane or a user input/variable
Manual opening
Scanner infile = null;
try
{
infile = new Scanner(new FileReader("document.txt"));
}
catch (FileNotFoundException e)
{
System.out.println("File not found");
e.printStackTrace(); // prints error(s)
System.exit(0); // Exits entire program
}
using user input/variable
String fileName = sc.next();
Scanner infile = null;
try
{
infile = new Scanner(new FileReader(fileName));
}
catch (FileNotFoundException e)
{
System.out.println("File not found");
e.printStackTrace(); // prints error(s)
System.exit(0); // Exits entire program
}// What type of file? Input/Output
Opening a file for OUTPUT
PrintWriter outfile = null;
try
{
outfile = new PrintWriter("results.txt");
}
catch (FileNotFoundException e)
{
System.out.println("File not found");
e.printStackTrace(); // prints error(s)
System.exit(0); // Exits entire program
}
6
Notice “infile” “outfile” – so anything that I may have for infile, can be switched
for an outfile
(inputs from a file) infile
(outputs to a file) outfile
next
print
(inputs from a keyboard)
(outputs to the monitor)
Closing a file (for input or output)




file MUST be closed when done
important for outfile since it saves AND closes the file at the same time
can reopen closed files
can have several files opened
infile.close( );
outfile.close();
What does token mean?
 Item separated by a space or \n (\r)
Actual File
Token Count
Lupoli is da bomb!!
Matt failed his midterm!!! Ha Ha!!!
4
6
7
String Tokenizer
M
r
.
L
u
p
o
l
i
i
s
d
a
b
o
m
b
tokens
(separated by spaces)
String inputline = infile.nextLine( );
StringTokenizer tokenizer = new StringTokenizer(inputline);
 breaks input line into a sequence of Strings (words) separated by spaces
 CANNOT TOKENIZE INDIVIDUAL LETTERS (use .atChar( ))
 http://download.oracle.com/javase/6/docs/api/
Commonly used Tokenizer Methods
tokenizer.hasMoreTokens( );
 determines if more “tokens” in the String
 used usually in a conditional loop
String word = tokenizer.nextToken( );
 grabs next token, assigns to “word”
 records last place it grabbed a word
int count = tokenizer.countTokens( );
 returns the number of tokens remaining to be returned by NEXTTOKEN( )
Example in use (from Scenario #2)
while(infile.hasNextLine())
{
String line = infile.nextLine();
StringTokenizer tokenizer = new StringTokenizer(line);
// You should know what you are reading in
name = tokenizer.nextToken();
department = tokenizer.nextToken();
title = tokenizer.nextToken();
salary = Integer.parseInt(tokenizer.nextToken());
8
Not everything needs to be Tokenized!!
 If there are MANY pieces of data on one line
o Tokenize
 Only one piece of data per line
o Read String from file, and Convert to proper datatype
To tokenize or not tokenize?
Tokenizable Data
Data not to be tokenized
Tokenizable Code
Non-tokenizing code
int num1 = Integer.parseInt(tokenizer.nextToken());
String word1 = tokenizer.nextToken(); // no conversion
total += Double.parseDouble(infile.nextLine());
9
Make sure input/output files are in the right place
 we are about to do an exercise, it’s important where the files are located
Parallel with Source
... new Scanner(new FileReader("document.txt"));
Inside Source
... new Scanner(new FileReader("./src/Lupoli.txt"));
Inside a
package
Avoiding Data file placement problems
... new Scanner(new FileReader("./src/dataSet/scores.txt"));
10
Using the Tokenizer to Break down a String
 the exercise will ask you to use some of the tokenizer methods
 copy scenario 2 (and data file), get it running in the IDE
In Scenario 2, add code in the DEBUG condition that counts how many tokens were
ORGINALLY in the line (Should be 4)
In Scenario 2:
1. delete the entire while loop. (and everything inside)
2. Add the StringTokenizer again, but this time, use 4 (???) methods to pull EACH
item from the LINE and display them, then pull ANOTHER LINE, etc…
(Display ALL pieces of data)
3. Change #2 slightly USING a loop
4. Change #3 slightly USING A WHILE loop
What can’t you call nexttoken 20 time straight??
Employee File
Data File
class Employee
{
public
public
public
public
String name;
String department;
String title;
int salary;
Lupoli C.S. Professor 0
Keshev Garbage Collector 10000
Ahmed CIA Hacker 100000000
Thaha FBI Explosives 1000000
Chris Parent Father -1000000
// METHODS INSIDE THE CLASS!!!
public String getName( )
{ return name; }
public int getSalary( ) { return salary; }
public void setSalary( int newSalary) { salary = newSalary; }
public String toString( ) { return name + "\n" + department + "\n" + title + "\n" + salary;}
public void setAll(String n, String d, String t, int s)
{ name = n; department = d; title = t; salary=s; }
public String getDepartment( ) { return department; }
public String getTitle( ) { return title; }
}
11
Reading techniques
 There is a marker that tracks where you JUST read from in the file
 make sure YOU THE PROGRAMMER place a string like address at the END
of a line
o easier to pull from string
 all scenarios below are AFTER:
StringTokenizer tokenizer = new StringTokenizer(line);
Reading Scenarios
Scenario #1
23 41 a Luper
Int, int, char, string
int num1 = Integer.parseInt(tokenizer.nextToken());
int num2 = Integer.parseInt(tokenizer.nextToken());
char letter = Character(tokenizer.nextToken().charAt(0));
String word = tokenizer.nextToken(); // no conversion
Scenario #2
One two three
String, String, String
String word1 = tokenizer.nextToken(); // no conversion
String word2 = tokenizer.nextToken(); // no conversion
String word3 = tokenizer.nextToken(); // no conversion
Scenario #3
Lupoli Shawn 1600 Penn…
String, String, String (LONG)
String word1 = tokenizer.nextToken(); // no conversion
String word2 = tokenizer.nextToken(); // no conversion
String address = ""; // restOfLine
while(tokenizer.hasMoreTokens())
{ address += tokenizer.nextToken() + “ “; }
12
Scenario #4
I wish Lupoli was easier
String (LONG)
String restOfLine = "";
while(tokenizer.hasMoreTokens())
{restOfLine += tokenizer.nextToken() + “ “; }
// DON’T TOKENIZE THE STRING!!! sc.nextLine();
Scenario #5
23 Lupoli 45 Bledsoe
Int, String, Int, String
int num1 = Integer.parseInt(tokenizer.nextToken());
String word1 = tokenizer.nextToken(); // no conversion
int num2 = Integer.parseInt(tokenizer.nextToken());
String word2 = tokenizer.nextToken(); // no conversion
Scenario #6
Lohan Lindsay 23 1982
Answerb:
Scenario #7
Lupoli A A D C
Answerb:
13
The easiest reading scenario – Flat File
 in a “flat” file, each data item is on a seperate line
 makes the coding much easier to read in a file
 XML file
Sample File with Code
 Notice the pattern for each SET of data.
Jack
789 Avenue Cir.
Los Angeles
CA
13579
80
02/02/1989
Sam
123 Circle Ct.
Richmond
VA
24680
70
04/04/1989
Matt
456 Court Ter.
Detroit
MI
54321
60
05/05/1989
public void insertFromFile()
{
Scanner infile = null;
try { infile = new Scanner(new FileReader("datafile.txt")); }
catch (FileNotFoundException e)
{
System.out.println("File not found.");
e.printStackTrace();
System.exit(0);
}
while(infile.hasNextLine())
{
String newName = infile.nextLine();
String newStreet = infile.nextLine();
String newCity = infile.nextLine();
String newState = infile.nextLine();
int newZipCode = Integer.parseInt(infile.nextLine());
int newTestGrade = Integer.parseInt(infile.nextLine());
String newDateOfBirth = infile.nextLine();
// insert(….
}
}
14
Writing to a file
 Syntax – almost same as a displaying to a monitor, just a file instead
PrintWriter outfile = new PrintWriter("A:data.txt"); // opening a file for output
outfile.println( … ); // without it, the data will be printed on one line
outfile.println( x + “ “ + y); // you can also put multiple variables in one line
Writing different types of data
 There are some other options given in writing to a file that can be advantageous
Writing to a file shortcuts
writing a hard coded
writing a variable letter
letter/number
outfile.println(‘f’);
char quit = ‘q’;
outfile.println(‘2’);
outfile.println(quit);
writing a hard coded sentence
writing a variable int
outfile.println( “Hello Class”);
int age = 23;
outfile.println( age);
writing a string
String name = “Mr. Lupoli”;
outfile.println(name); // space or no space ok
Combining it all together
outfile.println(“hello class, “ + name + “ “ + age + “ “ + quit);
15
Steps for writing to a file
1. Identify types of variables you will be writing OUT to that file, unless you want
to hard code an output
2. Identify the ORDER you wish to place the data to the file
3. Next declare an initialized variable for each of them
4. Then open the file for OUTPUT
5. Write data
6. Close file
Where do I put the file to read in, or to output?
 That is up to you as a programmer OR up to the user (if you ask them)
 Simple way
o A:
o F: (thumb)
o Drive not specified
(“document.txt”)
 Means RIGHT beside the .cpp file you are creating
Using a While loop to read data continually
Example While Loop Setup
89
98
66
54
97
34
98
Tokenized Version
int scores = 0;
while(infile.hasNextLine())
{
String line = infile.nextLine();
StringTokenizer tokenizer = new StringTokenizer(line);
// You should know what you are reading in
scores = scores + Integer.parseInt(tokenizer.nextToken());
}
16
Non-Tokenized version
import
import
import
import
java.io.FileNotFoundException;
java.io.FileReader;
java.util.Scanner;
java.util.StringTokenizer;
public class Driver {
public static void main(String[] args) {
// TODO Auto-generated method stub
boolean debug = true;
Scanner infile = null;
try {
infile = new Scanner(new FileReader("data.txt"));
} catch (FileNotFoundException e) {
System.out.println("File not found");
e.printStackTrace(); // prints error(s)
System.exit(0); // Exits entire program
}
int scores = 0;
while (infile.hasNextLine()) {
scores = scores + Integer.parseInt(infile.nextLine());
if(debug) { System.out.println("score is -> " + scores); }
}
}
}
17
While Loops
Flow Layout
Notes
Things to Notice :
 Make sure { }’s are in Block Like structure!!!
 The initializing variable is BEFORE the loop
 The initializing variable is set so that the loop will
run
 used when NOT exactly sure when loop should or
will end, or when the USER will define when to
end
5
(Counter)
//initialization
int count = 0;
while (count <10) // condition
{
statements;
…
…
count ++; // incrementation
}
(Event Controlled)
//initialization
while (infile.hasNextLine())
{
statements;
…
…
} // stops the loop
18
Draw a mock file (on paper) that will work with this while loop below:
int scores = 0;
String fname = "";
String lname = "";
while(infile.hasNextLine())
{
String line = infile.nextLine();
StringTokenizer tokenizer = new StringTokenizer(line);
fname = tokenizer.nextToken();
lname = tokenizer.nextToken();
scores = Integer.parseInt(tokenizer.nextToken());
}
// place sample file data here
Same example using arrays
int [] scores = new int[155];
String [] fname = new String[155];
String [] lname = new String[155];
int i = 0;
while(infile.hasNextLine())
{
String line = infile.nextLine();
StringTokenizer tokenizer = new StringTokenizer(line);
fname[i] = tokenizer.nextToken();
lname[i] = tokenizer.nextToken();
scores[i] = Integer.parseInt(tokenizer.nextToken());
i++;
}
19
Full example using a While loop
Wegner Keith 70 60
Wong Alex 30 10
Allen Rodney 99 23
Meehan Mike 23 56
Scanner infile = null;
try
{
infile = new Scanner(new FileReader("document.txt"));
} catch (FileNotFoundException e) {
System.out.println("File not found");
e.printStackTrace(); // prints error(s)
System.exit(0); // Exits entire program
}
String
String
int []
int []
[] lname = new String[155];
[] fname = new String[155];
scores1 = new int[155];
scores2 = new int[155];
int i = 0;
while(infile.hasNextLine())
{
String line = infile.nextLine();
StringTokenizer tokenizer = new StringTokenizer(line);
lname[i] =
fname[i] =
scores1[i]
scores2[i]
i++;
tokenizer.nextToken();
tokenizer.nextToken();
= Integer.parseInt(tokenizer.nextToken());
= Integer.parseInt(tokenizer.nextToken());
}
20
Ignoring data in a file
 Remember you have to read in EACH piece of data, no skipping!!
 Don’t use it after you read it in!!
Ignoring data in a file
String
String
int []
int []
[] lname = new String[155];
ignore = “”;
scores1 = new int[155];
scores2 = new int[155];
int i = 0;
while(infile.hasNextLine())
{
String line = infile.nextLine();
StringTokenizer tokenizer = new StringTokenizer(line);
lname[i] = tokenizer.nextToken();
ignore = tokenizer.nextToken();
scores1[i] = Integer.parseInt(tokenizer.nextToken());
scores2[i] = Integer.parseInt(tokenizer.nextToken());
System.out.println(lname[i]+‘ ‘+scores1[i]+’ ‘+scores2[i]);
// ignore never used
i++;
}
21
What NOT to do with File IO
 Read ALL patterned data in ONE pass in a loop
o used a while loop
o but read ALL data in ONE iteration of the loop, so didn’t use the loop
No No #1
Reading ALL data in the file
double num1,num2,num3,num4,num5,num6,num7,num8,num9;
while (infile.hasNextLine())
{
num1
num2
num3
num4
num5
num6
num7
num8
num9
=
=
=
=
=
=
=
=
=
infile.nextDouble();
infile.nextDouble();
infile.nextDouble();
infile.nextDouble();
infile.nextDouble();
infile.nextDouble();
infile.nextDouble();
infile.nextDouble();
infile.nextDouble();
average += num1 + num2 + num3 + num4 + num5 + num6 + num7 + num8 + num9;
}
average = average/9;
// if the file ONLY HAS 9 values, how many times will the loop run??
// what would happen if there were 18 values?? (Hint: average)
Corrected Version
double num, total = 0;
int count = 0;
while (infile.hasNextLine())
{
total += Double.parseDouble(infile.nextLine());
count++;
}
average = total/count;
// How many items will “num” be used if reading a file with 10 values?
// Why is count used??
// Why is code more versatile??
// Which version (no-no or corrected) has less code??
22
 Use a for loop to read the values inside the file.
o You might know how many values are there with the files I GIVE you
 but what if you didn’t know how many where going to be there, but
you knew the pattern?
No No #2
Using a for loop to read data
for (int i = 0; i < 9; i++)
{
num = sc.nextInt();
total += num;
}
// how many values will this read?
Corrected Version
// Create the reading loop using a while loop that will reads UNLIMITED values
23
Creating a game plan
 it is VERY important to design an game plan before coding
 No matter the application, there will also be some constants
 For now, please use Debug statement to print what you are reading in or writing
out
File IO gameplan
24
FYI- File name issues
 The syntax to open ANY file is as follows, you have several options:
o “A:/Lupoli.txt”
o “A:Lupoli.txt”
o “Z:\\Lupoli_class\\C\\test1.txt”
o normal “\” WILL NOT WORK!!
o Why won’t this work??
 “A:\nope.txt”
YOU CAN HAVE SEVERAL FILES OPEN AT THE SAME TIME!!
Solutions
Tokenizing Exercises
Scenario #6
String lastName = tokenizer.nextToken();
String firstName = tokenizer.nextToken();
int age = Integer.parseInt(tokenizer.nextToken());
int yearBorn =Integer.parseInt(tokenizer.nextToken());
Scenario #7
String lastName = tokenizer.nextToken();
char letter1 = Character(tokenizer.nextToken().charAt(0));
char letter2 = Character(tokenizer.nextToken().charAt(0));
char letter3 = Character(tokenizer.nextToken().charAt(0));
25
FYI - Breaking down a String (review)
 you can look at individual chars inside a string
o GET THE STRING AS A TOKEN using Tokenizer
 use the char functions to determine what is inside that string
o isLetter
o isDigit
o isWhitespace
o isLowerCase
o isUpperCase
String line = “Goodbye, Cruel World”;
G o o d
√ √ √ √
Total = 17
b
√
y
√
e
√
,
C
√
r
√
u
√
e
√
l
√
W
√
o
√
r
√
l
√
d
√
!
\0
isLetter example
int count = 0;
for(int i = 0; i < line.length(); i++) // counts # of letters in the string
{
if(Character.isUpperCase(line.charAt(i))){ count++; }
}
System.out.println(count);
isDigit example
int count = 0;
for(int i = 0; i < line.length(); i++) // counts # of numbers in the string
{
if(Character.isDigit(line.charAt(i))){ count++; }
}
System.out.println(count);
26
FYI – Appending to a File
 slightly different, nothing crazy
 Uses FileWriter to help append
 Need imports:
o
o
import java.io.FileWriter;
import java.io.IOException;
o
import java.io.PrintWriter;
Appending to a File
File Before
Code
File After
public static void main(String args[])
{
int val1 = 23; int val2 = 33; int val3 = 43;
try
{
FileWriter out = new FileWriter("out.txt", true);
PrintWriter writer = new PrintWriter(out);
writer.println("Lupoli needs a life");
// one line
writer.println(val1 + " " + val2 + " " + val3);
writer.close();
}
catch (IOException e)
{ System.out.println("Was not able to write to
file");
}
}
27