Download Latitude and Longitude

Analyzing TZ Data
James A. Rome
TGA
August 2010
What are we trying to do?
I was hired by NASA to analyze air traffic in ZOB48
 I had access to ETMS data
 The most useful data were TZ messages
 But as we saw, they are highly quantized
 1–4 minute time intervals (no seconds) -> 6 nm
 Multiple TZs from one flight at a given time (due to
multiple radar reports)
Situation is far from ideal. What do we do?
 First step is to look at the data and see how bad
it is, and what we can (or cannot) do with it.
 We will need to calculate things like GCD
Handling data is non-trivial
ACID,DEPT_APRT,ARR_APRT,POSIT_TIME,CUR_LAT,CUR_LON,GROUNDSPEED,ALTITUDE
COA497,CLE,MCO,8/7/99 0:01,2487,4907,472,290
UAL1517,CLE,DEN,8/7/99 0:05,2487,4909,502,310
Initially, the data are
arranged by time
There are 3900 TZs
Why do we need the airports?
 Because the flight number applies to several legs of a
flight
The big question is how should we arrange these data?
 This depends on what we want to do with the data





Do any planes “collide” (<1000 ft or <5 nm)?
What path does the flight take through the sector?
When does the controller do something to the flight?
How many flights are there at any time?
...
How good are the data?
The TZ messages have inherent error bars:
 Lat/Lons are in minutes
 1 minute = 1 nm of latitude, and somewhat less in
longitude. So positions might be off by a mile or so.
 Time is in minutes, so if seconds are rounded
down, may be up to a minute earlier than actual
 This leads to distance inaccuracy of (speed/60) nm
 Speed is in knots (nm/h) and is accurate enough
 Altitude is in 100s of feet, so may be off by 100 ft
The distances can be off by 6–8 miles. 5 miles is
the required separation distance between planes.
How should we handle the data?
 You could not do this with a TI calculator
 Maybe you could use Excel if the number of
flights is small, but you would quickly find that
you would be unable to answer all the above
questions
 We will have to write a computer program to
make any progress
 Every scientist or engineer who wants to solve a
new problem will get to this point. If you cannot
write a computer program, you cannot succeed!
It is time to write a program
We are going to calculate the great circle distance
(GCD) between two Lat/Lon points
 This will be part of a utility class because we will
need these tools to do more interesting things.
 The methods (things it can do) should be
 Available to other programs (public)
 Because there is no need for any changing data in the
class as a whole, the methods can be static
This implies that multiple methods can call the static
method at the same time without conflict (thread safe)
Static methods can be called without instantiating the
class via ClassName.method
No new operator is needed
Make ZOB48 Project in NetBeans
Select File, New Project
Packages
In a language as large as Java, it is important to organize
your files to avoid name conflicts
 Your method “openFile” might conflict with a method of
the same name in some other part of Java
 Packages create a unique way of naming your files
 by convention, the package name is the reverse of
your URL: gov.ornl.my_project_name
 Packages also denote accessibility boundaries for
methods and fields
 private is in my class
 protected is in my package
 public is open access
 You can move things around later via refactoring
Make a new package
Note the
Directory
Tree this
creates
Package name = arc.airtraffic.util
(arc is left-over from my course for ARC)
What is an object?
 An object is a thing that has properties
(attributes or fields) and can do stuff (methods).
 In Java, almost everything is an object
 An object is defined by a template called a class
(the computer code that is used to create the
object).
 An object does not exist until it is instantiated.
 What type of objects might we want to create for
our ZOB48 analysis?
 A TZ object, a Flight object, a Utility object, …
What is a class?
A class is the template for an object.
 Unless a class has methods that are declared
public static, you must instantiate the object
from the class template before using it. new
does this.
Vector vec = new Vector(); // create vec
 To call a method of a class, you use the variable
name (e.g., vec above):
vec.add("This is a String");
 If the method is static, you use the class name:
double sinpi = Math.sin(3.14159265);
double pi = Math.PI; // a static field
Make a new class
You could have
created the package
in this dialog.
 NetBeans will create a new Java file for you.
 The name of the file must match the name of the
class. Capitalization counts!
 This file is LatLon.java
 It is placed in the bottom of the package tree.
Special JavaDocs fields
LatLon utility class
Write JavaDoc comments as
you go.
This is a JavaDocs comment
A static method can be called without
instantiating an instance of its class
End-of-line comment
We will need the radius of
the earth. final = a constant
PI is a static field
of the Java Math class
How do you know about Math.PI?
JavaDocs Package names
documents all Java
classes
 If you fill in the data,
you can make JavaDocs
of your own code too.
 NetBeans can give
you the JavaDocs for
any method if you
configure it.
Class names
NetBeans tip: How to change JRE
Add method to calculate angle
Calling static methods
I usually break up long formulas using temporary
variables (trm1, trm2) to aid in debugging
Method to calculate the distance
We never really want the angle between the points at the center of the
Earth, so make a method we can call directly to get the great-circle-distance.
Make the JavaDocs
Here are your JavaDocs
• As you type your code, NetBeans
will display your choices, and their
JavaDocs.
• You can double-click a choice to
insert it into your code.
• NetBeans will even guess at the
names of your variables to use as
arguments!
Time to test the code
 Although NetBeans finds typos and bad usage
for you, it does not guarantee that your math is
correct.
 So let us make a test program to find the
distance between two lat/lon points.
 We will make a new class called “Runner” in the
arc.airtraffic package.
 I decided to calculate the distance from
(approximately) Toledo to Cleveland
Google is great
Toledo
Cleveland
Note, I eyeballed the lat/lons
The Runner class
array of Strings
main signature always
NetBeans
matches
braces
Be careful about parentheses, which determine the
order of computation. I always use more than needed.
Click Run, Run main project . . .
You must tell NetBeans
where to find main()
You can have severaal
main() methods in a
project
It works!
TZ object
What might a TZ object consist of?
 Attributes for one flight at one time
 Time, lat, lon, altitude, speed, and a combination of
the flight number, departure and destination airports.
E.g., NWA527_CLV_MSP
 Actions it can do
 Store a new (or updated) value of an attribute
 Give you (another part of your program) any of these
values
 Objects generally do not allow other objects to
access their attributes directly. Encapsulation
TZ pseudocode
public methods and attributes
accessible to all other objects
Pseudocode lets you concentrate on what your
code does, rather than how to write it, although
this example is so simple, it is real code.
public class TZData {
everything after // is a comment
// Class attributes
private int altitude;
// in hundreds of feet?
private double latitude; // in radians
private double longitude; // in radians
private String acid;
// the aircraft ID
private double speed;
// knots
private long time;
// time in seconds past 1/1/70
// Class methods
capital S means it is a class
// next slide content goes here
}
accessible only to
TZData objects
TZData methods
public TZData() {;} // default constructor
public TZData(int alt, double lat, double lon,
double speed, String acid, long time) {
altitude = alt; latitude = lat; longitude = lon;
this.speed = speed;
this.acid = acid; // Only can use = for Strings
this.time = time;
}
These are the type of thing returned to the caller
// These are the setters. Note the capitalization
public int getAltitude() {return altitude;}
public double getLatitude() {return latitude;}
public double getLongitude() {return longitude;}
public double getSpeed() {return speed;}
public String getAcid() {return acid;}
public long getTime() {return time;}
TZData methods (continued)
// the setter methods
void means that nothing is returned to caller
public void setSpeed(double spd) {speed = spd;}
public void setLatitude(double lat) {latitude = lat;}
public void setLongitude(double lon) {longitude = lon;}
// This time we will use a constructor for String
public void setAcid(String id) {acid = new String(id);}
public void setAltitude(int alt) {altitude = alt;}
public void setTime(long time) {this.time = time;}
} // the end of the class (repeated from slide 1}
Notice how we used lowercase set and get for each of
these methods to prepend the variable name which we
capitalized. This makes this class into a Java Bean.
 Java Beans can be added as elements into NetBeans
no “return” needed for voids
What have we done?
NetBeans has done a lot
of organizing for you
In the ZOB48 project
directory is a src tree
The package name was
translated into a
directory tree
The class file is placed
in the directory at the
bottom of the tree
Note: The name of the .java file must
always be the same as the class name
Time to enter the class code
NetBeans can do the grunge work of typing in the getters and setters
Getters and setters
Right-click the TZDate.java file name, select Refactor, Encapsulate Fields
Our TZData class is complete
We could write a main method to test our class,
but TZData is an important, but rather
uninteresting class, so we will skip that for now.
How to read in data
We have a multi-pronged problem to solve
1.How to actually read data from a file
2.What to read the data into
 Clearly we will read each row of data into a
TZData object. (That's why we made them!)
 But there are 3900 lines of data. We need a
way (or several ways) of organizing these
3900 objects into things we can access
easily.
3.How to convert the strings we read into the right
types of data.
Look before you leap!!!!!!!!!
I cannot emphasize how important it is to look
hard at all the issues before you start to code.
 This is actually the fun part of writing a program
 I do woodworking. I will spend a month figuring
out how to make a jig to do something (e.g., a
wide finger joint which needs 1/64" accuracy).
 We are going to go though my thought process
on how to read and store the data.
 Many times . . .
Organizing TZData objects
Assume we have read in the data. How do we
organize it?
 Depends on the problem . . .
 Can we tell if any flights are too close?
Need positions of all flights at a given time for each time
Time
TZ1
TZ2
TZ3
TZ4
TZ5
0:00:01
NWA1
AAL3
UAL7
0:00:03
AAL3
UAL7
DAL8
NWA2
NWA3
0:00:05
AAL3
UAL7
DAL8
NWA2
NWA3
0:00:06
AAL3
DAL8
DAL9
NWA2
TZ6
…
...
Kind of a 2-D array. But the rows have different lengths. Hard to search.
Java has Collections
We will put the TZData at a given
time in a Vector
<E> is the type of Object in the Vector
We will organize the Vectors using
I originally chose a HashMap,
a TreeMap
but decided that it would be
useful to have the data
ordered by time. Using the two
is essentially the same.
A TreeMap allows you to store
Key-Value pairs, sorted by the
key.
Key = Long time
Value = Vector of TZData
A TreeMap is an ordered
HashMap. The Hash allows very
fast searching through the list
of keys. Example:
24 is a hash of "cat"
"cat" = 3 + 1 + 20 = 24
It is a bad hash:
"act" = 24 also – a
collision
Hash
Values
1
a
2
3
ab
ba
4
ac
baa
5
6
cab
Data organization
The Keys are actually Long objects
Long key = new Long(4356780L);
Key
Value
Vectors of TZData Objects at given times
4356780L
AAL2
NWA4
DAL10
NWA6
4356900L
AAL2
NWA4
NWA6
COA2
4357020L
COA2
AAL5
NWA6
4357140L
COA2
AAL5
NWA7
4357260L
NWA7
DAL2
DAL2
COA2
USA5
4357500L
TreeMap
These arrows are actually the memory locations of
the TZData Vectors. These are called pointers.
Pseudocode for loading Data
// open the data file
// while there are more lines {
//
read a line from the file
//
convert the data
//
insert it into a TZData
//
get the time
//
if(the time is not in keys) {
//
make a new TZ Vector
//
add TZData to Vector
//
put it into TreeMap
//
}
//
else {
//
get Vector for this time
//
see if this flight is there
//
if (it is there) toss it
//
else add it to the Vector
//
}
// } // end of while
 There are many ways to
do this
 We will assume the data
are not in order
 Notice that we have
now introduced the new
concepts of
 looping (can you find 2?)
 logical branching (if,
else, else if)
 But they naturally
express how you have to
think about the problem
Thinking more on the problem . . .
If we are going to decide whether
any planes "hit" each other, we
need to know all the flights in the
air at once.
 But the TZs seem to occur at different
intervals for different flights, and they are
not consistent at all!
 We cannot implement the previous data
arrangement because we are going to
have to "manufacture" the missing data
Arrange the data by flight, ordered by time
Use interpolation to create new TZData
objects at every minute
Then rearrange the data as needed
 How????
Data organization redux 2
• We do not need the flight names sorted. So use a TreeSet with flight name as key.
• We do need the times sorted. Implies that the times can be compared.
Key
Value
TreeSet of times (must be Longs)
NWA3
4356780L 4357020L 4357260L 4357440L 4357680L
AAL6
4356600L 4356780L 4357080L 4357320
COA8
4357080L 4356260L 4357320L
NWA7
4357020L 4357260
UAL1
4357260L 4357500L
4357500L 4357740L 4357920L
USA9
HashMap
These arrows are actually the memory locations of
the TreeSets
TreeSet
Data organization redux 3
But in an object-oriented world, we should really
create a Flight object to store its data and to act
upon it!
 Attributes:
TreeSet of TZData (sortedTZs)
 Methods:
interpolateTimes
addTZData // add a new TZData point
getStart
// get the entrance TZData for the sector
getEnd
// get the exit TZData for the sector
getSortedTZs and setSortedTZs
 This is just a start, I would think there will be
more
We need a Comparator
Compare is an Interface
 We have to tell the Sorted Tree how
to compare TZData objects
 Because Comparator is an
interface, we need to implement its
methods in a new class
package arc.airtraffic.data;
import java.util.Comparator;
class TZDataCompare implements
Comparator<TZData> {
public int compare(TZData tzd1,
TZData tzd2) {
long t1 = tzd1.getTime();
long t2 = tzd2.getTime();
// return -1, 0, +1 if tzd1 <, =, > tzd2
return (int)(t1-t2);
}
}
Click the lightbulb
Flight class
We still need a method that interpolates
any missing TZs to 1-minute intervals
File i/o (page 452)
A File object represents an existing File
File f = new File("full_path_to_file");
You always want to buffer i/o. Disk access is slow, so
read a big chunk at a time into a buffer.
import java.io.*; // need this
class FileReader {
public boolean readFile(String filename) {
try { // this may fail, so check for errors!
File f = new File(filename);
FileReader fileReader = new FileReader(f);
BufferedReader reader
= new BufferedReader(fileReader);
// the file is now open and ready for reading
File i/o (continued)
// Make a place to put a line you read
String line = null; // always initialize variables
while((line = reader.readLine()) != null) {
System.out.println(line); // do something with it
}
reader.close(); // always close the file when done!
} catch (Exception ex) {
ex.printStackTrace();
These parentheses make
return false; // it failed
sure that the string is put into
}
line before the test for null
return true; // It succeeded
} // End of method readFile
} // End of class
Some String methods
ACID,DEPT_APRT,ARR_APRT,POSIT_TIME,CUR_LAT,CUR_LON,GROUNDS
PEED,ALTITUDE
COA497,CLE,MCO,8/7/99 0:01,2487,4907,472,290
UAL1517,CLE,DEN,8/7/99 0:05,2487,4909,502,310
These are the lines we read.
 Data are comma delimited. We must split them.
 Must convert first 3 fields to UAL517_CLE_DEN
 Must convert the date/time to seconds past 1/1/70
 Times are always a pain
 Must store the data in the TZData
Splitting the String
String.split
public String[] split(String regex)
Splits this string around matches of the given regular expression.
// Assume that String line contains our line of data
String[] elements = line.split(","); // an array of Strings
int n = elements.length; // Java arrays know their length
String flightName = elements[0] + "_" + elements[1] + "_"
+ elements[2]; // Array indices start at 0 in Java
int altitude = Integer.parseInt(elements[7]);
int speed = Integer.parseInt(elements[6]);
int lon = -Integer.parseInt(elements[5]);
int lat = Integer.parseInt(elements[4]);
// We still need to get the time…
The time
We have times given as 8/7/99 0:01
String regex = "[/ :]";
note the space
String dates[]
= elements[3].split(regex);
int year = Integer.parseInt(dates[2]);
year = (year > 50) ? year + 1900 : year + 2000;
GregorianCalendar gc
= new GregorianCalendar( year
Integer.parseInt(dates[0]) – 1, // months start at 0
Integer.parseInt(dates[1]),
Integer.parseInt(dates[3]), Integer.parseInt(dates[4]));
long time = gc.getTimeInMillis() * 1000L; // in seconds
Interpolation
Interpolation means adding new data points
between existing ones.
 There are many ways to do this
original points
piecewise
constant
piecewise
linear
But which way (if any) is right for us?
splined
Interpolating TZs
So, what sort of v(t)
should we use for our
model?
Interpolating data
(unfinished business in Flight)
We know
 the points n minutes apart
 the distance between the points
 but maybe v2 ≠ v1
If we assume v varies linearly, it
may not yield arrival at point 2 at t2
Assume the velocity starts to
change linearly at some time ts
 Then we can find the
intermediate distances knowing
the speed.
 If the distances are small , we can
make the changes in altitude,
latitudes, and longitudes
proportional to the changes in d
 Will need times in hours!
But there is a problem!
Interpolating data (continued)
When I wrote this, I woke up
and realized that this is only
half the solution. (Yes, I do
my best work in bed.)
 What if ts < t1 or > t2 ??
 How can that happen?
 if ts = t1,
d12 = (t2 – t1)(v2+v1)/2
= Dtime * average velocity
 So the flight is spending too
little time at the faster speed
Or on third thought . . .
After I had plotted the
interpolated tracks, I found
that this scheme almost
always worked. 
 But what if the plane actually
flew faster than v2 or slower than
v1 in the interval?
 The code was getting very
complicated because the model
kept changing.
Time to rethink the model!
 Instead of solving for the middle
time transition, ts, solve for the
middle speed transition value.
 This should work for every case.
Some more numerics
The number of seconds since 1/1/1970 is a very big number
~ 38.5 * 365 * 24 * 60 * 60 = 1,214,136,000 seconds
It might not have enough significance left after we convert it to
hours. Because we only care about time differences for our
interpolation, we can convert to seconds into the day, and
then to a double:
long secondsInDay = 60L * 60L * 24L;
// if tzt is the seconds past 1/1/1970
long secondsIntoDay = tzt%secondsInDay; // % gives remainder
double hoursIntoDay = ((double)secondsIntoDay)/3600.0;
We keep seconds into the millenium in TZData because it lets
us convert back to a real date and time if necessary.
Good computation always requires worrying about accuracy
Pseudocode for interpolation
interpolate() {
// Get TreeSet. Make new One
// t1 = oldTZtime = first time
// t2 = newTZtime = second time
// do {
// if (t2–t1 > 60 && t2-t1< 3600){
// Calculate n = Dt in minutes
// Calculate GCD d12 in nm
// Convert t1, t2 to hours
// Calculate vm
// For each missing minute {
// calculate the distance flown
// and velocity at t1+(double)n/60
// using the formulas for t >< tc
// latn = lat1
//
//
//
//
//
//
//
//
//
//
//
//
+ (dn/d12)(lat2 - lat1)
do same for lon, altn
Calculate speedn from formula
Make a new TZData object
Insert it into the TreeSet
} End of each minute
} End of need interpolation
if(t2's TZData is last)
break out of do loop;
t1 = t2; t2 = nextTZTime
} // end of do
} End of class
interpolate (method of Flight)
Challenges are: Getting the right units, managing the list iteration, not doing
a lot of extra work, worrying about memory being freed.
Interpolate (continue)
Pseudocode for loading Data (try 2)
// open the data file
// while there are more lines {
//
read a line from the file
//
convert the data
//
insert it into a TZData
//
get the flight name
//
if(the flight is not in keys) {
//
make a new Flight
//
put it into a Flight HashMap
//
}
//
else {
//
get the Flight object
//
add this TZData to it
//
if(duplicate) free memory
//
}
// } // end of while
 Store the data in a
HashMap with Key=
flightName and Value=
Flight object
 Note that a TreeSet
automatically does not
add an element if the
comparator says it is
equal to any existing
ones 
 And this code is
simpler than the
previous try 
Where to put the code to read data?
We have two classes:
 Flight has the methods and data for a flight
 TZData has the information about 1 flight at 1 time
Where do we put the code to read the data into the
Flights? And to manage the Flights?
 Sector class should manage everything in the sector
 Read and store the data
 Answer questions about sector status
How many flights are in the sector now?
Are any of them too close?
How much work are the controllers doing?
Sector class
see below!
what to do about single TZs?
Now we have the data to do
something useful. . .
Question: How many flights are in ZOB48 versus
time?
 Where (in the classes) do we do this calculation?
 It is a Sector issue!
 How do we do it?
 We have the data by Flight and not by time
We could make a Time object and rearrange the TZDatas
Or we could just loop through the Flights to see when they
are active – try this.
 What is the output?
 An array of ints, with number of Flights at each minute
nFlightsVsTime (in Sector)
/**
*@return # flights each minute
*/
public int[] nFlightsVsTime() {
int[] density = newint[1440];
// Be sure the array is zeroed
for(int i = 0; i < 1440; i++)
{density[i] = 0;}
Collection<Flight> coll =
flights.values(); // values
Iterator<Flight> itr
= coll.iterator();
while(itr.hasNext()) {
Flight f = itr.next();
TreeSet<TZData> tzds
= f.getSortedTZs();
Iterator<TZData> itz
= tzds.iterator();
// Load density array
while(itz.hasNext()) {
long time
= itz.next().getTime();
// 86400 seconds in a day
long secIntoDay
= (time%86400L);
int minIntoDay
=(int)(secIntoDay/60L);
density[minIntoDay]++;
}
}
return density;
}
Change Runner
package arc.airtraffic;
import arc.airtraffic.data.Sector;
import java.io.BufferedWriter;
import java.io.FileWriter;
import java.io.IOException;
import java.io.PrintWriter;
import java.util.logging.Level;
import java.util.logging.Logger;
public class Runner {
public static void main(String []
args) {
PrintWriter pw = null;
Sector sector = new
Sector("ZOB48");
sector.loadData("C:\\nbprojects\\
ZOB48\\zob48-8-7.csv");
int[] density = //do the work!
sector.nFlightsVsTime();
try {
// make a buffered output writer
pw = new PrintWriter(new
BufferedWriter(new
FileWriter("densityZOB48.csv")));
pw.println("Time,# Flights");
for (int i = 0; i < 1440; i++) {
if (density[i] > 0) {
String s = i / 60 + ":" +
i % 60 + " " + density[i];
System.out.println(s);
}
String s = i / 60 + ":" + i % 60
+ "," + density[i];
pw.println(s);
}
} catch (IOException ex) {
Logger.getLogger
(Runner.class.getName()).
log(Level.SEVERE, null, ex);
} finally { // Always do this
pw.close(); // close the file
}
} // End of main
} // end of class
It works!
Why are there any
Flights with just 1 TZ?
I imported the output .csv file into Excel and
plotted it.
• Why is there a drop from 9-12 AM?
• Hint: What is the time zone of the data?