Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Analyzing TZ Data James A. Rome TGA August 2010 What are we trying to do? I was hired by NASA to analyze air traffic in ZOB48 I had access to ETMS data The most useful data were TZ messages But as we saw, they are highly quantized 1–4 minute time intervals (no seconds) -> 6 nm Multiple TZs from one flight at a given time (due to multiple radar reports) Situation is far from ideal. What do we do? First step is to look at the data and see how bad it is, and what we can (or cannot) do with it. We will need to calculate things like GCD Handling data is non-trivial ACID,DEPT_APRT,ARR_APRT,POSIT_TIME,CUR_LAT,CUR_LON,GROUNDSPEED,ALTITUDE COA497,CLE,MCO,8/7/99 0:01,2487,4907,472,290 UAL1517,CLE,DEN,8/7/99 0:05,2487,4909,502,310 Initially, the data are arranged by time There are 3900 TZs Why do we need the airports? Because the flight number applies to several legs of a flight The big question is how should we arrange these data? This depends on what we want to do with the data Do any planes “collide” (<1000 ft or <5 nm)? What path does the flight take through the sector? When does the controller do something to the flight? How many flights are there at any time? ... How good are the data? The TZ messages have inherent error bars: Lat/Lons are in minutes 1 minute = 1 nm of latitude, and somewhat less in longitude. So positions might be off by a mile or so. Time is in minutes, so if seconds are rounded down, may be up to a minute earlier than actual This leads to distance inaccuracy of (speed/60) nm Speed is in knots (nm/h) and is accurate enough Altitude is in 100s of feet, so may be off by 100 ft The distances can be off by 6–8 miles. 5 miles is the required separation distance between planes. How should we handle the data? You could not do this with a TI calculator Maybe you could use Excel if the number of flights is small, but you would quickly find that you would be unable to answer all the above questions We will have to write a computer program to make any progress Every scientist or engineer who wants to solve a new problem will get to this point. If you cannot write a computer program, you cannot succeed! It is time to write a program We are going to calculate the great circle distance (GCD) between two Lat/Lon points This will be part of a utility class because we will need these tools to do more interesting things. The methods (things it can do) should be Available to other programs (public) Because there is no need for any changing data in the class as a whole, the methods can be static This implies that multiple methods can call the static method at the same time without conflict (thread safe) Static methods can be called without instantiating the class via ClassName.method No new operator is needed Make ZOB48 Project in NetBeans Select File, New Project Packages In a language as large as Java, it is important to organize your files to avoid name conflicts Your method “openFile” might conflict with a method of the same name in some other part of Java Packages create a unique way of naming your files by convention, the package name is the reverse of your URL: gov.ornl.my_project_name Packages also denote accessibility boundaries for methods and fields private is in my class protected is in my package public is open access You can move things around later via refactoring Make a new package Note the Directory Tree this creates Package name = arc.airtraffic.util (arc is left-over from my course for ARC) What is an object? An object is a thing that has properties (attributes or fields) and can do stuff (methods). In Java, almost everything is an object An object is defined by a template called a class (the computer code that is used to create the object). An object does not exist until it is instantiated. What type of objects might we want to create for our ZOB48 analysis? A TZ object, a Flight object, a Utility object, … What is a class? A class is the template for an object. Unless a class has methods that are declared public static, you must instantiate the object from the class template before using it. new does this. Vector vec = new Vector(); // create vec To call a method of a class, you use the variable name (e.g., vec above): vec.add("This is a String"); If the method is static, you use the class name: double sinpi = Math.sin(3.14159265); double pi = Math.PI; // a static field Make a new class You could have created the package in this dialog. NetBeans will create a new Java file for you. The name of the file must match the name of the class. Capitalization counts! This file is LatLon.java It is placed in the bottom of the package tree. Special JavaDocs fields LatLon utility class Write JavaDoc comments as you go. This is a JavaDocs comment A static method can be called without instantiating an instance of its class End-of-line comment We will need the radius of the earth. final = a constant PI is a static field of the Java Math class How do you know about Math.PI? JavaDocs Package names documents all Java classes If you fill in the data, you can make JavaDocs of your own code too. NetBeans can give you the JavaDocs for any method if you configure it. Class names NetBeans tip: How to change JRE Add method to calculate angle Calling static methods I usually break up long formulas using temporary variables (trm1, trm2) to aid in debugging Method to calculate the distance We never really want the angle between the points at the center of the Earth, so make a method we can call directly to get the great-circle-distance. Make the JavaDocs Here are your JavaDocs • As you type your code, NetBeans will display your choices, and their JavaDocs. • You can double-click a choice to insert it into your code. • NetBeans will even guess at the names of your variables to use as arguments! Time to test the code Although NetBeans finds typos and bad usage for you, it does not guarantee that your math is correct. So let us make a test program to find the distance between two lat/lon points. We will make a new class called “Runner” in the arc.airtraffic package. I decided to calculate the distance from (approximately) Toledo to Cleveland Google is great Toledo Cleveland Note, I eyeballed the lat/lons The Runner class array of Strings main signature always NetBeans matches braces Be careful about parentheses, which determine the order of computation. I always use more than needed. Click Run, Run main project . . . You must tell NetBeans where to find main() You can have severaal main() methods in a project It works! TZ object What might a TZ object consist of? Attributes for one flight at one time Time, lat, lon, altitude, speed, and a combination of the flight number, departure and destination airports. E.g., NWA527_CLV_MSP Actions it can do Store a new (or updated) value of an attribute Give you (another part of your program) any of these values Objects generally do not allow other objects to access their attributes directly. Encapsulation TZ pseudocode public methods and attributes accessible to all other objects Pseudocode lets you concentrate on what your code does, rather than how to write it, although this example is so simple, it is real code. public class TZData { everything after // is a comment // Class attributes private int altitude; // in hundreds of feet? private double latitude; // in radians private double longitude; // in radians private String acid; // the aircraft ID private double speed; // knots private long time; // time in seconds past 1/1/70 // Class methods capital S means it is a class // next slide content goes here } accessible only to TZData objects TZData methods public TZData() {;} // default constructor public TZData(int alt, double lat, double lon, double speed, String acid, long time) { altitude = alt; latitude = lat; longitude = lon; this.speed = speed; this.acid = acid; // Only can use = for Strings this.time = time; } These are the type of thing returned to the caller // These are the setters. Note the capitalization public int getAltitude() {return altitude;} public double getLatitude() {return latitude;} public double getLongitude() {return longitude;} public double getSpeed() {return speed;} public String getAcid() {return acid;} public long getTime() {return time;} TZData methods (continued) // the setter methods void means that nothing is returned to caller public void setSpeed(double spd) {speed = spd;} public void setLatitude(double lat) {latitude = lat;} public void setLongitude(double lon) {longitude = lon;} // This time we will use a constructor for String public void setAcid(String id) {acid = new String(id);} public void setAltitude(int alt) {altitude = alt;} public void setTime(long time) {this.time = time;} } // the end of the class (repeated from slide 1} Notice how we used lowercase set and get for each of these methods to prepend the variable name which we capitalized. This makes this class into a Java Bean. Java Beans can be added as elements into NetBeans no “return” needed for voids What have we done? NetBeans has done a lot of organizing for you In the ZOB48 project directory is a src tree The package name was translated into a directory tree The class file is placed in the directory at the bottom of the tree Note: The name of the .java file must always be the same as the class name Time to enter the class code NetBeans can do the grunge work of typing in the getters and setters Getters and setters Right-click the TZDate.java file name, select Refactor, Encapsulate Fields Our TZData class is complete We could write a main method to test our class, but TZData is an important, but rather uninteresting class, so we will skip that for now. How to read in data We have a multi-pronged problem to solve 1.How to actually read data from a file 2.What to read the data into Clearly we will read each row of data into a TZData object. (That's why we made them!) But there are 3900 lines of data. We need a way (or several ways) of organizing these 3900 objects into things we can access easily. 3.How to convert the strings we read into the right types of data. Look before you leap!!!!!!!!! I cannot emphasize how important it is to look hard at all the issues before you start to code. This is actually the fun part of writing a program I do woodworking. I will spend a month figuring out how to make a jig to do something (e.g., a wide finger joint which needs 1/64" accuracy). We are going to go though my thought process on how to read and store the data. Many times . . . Organizing TZData objects Assume we have read in the data. How do we organize it? Depends on the problem . . . Can we tell if any flights are too close? Need positions of all flights at a given time for each time Time TZ1 TZ2 TZ3 TZ4 TZ5 0:00:01 NWA1 AAL3 UAL7 0:00:03 AAL3 UAL7 DAL8 NWA2 NWA3 0:00:05 AAL3 UAL7 DAL8 NWA2 NWA3 0:00:06 AAL3 DAL8 DAL9 NWA2 TZ6 … ... Kind of a 2-D array. But the rows have different lengths. Hard to search. Java has Collections We will put the TZData at a given time in a Vector <E> is the type of Object in the Vector We will organize the Vectors using I originally chose a HashMap, a TreeMap but decided that it would be useful to have the data ordered by time. Using the two is essentially the same. A TreeMap allows you to store Key-Value pairs, sorted by the key. Key = Long time Value = Vector of TZData A TreeMap is an ordered HashMap. The Hash allows very fast searching through the list of keys. Example: 24 is a hash of "cat" "cat" = 3 + 1 + 20 = 24 It is a bad hash: "act" = 24 also – a collision Hash Values 1 a 2 3 ab ba 4 ac baa 5 6 cab Data organization The Keys are actually Long objects Long key = new Long(4356780L); Key Value Vectors of TZData Objects at given times 4356780L AAL2 NWA4 DAL10 NWA6 4356900L AAL2 NWA4 NWA6 COA2 4357020L COA2 AAL5 NWA6 4357140L COA2 AAL5 NWA7 4357260L NWA7 DAL2 DAL2 COA2 USA5 4357500L TreeMap These arrows are actually the memory locations of the TZData Vectors. These are called pointers. Pseudocode for loading Data // open the data file // while there are more lines { // read a line from the file // convert the data // insert it into a TZData // get the time // if(the time is not in keys) { // make a new TZ Vector // add TZData to Vector // put it into TreeMap // } // else { // get Vector for this time // see if this flight is there // if (it is there) toss it // else add it to the Vector // } // } // end of while There are many ways to do this We will assume the data are not in order Notice that we have now introduced the new concepts of looping (can you find 2?) logical branching (if, else, else if) But they naturally express how you have to think about the problem Thinking more on the problem . . . If we are going to decide whether any planes "hit" each other, we need to know all the flights in the air at once. But the TZs seem to occur at different intervals for different flights, and they are not consistent at all! We cannot implement the previous data arrangement because we are going to have to "manufacture" the missing data Arrange the data by flight, ordered by time Use interpolation to create new TZData objects at every minute Then rearrange the data as needed How???? Data organization redux 2 • We do not need the flight names sorted. So use a TreeSet with flight name as key. • We do need the times sorted. Implies that the times can be compared. Key Value TreeSet of times (must be Longs) NWA3 4356780L 4357020L 4357260L 4357440L 4357680L AAL6 4356600L 4356780L 4357080L 4357320 COA8 4357080L 4356260L 4357320L NWA7 4357020L 4357260 UAL1 4357260L 4357500L 4357500L 4357740L 4357920L USA9 HashMap These arrows are actually the memory locations of the TreeSets TreeSet Data organization redux 3 But in an object-oriented world, we should really create a Flight object to store its data and to act upon it! Attributes: TreeSet of TZData (sortedTZs) Methods: interpolateTimes addTZData // add a new TZData point getStart // get the entrance TZData for the sector getEnd // get the exit TZData for the sector getSortedTZs and setSortedTZs This is just a start, I would think there will be more We need a Comparator Compare is an Interface We have to tell the Sorted Tree how to compare TZData objects Because Comparator is an interface, we need to implement its methods in a new class package arc.airtraffic.data; import java.util.Comparator; class TZDataCompare implements Comparator<TZData> { public int compare(TZData tzd1, TZData tzd2) { long t1 = tzd1.getTime(); long t2 = tzd2.getTime(); // return -1, 0, +1 if tzd1 <, =, > tzd2 return (int)(t1-t2); } } Click the lightbulb Flight class We still need a method that interpolates any missing TZs to 1-minute intervals File i/o (page 452) A File object represents an existing File File f = new File("full_path_to_file"); You always want to buffer i/o. Disk access is slow, so read a big chunk at a time into a buffer. import java.io.*; // need this class FileReader { public boolean readFile(String filename) { try { // this may fail, so check for errors! File f = new File(filename); FileReader fileReader = new FileReader(f); BufferedReader reader = new BufferedReader(fileReader); // the file is now open and ready for reading File i/o (continued) // Make a place to put a line you read String line = null; // always initialize variables while((line = reader.readLine()) != null) { System.out.println(line); // do something with it } reader.close(); // always close the file when done! } catch (Exception ex) { ex.printStackTrace(); These parentheses make return false; // it failed sure that the string is put into } line before the test for null return true; // It succeeded } // End of method readFile } // End of class Some String methods ACID,DEPT_APRT,ARR_APRT,POSIT_TIME,CUR_LAT,CUR_LON,GROUNDS PEED,ALTITUDE COA497,CLE,MCO,8/7/99 0:01,2487,4907,472,290 UAL1517,CLE,DEN,8/7/99 0:05,2487,4909,502,310 These are the lines we read. Data are comma delimited. We must split them. Must convert first 3 fields to UAL517_CLE_DEN Must convert the date/time to seconds past 1/1/70 Times are always a pain Must store the data in the TZData Splitting the String String.split public String[] split(String regex) Splits this string around matches of the given regular expression. // Assume that String line contains our line of data String[] elements = line.split(","); // an array of Strings int n = elements.length; // Java arrays know their length String flightName = elements[0] + "_" + elements[1] + "_" + elements[2]; // Array indices start at 0 in Java int altitude = Integer.parseInt(elements[7]); int speed = Integer.parseInt(elements[6]); int lon = -Integer.parseInt(elements[5]); int lat = Integer.parseInt(elements[4]); // We still need to get the time… The time We have times given as 8/7/99 0:01 String regex = "[/ :]"; note the space String dates[] = elements[3].split(regex); int year = Integer.parseInt(dates[2]); year = (year > 50) ? year + 1900 : year + 2000; GregorianCalendar gc = new GregorianCalendar( year Integer.parseInt(dates[0]) – 1, // months start at 0 Integer.parseInt(dates[1]), Integer.parseInt(dates[3]), Integer.parseInt(dates[4])); long time = gc.getTimeInMillis() * 1000L; // in seconds Interpolation Interpolation means adding new data points between existing ones. There are many ways to do this original points piecewise constant piecewise linear But which way (if any) is right for us? splined Interpolating TZs So, what sort of v(t) should we use for our model? Interpolating data (unfinished business in Flight) We know the points n minutes apart the distance between the points but maybe v2 ≠ v1 If we assume v varies linearly, it may not yield arrival at point 2 at t2 Assume the velocity starts to change linearly at some time ts Then we can find the intermediate distances knowing the speed. If the distances are small , we can make the changes in altitude, latitudes, and longitudes proportional to the changes in d Will need times in hours! But there is a problem! Interpolating data (continued) When I wrote this, I woke up and realized that this is only half the solution. (Yes, I do my best work in bed.) What if ts < t1 or > t2 ?? How can that happen? if ts = t1, d12 = (t2 – t1)(v2+v1)/2 = Dtime * average velocity So the flight is spending too little time at the faster speed Or on third thought . . . After I had plotted the interpolated tracks, I found that this scheme almost always worked. But what if the plane actually flew faster than v2 or slower than v1 in the interval? The code was getting very complicated because the model kept changing. Time to rethink the model! Instead of solving for the middle time transition, ts, solve for the middle speed transition value. This should work for every case. Some more numerics The number of seconds since 1/1/1970 is a very big number ~ 38.5 * 365 * 24 * 60 * 60 = 1,214,136,000 seconds It might not have enough significance left after we convert it to hours. Because we only care about time differences for our interpolation, we can convert to seconds into the day, and then to a double: long secondsInDay = 60L * 60L * 24L; // if tzt is the seconds past 1/1/1970 long secondsIntoDay = tzt%secondsInDay; // % gives remainder double hoursIntoDay = ((double)secondsIntoDay)/3600.0; We keep seconds into the millenium in TZData because it lets us convert back to a real date and time if necessary. Good computation always requires worrying about accuracy Pseudocode for interpolation interpolate() { // Get TreeSet. Make new One // t1 = oldTZtime = first time // t2 = newTZtime = second time // do { // if (t2–t1 > 60 && t2-t1< 3600){ // Calculate n = Dt in minutes // Calculate GCD d12 in nm // Convert t1, t2 to hours // Calculate vm // For each missing minute { // calculate the distance flown // and velocity at t1+(double)n/60 // using the formulas for t >< tc // latn = lat1 // // // // // // // // // // // // + (dn/d12)(lat2 - lat1) do same for lon, altn Calculate speedn from formula Make a new TZData object Insert it into the TreeSet } End of each minute } End of need interpolation if(t2's TZData is last) break out of do loop; t1 = t2; t2 = nextTZTime } // end of do } End of class interpolate (method of Flight) Challenges are: Getting the right units, managing the list iteration, not doing a lot of extra work, worrying about memory being freed. Interpolate (continue) Pseudocode for loading Data (try 2) // open the data file // while there are more lines { // read a line from the file // convert the data // insert it into a TZData // get the flight name // if(the flight is not in keys) { // make a new Flight // put it into a Flight HashMap // } // else { // get the Flight object // add this TZData to it // if(duplicate) free memory // } // } // end of while Store the data in a HashMap with Key= flightName and Value= Flight object Note that a TreeSet automatically does not add an element if the comparator says it is equal to any existing ones And this code is simpler than the previous try Where to put the code to read data? We have two classes: Flight has the methods and data for a flight TZData has the information about 1 flight at 1 time Where do we put the code to read the data into the Flights? And to manage the Flights? Sector class should manage everything in the sector Read and store the data Answer questions about sector status How many flights are in the sector now? Are any of them too close? How much work are the controllers doing? Sector class see below! what to do about single TZs? Now we have the data to do something useful. . . Question: How many flights are in ZOB48 versus time? Where (in the classes) do we do this calculation? It is a Sector issue! How do we do it? We have the data by Flight and not by time We could make a Time object and rearrange the TZDatas Or we could just loop through the Flights to see when they are active – try this. What is the output? An array of ints, with number of Flights at each minute nFlightsVsTime (in Sector) /** *@return # flights each minute */ public int[] nFlightsVsTime() { int[] density = newint[1440]; // Be sure the array is zeroed for(int i = 0; i < 1440; i++) {density[i] = 0;} Collection<Flight> coll = flights.values(); // values Iterator<Flight> itr = coll.iterator(); while(itr.hasNext()) { Flight f = itr.next(); TreeSet<TZData> tzds = f.getSortedTZs(); Iterator<TZData> itz = tzds.iterator(); // Load density array while(itz.hasNext()) { long time = itz.next().getTime(); // 86400 seconds in a day long secIntoDay = (time%86400L); int minIntoDay =(int)(secIntoDay/60L); density[minIntoDay]++; } } return density; } Change Runner package arc.airtraffic; import arc.airtraffic.data.Sector; import java.io.BufferedWriter; import java.io.FileWriter; import java.io.IOException; import java.io.PrintWriter; import java.util.logging.Level; import java.util.logging.Logger; public class Runner { public static void main(String [] args) { PrintWriter pw = null; Sector sector = new Sector("ZOB48"); sector.loadData("C:\\nbprojects\\ ZOB48\\zob48-8-7.csv"); int[] density = //do the work! sector.nFlightsVsTime(); try { // make a buffered output writer pw = new PrintWriter(new BufferedWriter(new FileWriter("densityZOB48.csv"))); pw.println("Time,# Flights"); for (int i = 0; i < 1440; i++) { if (density[i] > 0) { String s = i / 60 + ":" + i % 60 + " " + density[i]; System.out.println(s); } String s = i / 60 + ":" + i % 60 + "," + density[i]; pw.println(s); } } catch (IOException ex) { Logger.getLogger (Runner.class.getName()). log(Level.SEVERE, null, ex); } finally { // Always do this pw.close(); // close the file } } // End of main } // end of class It works! Why are there any Flights with just 1 TZ? I imported the output .csv file into Excel and plotted it. • Why is there a drop from 9-12 AM? • Hint: What is the time zone of the data?