VATSIM Analysis. Calvin Kim & Josh Pedowitz. Middlebury College, Middlebury, VT 05753

Transcription

1 VATSIM Analysis Calvin Kim & Josh Pedowitz Middlebury College, Middlebury, VT Abstract. VATSIM is a flight simulation software used by thousands of people every day. Currently, there is no tool to visualize the data generated by this software. In our work we create a data set from the simulations and provide a starting point for a visualization and analysis tool. This paper breaks down this process and explains how the work can be expanded upon. 1 Introduction Virtual Air Traffic Simulation Network (VATSIM) is a flight simulator that was created in July Since its creation it has grown to be the largest online flight simulator in the world with over 209,000 registered users. This user base is made up of people who engage in the simulation as both pilots and air traffic controllers (ATCs) in order to keep the simulation as close to real-life aviation procedures. Due to its realism, VATSIM is not only used as a hobby but also as a training tool for student pilots as well as private and commercial pilots who are looking to improve or brush up on their skills. For example, in 2009 the Israeli airline El Al required upwards of 700 of its new pilots to go through training sessions using VATSIM to provide realistic training before letting them fly with passengers. [1] 1.1 Motivation VATSIM keeps track of tons of data about their current flights including GPS coordinates, current altitude, and true airspeed. All of the data from these simulations is shared in a CSV file which updates every two minutes. The motivation for this project was to help these pilots visualize the data that is interesting to them in a way that is intuitive and easy for them to understand.

2 2 Calvin Kim & Josh Pedowitz 2 Implementation In this section, we will be describing the languages, libraries, and databases we used in our implementation. 2.1 CSV In order to do meaningful analysis on the data, we wanted to be able to aggregate tons of data about flights from throughout the day rather than just looking at data for current active flights. This meant that we needed some way to parse VATSIM s data and somewhere to store it. Thankfully, VATSIM shares their data as a CSV file which means it is easy to work with. This data can be found at Scraper We chose to use Python as our programming language because it s a language that we were both familiar with and one that is relatively easy to read and understand. In addition, there is a library for Python called pandas that is extremely useful when performing statistical operation and doing analysis on large data sets. Our scraper had two main sections to it. The first dealt with inserting data about active flights into our database and the second dealt with building a data set of every flight that has occurred since we started the project. Currently the active flights portion of the database is not nearly as useful as the main flights table so we will focus on the main flights. In the future, however, the active flights will prove to be very useful. In order to insert our flights into the table with the necessary information we had to not only grab the relevant data from the CSV, but also perform a few calculations on that data. One of the main things that we calculated on our own was ground time. To do this we used airport location data, current GPS location of the flight, and the current ground speed of the plane. If planes were within a five mile radius of their departure airport and moving at a speed of less than 15 knots we decided they were still on the ground. Once outside of the radius or above the speed we logged the time and subtracted the time that the flight

3 VATSIM Analysis 3 was begun. This gave us ground time at the departure airport. Once the flight was within a five mile radius of the destination airport and moving at a speed of less than 15 knots we decided the plane was back on the ground again which helped us figure out the total duration of the flight. Since the CSV is updated about every two minutes, we scheduled the scraper to run every two minutes using cron so that we don t miss any data. In order to make sure we don t run our scraper on the same update twice we used a field that shows what time the CSV was updated and compare our last update with it. 2.3 Database We decided to use sqlite3 as our database to store our data since the data is stored locally and because we were both relatively familiar with SQL commands. SQL is also a relational database that meets the needs of our dataset since the data that we re working with is highly structured. 2.4 Data Analysis One of the main motivations for using Python was because of the open source pandas library, which uses numpy. pandas is a powerful and easy to use library that includes functions and data structures that make data analysis and visualization fast and easy. 3 Results Our project consisted of two major visualizations: the Misery Map (Figure 1) and the Traffic Graph (Figure 2). For both of these visualization, we set the threshold for a bad ground time to be anything over an hour. 3.1 Misery Map The bottom layer of the Misery Map displays of all of the airports in the world as a way of representing a map. On the layer, the map colors airports on a gradient between blue

4 4 Calvin Kim & Josh Pedowitz and red depending on how bad the average ground time is for that airport (blue = good, red = bad). The size of the airport is also increased based on it s popularity relative to other airports. The popularity of an airport is based on the number of flights that depart and arrive at that airport over time. Fig. 1. A map of the airports in the U.S. 3.2 Traffic Graph The Traffic Graph is a way to both visualize the overall traffic of VATSIM over time as well as the number of flights that had bad ground time vs. good ground time on each specific day. The graph is an area map with number of flights on the x-axis and date on the y-axis. The blue area displays the number of flights with good ground time on that day and the red area displays the number of flights with bad ground time. This allows us to see things such as specific days where ground time was especially bad or days that just had very high traffic.

5 VATSIM Analysis 5 Fig. 2. A graph of the number of flights per day. 3.3 Issues We ran into a number of problems while working on this project. Fields in the CSV were often not populated and the CSV had many sections, which were not formatted as CSV, that we had to sort through and ignore in order to get the data we needed. The Airports database had incorrect or incomplete names as well as certain unicode characters that couldn t be handled by the database so we had to go into the SQL file to manually change the characters ourselves. Another issue we ran into is that cron only runs when the computer is on which means that in order to regularly fetch data, we need a dedicated computer or server that is always on. There was also the occasional human error that would mess up our data. For example, there was a flight with a ground time of over 24 hours due to human error. This meant that we had to set a threshold for what data we would accept as kosher and remove the remaining outliers.

6 6 Calvin Kim & Josh Pedowitz 4 Future Work While we accomplished what we set out to do this semester, there are a number of improvements we would like to make. In the future, we would like to move our scraper onto a dedicated server so that we can run the program continuously and get all of the data points. In addition, we would like to move our database into an online database server such as Amazon RDS so that the data isn t just stored locally. The next step for this project would probably be to create an app or an webpage for our results. This would allow more people to access what we ve accomplished and would provide us with a space to create a more scalable product. On this app we would display information about current flights and past flights in addition to our analysis. Users would be able to see their flight routes, statistics about themselves and more. We would also like to make our visualizations dynamic instead of static graphs like we have now. We could potentially move our visualizations over to d3.js which would allow us to have clickable nodes on our map and would allow our graphs to be interactive. Having interactive maps would enable us to include functionality such as letting a user select a date range to use for either visualization or even setting their own threshold for bad ground time. References 1. Wikipedia. Virtual air traffic simulation network wikipedia, the free encyclopedia, [Online; accessed 14-December-2015].