GeoTweet: A Geographical Analysis of Word Frequency in Tweets Across the U.S.

Transcription

1 GeoTweet: A Geographical Analysis of Word Frequency in Tweets Across the U.S. Garrett Griffin and Hanna Nowicki Abstract GeoTweet is a convenient tool for collecting tweets sent by users in the United States, storing them in a database, and performing some simple word-frequency linguistic analysis by geography. We implemented a very simple caching system utilizing the database, where if a user searches for a tweet, it stores the results from the search and allows our user to pull them up without the processing time that it would normally have taken to search for the frequency of a word in the tweets that this user has collected. We have used MongoDB to store tweets as well as to act as a cache, Python to process tweets and interact with MongoDB, JavaScript to process user interaction and display the results that we receive, and D3 to enact the actual visualization of word frequency in the tweets. With GeoTweet, we set out to create a tool for neatly visualizing tweets according to the geographical location of the person who tweeted them, specifically relegated to tweets sent within the United States. This, we determined, was a project that would be both easy to use for any user and easy to modify for any developer. Alongside this, our project would provide some interesting, tangible artifacts for linguistic analysis; the graphs produced by our word frequency visualization of tweets would be likely to reveal some interesting linguistic trends, confirming certain of our expectations and rejecting others. Ultimately, what we aimed to offer our users was an efficient, easy to use tool to create simple and elegant visualizations of data that provide more information upon a given user s interaction with the visualization. Goals Our goal for this project was to create a useful visualization tool using large amounts of collected data. We both felt that we hadn t had a chance to work on a project like this in our time at Middlebury, so this implementation project presented a good opportunity to bring our other interests into our computer science work. Another goal that we had was to broaden our understanding of working with databases and creating a simple web application. 1

2 Figure 1: A search for the word beach Accomplishments Ultimately, what we produced was a system that allows a user to run a script to collect and parse tweets, storing them in an instance of MongoDB and then run a web server, which will offer them a box in which to input a search term that they wish to find amongst the tweets that they have collected. Finally, the server will visualize the frequency of the search term as a heat map of the United States, according to state. As can be seen in Figure 1, the user has entered and searched for the term beach. The heat map of the United States is darkened in areas where the term appears more frequently in the collected tweets, with the term appearing most frequently in the state of Hawaii. If the term does not appear at all in the tweets collected from a given state, it will be displayed as white, as, for example, in Alaska and South Dakota. The final feature that needs a brief introduction is the tooltip, which appears in the approximate middle of the top of the state whenever the cursor is placed within the boundaries of a state; it contains information on the search term, such as the percentage of tweets in which the word appears, the total number of tweets collected from that state, the total number of tweets collected from that state containing the given word, and the average follower count of the users that sent tweets with the search term in them. 2

3 Implementation In implementing this project, we used Python and JavaScript and various libraries for these languages, as well as examples of the usage of these libraries. The technology at the base of our chain was Python, which we used to process tweets, parsing out the parts of the tweet in which we were interested, and then to store these tweets in an instance of MongoDB[5]. For our interactions with Twitter using Python, we used a wrapper library for the Twitter API[9] called tweepy, which allows our user to filter a live-stream of tweets whenever the script is running. We have used Twitter s coordinate filter to decide what tweets we were interested in, providing tweepy with a list of bounding-boxes within which to collect tweets. Specifically, we have a bounding-box for the continental United States, for Hawaii, and for Alaska. From there, we only store tweets with the country code US, and we store tweets according to their state abbreviation, searching the data for either a state abbreviation or for a state name, in which case we have mapped state names to state abbreviations and store the tweet accordingly. If the location data of a given user is improperly formatted or incomprehensible (for example, we would not be able to tell what state a tweet in Nueva York belongs to), we will simply throw it out. Next, we have a search bar that we implemented using jquery and Bootstrap.[2] This bar listens for the user hitting the enter key or pressing the submit button, and when this event occurs, we send a request to a Python script that is running a server on port 5000 of the localhost using flask[6], a very lightweight framework for running Python servers. This Python server, upon receiving a request that contains a search term, checks our cache to see if the term has been searched before, and if it has, returns the JSON object representing the counts of that search term throughout various states. If the term has not been searched prior to this point, the server iterates over every tweet object in our database instance, checking each tweet to see if it contains the search term and if it does, incrementing the count of tweets for that state while keeping track of the average number of followers who sent tweets containing that word. Eventually, either way, we will receive a JSON-formatted object that has state abbreviations mapped to the various variables that we display in our tooltip. The final aspect of our project is the map visualization itself. The map is displayed using the topojson JavaScript library, which receives a JSON file that contains coordinates defining the borders of states and then creates an Albers projection of the United States based on these coordinates. After this, we bind our data to each state using the JSON object that is returned after our search. We color states according to the relative frequency of a tweet in a given state; so, one tweet in Montana is, in theory, equivalent to about one hundred tweets in California. Finally, we add tooltips on top of each state, so that whenever our mouse cursor is within the boundaries of a given state, a tooltip pops up, displaying the information about tweets containing the search term in that given state. 3

4 Figure 2: A search for the word hella Results Our Twitter visualization tool eventually returned several interesting results for different search terms. Displayed here are a few examples with brief explanations and proposals as to why we have gotten these results. One of the use cases for our application that we found most interesting is local slang and dialect analysis. In Figure 2, we can see that the search term hella is used more and more frequently the further west we go, which confirms what we would expect, as the term is originally Northern California slang. As we can see in Figure 3, a striking example of local language usage in tweets is the frequency of the occurrence of the word aloha in the state of Hawaii. This map does not take too much explaining: the reasons why native Hawaiians are using the word aloha at a much higher rate than other Americans are self-explanatory due to local language usage. As illustrated by Figure 4, one example of geographical themes or trends in tweets can be found with the search term ski, which occurs at an extremely high frequency in the American northeast, particularly in the state of Vermont, where skiing is less a hobby and more a lifestyle. As can be seen from these few examples, our application has many different use cases in linguistic analysis and can display interesting trends in geographical analysis. 4

5 Figure 3: A search for the word aloha Figure 4: A search for the word ski 5

6 Difficulties In order to accomplish what we set out to, we needed to use a few technologies that neither of us was very familiar with. We both took software development this semester, which was a great help later in the semester because of its focus on Javascript and databases. However, at the start of the semester neither of us had a good understanding of Javascript or databases, and this lack of foundation gave us some difficulties in creating a plan for implementation. The project was introduced very early in the semester, so we had to make long-term decisions based on little knowledge, and we ran into some problems later because of this. Looking at our data now and how structured it is, we realize perhaps that we should have used a SQL database instead of MongoDB, but it took us too long to come this realization. Another area of the implementation that presented some difficulties was using D3 for the visualizations. We found D3 to be somewhat finicky and our inexperience with programming in it made using it frustrating at times. We had intended to integrate node.js into the project, and had tried for a while to accomplish this, but neither of us had a good enough understanding of it to get it completely functional before the final deadline. We ended up having to move away from the idea of using node.js near the end of the semester and instead used a python web server to run our python code alongside our Javascript code. Though this ended up working out, it gave us a lot of difficulty up until the project due date. As expected, we ran into difficulties with time management, which was especially problematic because of the project s self-guided nature. The project demo gave us a goal to work towards in the first half of the semester, but following the demo and demo presentations, we had difficulties structuring our time to keep on a good trajectory in the second half of the semester. Because of this, we ended up having to complete a large portion of the project near the end of the semester, which did not leave us with the freedom to experiment and think deeply about the best ways to accomplish things In general, as we went along, we found that our initial goals for the project had been grander than we were capable of under the time constraints, and ended up having to scale our project back in order to meet the final deadline. Future Work In the future, we would like to add a range of visualizations to GeoTweet for users. These added visualizations, such as a zooming function that would allow users to view statistics of word usage in specific towns or cities in the U.S., or a timeline that would allow users to analyze tweet data at different times, would build on the current map visualization and provide more tools to allow users to effectively examine linguistic trends in tweets across the US. When we showed our program to another student to ask their opinion about what they would like to see in addition to the visualization that we have in place, the student stated that more statistics on the posters of the tweets, such as their 6

7 age, would be helpful in analyzing trends in the data. From the development side, we would like to make our program more efficient, so new user queries would display much quicker. Currently, our program displays results immediately when queried for a word it has previously searched, as these results are stored when they are returned, but a newly searched word takes around 20 seconds to load. A feature that we weren t able to incorporate into GeoTweet during this semester is updating tweet data in real-time. Currently, we have a twitter stream parser that pulls tweets into a Mongo database when we run it, and then we run the program from these stored tweets instead of using current tweets to update the statistics we display. This would be a helpful feature to include in the future because trends in social media are continually changing, unless we are frequently streaming more tweets into our database, the information displayed does not portray this well. Another improvement that we could make in our next steps would be to improve our parser. As it stands, we take the users input and collect tweets from the database that match this input exactly. In the future, we could change our parser to deal with misspellings and punctuation in order to display more useful results for users. We would have also like to get our webpage hosted on an external server, since we are currently running the program locally on our computers, but we did not have time to accomplish this during the class. From a linguistics point of view, there are many more complex analyses that we could make in addition to our analysis of word frequency according to state. Though this semester we focused on learning new technologies to make this project possible, it would be interesting to focus more on research in linguistics to add depth and utility to the project. We researched a few papers on twitter linguistic analyses like determining personality traits based on twitter profiles and language usage [8], using machine learning to classify sarcasm in tweets [4], and the linguistic register [7] that would be interesting to return to for future work. Initially, we had thought that it would be interesting to analyze word usage on a global scale, and in the process incorporating translation services into our project, so a user could search a word in one language and view statistics on the equivalent words in different languages across the globe. We quickly realized that we would have to scale our project back, but as twitter is widely used around the world, incorporating more countries data into our visualizations would increase GeoTweet s usefulness and relevance. Conclusion In spite of the challenges, we created a useful and easy to use tool to analyze tweets by state across the U.S. through this project. The project produced some interesting results and acted as a way for us to improve our coding skills and widen our knowledge of different technologies like D3, Javascript, and MongoDB. We are interested in continuing to develop this project in the future. 7

8 References [1] Douglas Biber. Dimensions of register variation: A cross-linguistic comparison. Cambridge University Press, [2] Bootstrap. Bootstrap Components. components/. Accessed: [3] Mike Bostock. Bostock s Blocks. Accessed: [4] CC Liebrecht, FA Kunneman, and APJ van den Bosch. The perfect solution for detecting sarcasm in tweets# not [5] MongoDB. The MongoDB 3.2 Manual. manual/. Accessed: [6] Armin Ronacher. Flask. Accessed: [7] Harold Schiffman. Linguistic Register. ~haroldfs/messeas/regrep/node2.html. Accessed: [8] Chris Sumner, Alison Byers, Rachel Boochever, and Gregory J Park. Predicting dark triad personality traits from twitter usage and a linguistic analysis of tweets. In Machine Learning and Applications (ICMLA), th International Conference on, volume 2, pages IEEE, [9] Twitter. Twitter API Overview. api. Accessed: [10] Nick Qi Zhu. Data Visualization with D3. js Cookbook. Packt Publishing Ltd,