Predicting the NFL Using Twitter. Shiladitya Sinha, Chris Dyer, Kevin Gimpel, Noah Smith

Transcription

1 Predicting the NFL Using Twitter Shiladitya Sinha, Chris Dyer, Kevin Gimpel, Noah Smith

2 Disclaimer This talk will not teach you how to become a successful sports bettor.

3 Questions What is the NFL and what about it are we predicting? Why are we using Twitter?

4 NFL = National Football League

5 What about the NFL are we predicting? Two game outcomes are commonly bet upon: Winner with the spread (Winner WTS) Over-Under Outcomes are determined at the end of the game using numbers set by bookies before the game Due to transaction costs, an accuracy of 53% is needed to be profitable

6 Point Spread and Over Under Point Spread: Signed number used to determine winner with the spread (WTS) given game score If home_score + spread > away_score: Home wins WTS < away_score: Away wins WTS = away_score: Push Over-Under: Betting line for total points scored by the two teams; bettor chooses over or under Push if point differential equals point spread or total points equals over-under

7 Point Spread and Over Under Atlanta Falcons win a home game against the Philadelphia Eagles But did they win with the spread? Point Spread Outcome < - 7 Falcons lose WTS > - 7 Falcons win WTS = - 7 Push

8 Why use Twitter? Twitter has been used to predict and/or explain: Opinion Polls Elections Spread of contagious diseases The stock market Movie revenue Food poisoning Civil unrest in Latin America and the Middle East Citations of scientific papers...and the list goes on Why not sports?

9 Why use Twitter to predict NFL games? Structure of the NFL 32 Teams spread geographically across United States. Regular season partitioned into 17 weeks Roughly a week between games for a given team Participation of fans and spectators Discussion on social media, including Twitter Sports betting Wisdom of the crowd

10 Fan Participation

11 Data Gathering

12 Data Gathering Current and historical NFL game data from NFLdata.com Tweets from seasons via Twitter Garden Hose stream (10% of all tweets) How to isolate relevant tweets?

13 Data Alignment Use a preset list of team hashtags to identify relevant tweets Discard tweets containing hashtags corresponding to multiple teams Label tweets with the team their hashtags refer to Pittsburgh Steelers hashtags #steelers #pittsburghsteelers #stillers #gosteelers #gosteelersgo #letsgosteelers #gostillers #gostillersgo #letsgostillers #stillernation #stillersnation #steelernation #steelersnation

14 Fan Participation (Annotated)

15 How many Tweets? Season Weekly Tweets (12 hours after previous game to 1 hour before upcoming game) Pregame Tweets (24 hours to 1 hour before upcoming game) Postgame Tweets (4 hours to 28 hours after previous game) ,385 53, , , , , , ,879 1,014,473

16 Data Alignment Given the set a tweet is contained in, label it with the appropriate week of the season For pregame and weekly tweets, this is the week the upcoming game will be played For postgame tweets, this is the week the previous game was played The team and week labels of a tweet determine the unique game the tweet corresponds to

17 A Simple Task Sanity check: Can we look at a postgame tweet and determine if the team it corresponds to won or lost?

18 Postgame Tweet Analysis Use a bag-of-words model to extract features for each tweet In-tweet word frequency features TF-IDF features: In-tweet word frequency Word frequency over all postgame tweets from the week Classify tweets as wins for Home or Away team For k in [1,16]: Train classifier on all postgame tweets from 2010 to week k of the 2012 season

19 Postgame Tweet Analysis Average accuracy of 67% over 2012 season Very simple features and parameter settings Room for improvement Highly weighted features (Top or bottom 30) for a Home team win: home: win home: victory away: loss home: won home: WIN away: lost home: Great away: lose away: refs

20 Forecasting

21 2012 Season Live Predictions During the 2012 season, we tweeted predictions before games We predicted 34/58 or 65.4% of winner WTS results correctly! We did not encounter our tweets in the twitter garden-hose stream

22 Training and Testing Predict outcomes of upcoming games using Logistic Regression: Seasons 2010, 2011 and Weeks [1, k-3] of 2012 Train on all games, with the exception of the last week of a season. Weeks [k-2, k-1] of 2012 Tune L1/L2 regularization parameters. Week k of 2012 Predict and Test Apply procedure over weeks 3-16 of 2012 (or current season)

23 Predicting the NFL without using Twitter Use historical game data to create simple feature sets Point spreads, over unders, scoring, etc. Combine pairs of simple feature to get a preliminary list of game based feature sets Highest average accuracy of 56% Serve as a baseline for Twitter derived feature sets

24 Simple features derived from tweets Word level features (Twitter unigrams) Use only words that appear in at least.1% of all tweets Use log (word frequency + 1) over all weekly tweets corresponding to the game Creates a high dimensional feature space (~20k words) This is high due to the small set of games How to combine these features with features generated from historical game data?

25 Dimensionality Reduction Canonical Correlation Analysis Dimensionality Reduction Combining of multiple data streams Results of applying CCA to Twitter unigrams and game statistics features: Feature Set Winner WTS Accuracy Twitter Unigrams component CCA component CCA component CCA component CCA 48.1

26 The Rate Feature Measure the difference in volume of a team s tweets across consecutive weeks Easy to compute Considers tweets collectively rather than individually Doesn t use point spread or any game statistics Highest average accuracy using a rate feature set is 56%

27 Results

28 Conclusion Our results suggest that a social media driven approach can be effective in predicting sporting events. Download our dataset of NFL game outcomes and tweet IDs!