Wiley. Automated Data Collection with R. Text Mining. A Practical Guide to Web Scraping and

Transcription

1 Automated Data Collection with R A Practical Guide to Web Scraping and Text Mining Simon Munzert Department of Politics and Public Administration, Germany Christian Rubba University ofkonstanz, Department of Political Science, University of Zurich and National Center of Competence in Research, Switzerland Peter MeiBner Department ofpolitics and Public Administration, University ofkonstanz, Germany Dominic Nyhuis Department of Political Science, University ofmannheim, Germany Wiley

2 Contents Preface xv 1 Introduction Case study: World Heritage Sites in Danger Some remarks on web data quality Technologies for disseminating, extracting, and storing web data Technologies for disseminating content on the Web Technologies for information extraction from web documents Technologies for data storage Structure of the book 13 Part One A Primer on Web and Data Technologies 15 2 HTML Browser presentation and source code Syntax rules Tags, elements, and attributes Tree structure Comments Reserved and special characters Document type definition Spaces and line breaks Tags and attributes The anchor tag <a> The metadata tag <meta> The external reference tag <link> Emphasizing tags <b>, <i>, <strong> The paragraphs tag <p> Heading tags <hl>, <h2>, <h3>, Listing content with <ul>, <ol>, and <dl> The organizational tags <div> and <span> 27

3 viii CONTENTS The <form> tag and its companions The foreign script tag <script> Table tags <table>, <tr>, <td>, and <th> Parsing What is parsing? Discarding nodes Extracting information in the building process 37 Summary 38 Further reading 38 Problems 39 3 XMLandJSON A short example XML document XML syntax rules Elements and attributes XML structure Naming and special characters Comments and character data XML syntax summary When is an XML document well formed or valid? XML extensions and technologies Namespaces Extensions of XML Example: Really Simple Syndication Example: scalable vector graphics XML and R in practice Parsing XML Basic operations on XML documents From XML to data frames or lists Event-driven parsing A short example JSON document JSON syntax rules JSON and R in practice 71 Summary 76 Further reading 76 Problems 76 4 XPath XPath a query language for web documents Identifying node sets with XPath Basic structure of an XPath query Node relations XPath predicates Extracting node elements Extending the fun argument XML namespaces Little XPath helper tools 97

4 CONTENTS ix Summary 98 Further reading 99 Problems 99 5 HTTP HTTP fundamentals A short conversation with a web server URL syntax HTTP messages Request methods Status codes Header fields Advanced features of HTTP Identification Authentication Proxies Protocols beyond HTTP HTTP Secure FTP HTTP in action The libcurl library Basic request methods A low-level function of RCurl Maintaining connections across multiple requests Options Debugging Error handling RCurl or httr what to use? 144 Summary 144 Further reading 144 Problems AJAX JavaScript How JavaScript is used DOM manipulation XHR Loading external HTML/XML documents Loading JSON Exploring AJAX with Web Developer Tools Getting started with Chrome's Web Developer Tools The Elements panel The Network panel 160 Summary 161 Further reading 162 Problems 162

5 CONTENTS 7 SQL and relational databases Overview and terminology Relational Databases Storing data in tables Normalization Advanced features of relational databases and DBMS SQL: a language to communicate with Databases General remarks on SQL, syntax, and our running example Data control language DCL Data definition language DDL Data manipulation language DML Clauses Transaction control language TCL Databases in action R packages to manage databases Speaking R-SQL via DBI-based packages Speaking R-SQL via RODBC 191 Summary 192 Further reading 193 Problems Regular expressions and essential string functions Regular expressions Exact character matching Generalizing regular expressions The introductory example reconsidered String processing The stringr package A couple more handy functions A word on character encodings 214 Summary 216 Further reading 217 Problems 217 Part Two A Practical Toolbox for Web Scraping and Text Mining Scraping the Web Retrieval scenarios Downloading ready-made files Downloading multiple files from an FTP index Manipulating URLs to access multiple pages Convenient functions to gather links, lists, and tables from HTML documents Dealing with HTML forms HTTP authentication Connections via HTTPS Using cookies 247

6 CONTENTS Scraping data from AJAX-enriched webpages with Selenium/Rwebdriver Retrieving data from APIs Authentication with OAuth Extraction strategies Regular expressions XPath Application Programming Interfaces Web scraping: Good practice Is web scraping legal? What is robots.txt? Be friendly! Valuable sources of inspiration 290 Summary 291 Further reading 292 Problems Statistical text processing The running example: Classifying press releases of the British government Processing textual data Large-scale text operations The tm package Building a term-document matrix Data cleansing Sparsity and n-grams Supervised learning techniques Support vector machines Random Forest Maximum entropy The RTextTools package Application: Government press releases Unsupervised learning techniques Latent Dirichlet Allocation and correlated topic models Application: Government press releases 314 Summary 320 Further reading Managing data projects Interacting with the file system Processing multiple documents/links Using /or-loops Using while-loops and control structures Using the plyr package Organizing scraping procedures Implementation of progress feedback: Messages and progress bars Error and exception handling 333

7 xii CONTENTS 11.4 Executing R scripts on a regular basis Scheduling tasks on Mac OS and Linux Scheduling tasks on Windows platforms 337 Part Three A Bag of Case Studies Collaboration networks in the US Senate Information on the bills Information on the senators Analyzing the network structure Descriptive statistics Network analysis Conclusion Parsing information from semistructured documents Downloading data from the FTP server Parsing semistructured text data Visualizing station and temperature data Predicting the 2014 Academy Awards using Twitter Twitter APIs: Overview The REST API The Streaming APIs Collecting and preparing the data Twitter-based forecast of the 2014 Academy Awards Visualizing the data Mining tweets for predictions Conclusion Mapping the geographic distribution of names Developing a data collection strategy Website inspection Data retrieval and information extraction Mapping names Automating the process 389 Summary Gathering data on mobile phones Page exploration Searching mobile phones of a specific brand Extracting product information Scraping procedure Retrieving data on several producers Data cleansing Graphical analysis 406

8 CONTENTS xiii 16.4 Datastorage General considerations Table definitions for storage Table definitions for future storage View definitions for convenient data access Functions for storing data Data storage and inspection Analyzing sentiments of product reviews Introduction Collecting the data Downloading the files Information extraction Database storage Analyzing the data Data preparation Dictionary-based sentiment analysis Mining the content of reviews Conclusion 434 References 435 General index 442 Package index 448 Function index 449