Raspberry Pi Exercises

Transcription

1 Raspberry Pi Exercises Working with the Raspberry Pi, and in particular working with Google Coder, Raspbian, the version of Debian Linux that has been adapted for use with the Raspberry Pi, and Python, are an important part of LIS 2000 Understanding Information and the introduction that it provides to students in the MLIS degree program. The following exercises have been developed in order to familiarize each of you with information technology at a basic level and in terms that we hope will foster your creativity, if not now then at some time in the not-toodistant future. Exercise 1 Exercise 2 pg. 1

2 Exercise 3, Part 1 pg. 2

3 pg. 3

4 Exercise 3, Part 2 pg. 4

5 Web Scraping with Python A key element in the process of beginning to master information technology is learning about the programming process and the creation of programming instructions, or code. This two-part exercise is not intended as an introduction to computational thinking, programming principles, or programming languages, not in the sense that any part of it could be regarded as even remotely sufficient in dealing with those subjects. But it is intended to be a pathfinder of sorts, in that it offers the first step or two toward a greater knowledge of computing and programming. Why Python? There are several reasons. First of all, Python is widely regarded as a programming language that is at once relatively easy to learn yet powerful in its potential efforts. Second, Python code is easy to read (and understand) and enforces good programming style, without being overly strict about syntax. The third reason? For what it is worth, Brewster Kahle, the founder and director of pg. 5

6 the Internet Archive, observed earlier this year in a talk he delivered at the School of Information Sciences that he regarded the ability to program in Python as essential qualification of the archivists and librarians employed by the Internet Archive, citing examples of the roles in which Python coding is a key capability. (Access to the video recording of Kahle s talk will require a CSSD userid and password.) pg. 6

7 Web Scraping Assignment pg. 7

8 Prerequisites Before you can begin, your computer needs the following tools installed and working to participate: A command-line interface to interact with your computer, either through a keyboard/display combination connected directly to the Raspberry Pi or via an SSH connection over the local area network; A text editor to work with plain text files in this instance, nano will be appropriate; Version 2.7 of the Python programming language; and The pip package manager for Python, which is a command-line tool that downloads and installs open-source libraries for Python from Webbased repositories. pg. 8

9 pip is not installed by default. To put it in place, use the following command: sudo apt-get install python-pip pg. 9

10 Python Installing Dependencies Coding for the Web scraper will use Python s BeautifulSoup toolkit to parse the site s HTML and extract the data. It will also use the Requests library to open the URL, download the HTML and pass the data to BeautifulSoup. Since neither Beautiful Soup or Requests is included in Python s standard library, each must be installed, using pip. To install them, issue the following commands: $ sudo pip install BeautifulSoup $ sudo pip install Requests Selecting a Web Page to Scrape Selecting a Web page is an important and deceptively simple aspect of the assignment. On the face of it, all that is required is a Web page that presents data in the form of an HTML table, as identifed by the <table> tag. And, of course, Web pages like this abound. The less-than-easy aspect of making the pg. 10

11 selection is finding a Web page that has been coded in a way that lends itself to the ready identification of the table and its constituent parts. You will discover that the code underlying many pages is a complicated tangle of HTML code, scripting, and inline formatting. (One of the reasons why CSS Cascading Style Sheets was developed was to establish a mechanism that would enable the separation of formatting information from the structural markup of the HTML document. While CSS has been in widespread use for more than 15 years, you will note in your investigation of various Web pages that many developers continue to employ inline formatting techniques, a choice that often renders the code they create difficult to read.) pg. 11

12 Extracting an HTML table The goal? To write a Python script that downloads the HTML table, extracts the data contained by the table, and then writes it to a file in the *.csv, or commadelimited format. pg. 12

13 You should begin by opening the terminal and creating a new directory within your home directory. The command to create a new directory in Linux is: $ mkdir [directory_name] To ensure the integrity of your work, the directory should be given a name that signifying both its contents and creator. So, for example, it might called code_smith, and absolute path from the root of the Raspbian file system to this directory would be, assuming that it was created by the default user, /home/pi/code_smith. Navigate to code_smith directory. The command is: $ cd code_smith Then, open the text editor, nano, and save an empty file called webscrape.py. Or, you can open the Python environment under Raspbian: pg. 13

14 (It is preferable, and almost certainly simpler, that this work is done on the Raspberry Pi. But it is also possible to undertake the programming on a Mac or PC. Under Microsoft Windows, Notepad is available for text editing. On a Mac, under OS X, you can use TextEdit, but another editor, such as TextWrangler or Atom, would be better, since TextEdit defaults to *.rtf, the Rich Text Format, which must overriden in order to save files in the ASCII (American Standard Coded Information Interchange), or *.txt, format. Since the terms of the assignment require you to run the Web scraper from the Raspberry Pi, working from a personal computer will entail additional steps; specifically, it will be necessary to set up Raspbian to support file transfers via ftp. For information setting up the Raspberry Pi as an ftp server, see How to setup an FTP server with vsftpd on Raspberry Pi.) pg. 14

15 At this point, please turn your attention to the instructions presented via Act 2: Python and Act 3: Web Scraping, which is part of a document entitled First Web Scraper, which was initially developed by Chase Davis, Jackie Kazil, Sisi Wei and Matt Wynn for a short course offered at the University of Missouri in For reference, the code repository is located at github.com/ireapps/firstweb-scraper/, the documentation is at first-web-scraper.rtfd.org/, and a discussion of related issues is conducted at github.com/ireapps/first-webscraper/issues/. Presenting the Results of Web Scraping with Python Code This example, presented via Microsoft Excel, is made up of data scraped from home page of the School of Information Sciences. Since Excel, Apple s Numbers, and OpenOffice all support the export of a spreadsheet as a PDF, using any one of these applications as a means of presenting the dataset scraped from the Web is an acceptable option. pg. 15

16 Exercise 3, Part 3 Installing Webmin Webmin is a web-based system configuration tool for Unix-like systems, although recent versions can also be installed and run on Windows. With Webmin, it is possible to configure operating system internals, such as users, disk quotas, services or configuration files, as well as modify and control open source applications, such as the Apache HTTP Server, PHP or MySQL. Webmin is largely based on Perl, running as its own process and Web server. It defaults to TCP port for communicating, and can be configured to use SSL if OpenSSL is installed with additional required Perl modules. Webmin is built around modules, which have an interface to the configuration files and the Webmin server. It can be expanded by installing additional pg. 16

17 modules, including custom made modules. In addition, there are two other major projects that extend the functionality of Webmin: Usermin, which presents and controls a subset of the features available in Webmin, such as webmail and other user-level tasks, rather than administrator-level tasks; and Virtualmin, which is a web hosting control panel. pg. 17

18 Webmin Download Extract pg. 18

19 Completion of Installation pg. 19

20 Webmin Setup pg. 20

21 Webmin Login pg. 21

22 Webmin Opening Menu pg. 22

23 pg. 23

24 pg. 24

25 pg. 25

26 MySQL Database Server Menu pg. 26

27 MySQL Database Server Configuration pg. 27

28 Creating a MySQL Database pg. 28

29 Creating MySQL Users and User Permissions pg. 29

30 Creating Host Permissions pg. 30

31 Exercise 3, Part 4 pg. 31