Dive Into Python. Python from novice to pro. Mark Pilgrim

Transcription

1 Dive Into Python Python from novice to pro Mark Pilgrim

2 Dive Into Python: Python from novice to pro by Mark Pilgrim Published 20 May 2004 Copyright 2000, 2001, 2002, 2003, 2004 Mark Pilgrim This book lives at If you're reading it somewhere else, you may not have the latest version. Permission is granted to copy, distribute, and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in Appendix G, GNU Free Documentation License. The example programs in this book are free software; you can redistribute and/or modify them under the terms of the Python license as published by the Python Software Foundation. A copy of the license is included in Appendix H, Python license.

3 Table of Contents 1. Chapter Which Python is right for you?... 1 Python on Windows... 1 Python on Mac OS X... 3 Python on Mac OS Python on RedHat Linux... 5 Python on Debian GNU/Linux... 6 Python Installation from Source... 7 The Interactive Shell... 8 Summary Chapter Diving in Declaring Functions How Python's Datatypes Compare to Other Programming Languages Documenting Functions Everything Is an Object The Import Search Path What's an Object? Indenting Code Testing Modules Chapter Introducing Dictionaries Defining Dictionaries Modifying Dictionaries Deleting Items From Dictionaries Introducing Lists Defining Lists Adding Elements to Lists Searching Lists Deleting List Elements Using List Operators Introducing Tuples Declaring variables Referencing Variables Assigning Multiple Values at Once Formatting Strings Mapping Lists Joining Lists and Splitting Strings Historical Note on String Methods Summary Chapter Diving In Using Optional and Named Arguments Using type, str, dir, and Other Built-In Functions The type Function The str Function Built-In Functions Getting Object References With getattr getattr with Modules getattr As a Dispatcher Filtering Lists iii

4 Dive Into Python The Peculiar Nature of and and or Using the and-or Trick Using lambda Functions Real-World lambda Functions Putting It All Together Summary Chapter Diving In Importing Modules Using from module import Defining Classes Initializing and Coding Classes Knowing When to Use self and init Instantiating Classes Garbage Collection Exploring UserDict: A Wrapper Class Special Class Methods Getting and Setting Items Advanced Special Class Methods Introducing Class Attributes Private Functions Summary Chapter Handling Exceptions Using Exceptions For Other Purposes Working with File Objects Reading Files Closing Files Handling I/O Errors Writing to Files Iterating with for Loops Using sys.modules Working with Directories Putting It All Together Summary Chapter Diving In Case Study: Street Addresses Case Study: Roman Numerals Checking for Thousands Checking for Hundreds Using the {n,m} Syntax Checking for Tens and Ones Verbose Regular Expressions Case study: Parsing Phone Numbers Summary Chapter Diving in Introducing sgmllib.py Extracting data from HTML documents Introducing BaseHTMLProcessor.py locals and globals Dictionary-based string formatting Quoting attribute values Introducing dialect.py iv

5 Dive Into Python Putting it all together Summary Chapter Diving in Packages Parsing XML Unicode Searching for elements Accessing element attributes Segue Chapter Abstracting input sources Standard input, output, and error Caching node lookups Finding direct children of a node Creating separate handlers by node type Handling command-line arguments Putting it all together Summary Chapter Diving in How not to fetch data over HTTP Features of HTTP User-Agent Redirects Last-Modified/If-Modified-Since ETag/If-None-Match Compression Debugging HTTP web services Setting the User-Agent Handling Last-Modified and ETag Handling redirects Handling compressed data Putting it all together Summary Chapter Diving In Installing the SOAP Libraries Installing PyXML Installing fpconst Installing SOAPpy First Steps with SOAP Debugging SOAP Web Services Introducing WSDL Introspecting SOAP Web Services with WSDL Searching Google Troubleshooting SOAP Web Services Summary Chapter Introduction to Roman numerals Diving in Introducing romantest.py Testing for success Testing for failure v

6 Dive Into Python Testing for sanity Chapter roman.py, stage roman.py, stage roman.py, stage roman.py, stage roman.py, stage Chapter Handling bugs Handling changing requirements Refactoring Postscript Summary Chapter Diving in Finding the path Filtering lists revisited Mapping lists revisited Data-centric programming Dynamically importing modules Putting it all together Summary Chapter Diving in plural.py, stage plural.py, stage plural.py, stage plural.py, stage plural.py, stage plural.py, stage Summary Chapter Diving in Using the timeit Module Optimizing Regular Expressions Optimizing Dictionary Lookups Optimizing List Operations Optimizing String Manipulation Summary A. Further reading B. A 5-minute review C. Tips and tricks D. List of examples E. Revision history F. About the book G. GNU Free Documentation License Preamble Applicability and definitions Verbatim copying Copying in quantity Modifications Combining documents Collections of documents Aggregation with independent works vi

7 Dive Into Python 8. Translation Termination Future revisions of this license How to use this License for your documents H. Python license A. History of the software B. Terms and conditions for accessing or otherwise using Python B.. PSF license agreement B.. BeOpen Python open source license agreement version B.. CNRI open source GPL-compatible license agreement B.. CWI permissions statement and disclaimer vii

8 List of Examples 1.1. Two versions of Python Installing on RedHat Linux Installing on Debian GNU/Linux Installing from source First Steps in the Interactive Shell odbchelper.py Defining the buildconnectionstring Function's doc string Accessing the buildconnectionstring Function's doc string Import Search Path Indenting the buildconnectionstring Function if Statements Defining a Dictionary Modifying a Dictionary Dictionary Keys Are Case-Sensitive Mixing Datatypes in a Dictionary Deleting Items from a Dictionary Defining a List Negative List Indices Slicing a List Slicing Shorthand Adding Elements to a List The Difference between extend and append Searching a List Removing Elements from a List List Operators Defining a tuple Tuples Have No Methods Defining the myparams Variable Referencing an Unbound Variable Assigning multiple values at once Assigning Consecutive Values Introducing String Formatting String Formatting vs. Concatenating Formatting Numbers Introducing List Comprehensions The keys, values, and items Functions List Comprehensions in buildconnectionstring, Step by Step Output of odbchelper.py Splitting a String apihelper.py Sample Usage of apihelper.py Advanced Usage of apihelper.py Valid Calls of info Introducing type Introducing str Introducing dir Introducing callable Built-in Attributes and Functions Introducing getattr The getattr Function in apihelper.py Creating a Dispatcher with getattr viii

9 Dive Into Python getattr Default Values Introducing List Filtering Introducing and Introducing or Introducing the and-or Trick When the and-or Trick Fails Using the and-or Trick Safely Introducing lambda Functions split With No Arguments Getting a doc string Dynamically Why Use str on a doc string? Introducing ljust Printing a List fileinfo.py import module vs. from module import The Simplest Python Class Defining the FileInfo Class Initializing the FileInfo Class Coding the FileInfo Class Creating a FileInfo Instance Trying to Implement a Memory Leak Defining the UserDict Class UserDict Normal Methods Inheriting Directly from Built-In Datatype dict The getitem Special Method The setitem Special Method Overriding setitem in MP3FileInfo Setting an MP3FileInfo's name More Special Methods in UserDict Introducing Class Attributes Modifying Class Attributes Trying to Call a Private Method Opening a Non-Existent File Supporting Platform-Specific Functionality Opening a File Reading a File Closing a File File Objects in MP3FileInfo Writing to Files Introducing the for Loop Simple Counters Iterating Through a Dictionary for Loop in MP3FileInfo Introducing sys.modules Using sys.modules The module Class Attribute sys.modules in fileinfo.py Constructing Pathnames Splitting Pathnames Listing Directories Listing Directories in fileinfo.py Listing Directories with glob listdirectory Matching at the End of a String ix

10 Dive Into Python 7.2. Matching Whole Words Checking for Thousands Checking for Hundreds The Old Way: Every Character Optional The New Way: From n o m Checking for Tens Validating Roman Numerals with {n,m} Regular Expressions with Inline Comments Finding Numbers Finding the Extension Handling Different Separators Handling Numbers Without Separators Handling Leading Characters Phone Number, Wherever I May Find Ye Parsing Phone Numbers (Final Version) BaseHTMLProcessor.py dialect.py Output of dialect.py Sample test of sgmllib.py Introducing urllib Introducing urllister.py Using urllister.py Introducing BaseHTMLProcessor BaseHTMLProcessor output Introducing locals Introducing globals locals is read-only, globals is not Introducing dictionary-based string formatting Dictionary-based string formatting in BaseHTMLProcessor.py More dictionary-based string formatting Quoting attribute values Handling specific tags SGMLParser Overriding the handle_data method The translate function, part The translate function, part 2: curiouser and curiouser The translate function, part kgp.py toolbox.py Sample output of kgp.py Simpler output from kgp.py Loading an XML document (a sneak peek) File layout of a package Packages are modules, too Loading an XML document (for real this time) Getting child nodes toxml works on any node Child nodes can be text Drilling down all the way to text Introducing unicode Storing non-ascii characters sitecustomize.py Effects of setting the default encoding Specifying encoding in.py files x

11 Dive Into Python russiansample.xml Parsing russiansample.xml binary.xml Introducing getelementsbytagname Every element is searchable Searching is actually recursive Accessing element attributes Accessing individual attributes Parsing XML from a file Parsing XML from a URL Parsing XML from a string (the easy but inflexible way) Introducing StringIO Parsing XML from a string (the file-like object way) openanything Using openanything in kgp.py Introducing stdout and stderr Redirecting output Redirecting error information Printing to stderr Chaining commands Reading from standard input in kgp.py loadgrammar Using the ref element cache Finding direct child elements Class names of parsed XML objects parse, a generic XML node dispatcher Functions called by the parse dispatcher Introducing sys.argv The contents of sys.argv Introducing getopt Handling command-line arguments in kgp.py openanything.py Downloading a feed the quick-and-dirty way Debugging HTTP Introducing urllib Adding headers with the Request Testing Last-Modified Defining URL handlers Using custom URL handlers Supporting ETag/If-None-Match Accessing web services without a redirect handler Defining the redirect handler Using the redirect handler to detect permanent redirects Using the redirect handler to detect temporary redirects Telling the server you would like compressed data Decompressing the data Decompressing the data directly from the server The openanything function The fetch function Using openanything.py search.py Sample Usage of search.py Verifying PyXML Installation Verifying fpconst Installation xi

12 Dive Into Python Verifying SOAPpy Installation Getting the Current Temperature Debugging SOAP Web Services Discovering The Available Methods Discovering A Method's Arguments Discovering A Method's Return Values Calling A Web Service Through A WSDL Proxy Introspecting Google Web Services Searching Google Accessing Secondary Information From Google Calling a Method With an Incorrectly Configured Proxy Calling a Method With the Wrong Arguments Calling a Method and Expecting the Wrong Number of Return Values Calling a Method With An Application-Specific Error romantest.py testtoromanknownvalues Testing bad input to toroman Testing bad input to fromroman Testing toroman against fromroman Testing for case roman1.py Output of romantest1.py against roman1.py roman2.py How toroman works Output of romantest2.py against roman2.py roman3.py Watching toroman handle bad input Output of romantest3.py against roman3.py roman4.py How fromroman works Output of romantest4.py against roman4.py roman5.py Output of romantest5.py against roman5.py The bug Testing for the bug (romantest61.py) Output of romantest61.py against roman61.py Fixing the bug (roman62.py) Output of romantest62.py against roman62.py Modifying test cases for new requirements (romantest71.py) Output of romantest71.py against roman71.py Coding the new requirements (roman72.py) Output of romantest72.py against roman72.py Compiling regular expressions Compiled regular expressions in roman81.py Output of romantest81.py against roman81.py roman82.py Output of romantest82.py against roman82.py roman83.py Output of romantest83.py against roman83.py roman9.py Output of romantest9.py against roman9.py regression.py Sample output of regression.py fullpath.py xii

13 Dive Into Python Further explanation of os.path.abspath Sample output from fullpath.py Running scripts in the current directory Introducing filter filter in regression.py Filtering using list comprehensions instead Introducing map map with lists of mixed datatypes map in regression.py Importing multiple modules at once Importing modules dynamically Importing a list of modules dynamically The regressiontest function Step 1: Get all the files Step 2: Filter to find the files you care about Step 3: Map filenames to module names Step 4: Mapping module names to modules Step 5: Loading the modules into a test suite Step 6: Telling unittest to use your test suite plural1.py Introducing re.sub Back to plural1.py More on negation regular expressions More on re.sub plural2.py Unrolling the plural function plural3.py plural4.py plural4.py continued Unrolling the rules definition plural4.py, finishing up Another look at buildmatchandapplyfunctions Expanding tuples when calling functions rules.en plural5.py plural6.py Introducing generators Using generators instead of recursion Generators in for loops Generators that generate dynamic functions soundex/stage1/soundex1a.py Introducing timeit Best Result So Far: soundex/stage1/soundex1e.py Best Result So Far: soundex/stage2/soundex2c.py Best Result So Far: soundex/stage2/soundex2c.py xiii

14 Chapter 1. Installing Python Welcome to Python. Let's dive in. In this chapter, you'll install the version of Python that's right for you. Which Python is right for you? The first thing you need to do with Python is install it. Or do you? If you're using an account on a hosted server, your ISP may have already installed Python. Most popular Linux distributions come with Python in the default installation. Mac OS X 10.2 and later includes a command-line version of Python, although you'll probably want to install a version that includes a more Mac-like graphical interface. Windows does not come with any version of Python, but don't despair! There are several ways to pointand-click your way to Python on Windows. As you can see already, Python runs on a great many operating systems. The full list includes Windows, Mac OS, Mac OS X, and all varieties of free UNIX-compatible systems like Linux. There are also versions that run on Sun Solaris, AS/400, Amiga, OS/2, BeOS, and a plethora of other platforms you've probably never even heard of. What's more, Python programs written on one platform can, with a little care, run on any supported platform. For instance, I regularly develop Python programs on Windows and later deploy them on Linux. So back to the question that started this section, Which Python is right for you? The answer is whichever one runs on the computer you already have. Python on Windows On Windows, you have a couple choices for installing Python. ActiveState makes a Windows installer for Python called ActivePython, which includes a complete version of Python, an IDE with a Python-aware code editor, plus some Windows extensions for Python that allow complete access to Windows-specific services, APIs, and the Windows Registry. ActivePython is freely downloadable, although it is not open source. It is the IDE I used to learn Python, and I recommend you try it unless you have a specific reason not to. One such reason might be that ActiveState is generally several months behind in updating their ActivePython installer when new version of Python are released. If you absolutely need the latest version of Python and ActivePython is still a version behind as you read this, you'll want to use the second option for installing Python on Windows. The second option is the official Python installer, distributed by the people who develop Python itself. It is freely downloadable and open source, and it is always current with the latest version of Python. Procedure 1.1. Option 1: Installing ActivePython Here is the procedure for installing ActivePython: 1. Download ActivePython from 1

15 Chapter 1 2. If you are using Windows 95, Windows 98, or Windows ME, you will also need to download and install Windows Installer 2.0 [ W9XMe/EN-US/InstMsiA.exe] before installing ActivePython. 3. Double-click the installer, ActivePython win32-ix86.msi. 4. Step through the installer program. 5. If space is tight, you can do a custom installation and deselect the documentation, but I don't recommend this unless you absolutely can't spare the 14MB. 6. After the installation is complete, close the installer and choose Start->Programs->ActiveState ActivePython 2.2->PythonWin IDE. You'll see something like the following: PythonWin (#37, Nov , 10:24:37) [MSC 32 bit (Intel)] on win32. Portions Copyright Mark Hammond (mhammond@skippinet.com.au) - see 'Help/About PythonWin' for further copyright information. >>> Procedure 1.2. Option 2: Installing Python from Python.org [ ] 1. Download the latest Python Windows installer by going to and selecting the highest version number listed, then downloading the.exe installer. 2. Double-click the installer, Python-2.xxx.yyy.exe. The name will depend on the version of Python available when you read this. 3. Step through the installer program. 4. If disk space is tight, you can deselect the HTMLHelp file, the utility scripts (Tools/), and/or the test suite (Lib/test/). 5. If you do not have administrative rights on your machine, you can select Advanced Options, then choose Non-Admin Install. This just affects where Registry entries and Start menu shortcuts are created. 6. After the installation is complete, close the installer and select Start->Programs->Python 2.3->IDLE (Python GUI). You'll see something like the following: Python (#49, Oct , 20:02:00) [MSC v bit (Intel)] on win32 Type "copyright", "credits" or "license()" for more information. **************************************************************** Personal firewall software may warn about the connection IDLE makes to its subprocess using this computer's internal loopback interface. This connection is not visible on any external interface and no data is sent to or received from the Internet. **************************************************************** IDLE 1.0 >>> 2

16 Chapter 1 Python on Mac OS X On Mac OS X, you have two choices for installing Python: install it, or don't install it. You probably want to install it. Mac OS X 10.2 and later comes with a command-line version of Python preinstalled. If you are comfortable with the command line, you can use this version for the first third of the book. However, the preinstalled version does not come with an XML parser, so when you get to the XML chapter, you'll need to install the full version. Rather than using the preinstalled version, you'll probably want to install the latest version, which also comes with a graphical interactive shell. Procedure 1.3. Running the Preinstalled Version of Python on Mac OS X To use the preinstalled version of Python, follow these steps: 1. Open the /Applications folder. 2. Open the Utilities folder. 3. Double-click Terminal to open a terminal window and get to a command line. 4. Type python at the command prompt. Try it out: Welcome to Darwin! [localhost:~] you% python Python 2.2 (#1, 07/14/02, 23:25:09) [GCC Apple cpp-precomp 6.14] on darwin Type "help", "copyright", "credits", or "license" for more information. >>> [press Ctrl+D to get back to the command prompt] [localhost:~] you% Procedure 1.4. Installing the Latest Version of Python on Mac OS X Follow these steps to download and install the latest version of Python: 1. Download the MacPython-OSX disk image from 2. If your browser has not already done so, double-click MacPython-OSX dmg to mount the disk image on your desktop. 3. Double-click the installer, MacPython-OSX.pkg. 4. The installer will prompt you for your administrative username and password. 5. Step through the installer program. 6. After installation is complete, close the installer and open the /Applications folder. 7. Open the MacPython-2.3 folder 3

17 Chapter 1 8. Double-click PythonIDE to launch Python. The MacPython IDE should display a splash screen, then take you to the interactive shell. If the interactive shell does not appear, select Window->Python Interactive (Cmd-0). The opening window will look something like this: Python 2.3 (#2, Jul , 11:45:28) [GCC (prerelease)] Type "copyright", "credits" or "license" for more information. MacPython IDE >>> Note that once you install the latest version, the pre-installed version is still present. If you are running scripts from the command line, you need to be aware which version of Python you are using. Example 1.1. Two versions of Python [localhost:~] you% python Python 2.2 (#1, 07/14/02, 23:25:09) [GCC Apple cpp-precomp 6.14] on darwin Type "help", "copyright", "credits", or "license" for more information. >>> [press Ctrl+D to get back to the command prompt] [localhost:~] you% /usr/local/bin/python Python 2.3 (#2, Jul , 11:45:28) [GCC (prerelease)] on darwin Type "help", "copyright", "credits", or "license" for more information. >>> [press Ctrl+D to get back to the command prompt] [localhost:~] you% Python on Mac OS 9 Mac OS 9 does not come with any version of Python, but installation is very simple, and there is only one choice. Follow these steps to install Python on Mac OS 9: 1. Download the MacPython23full.bin file from 2. If your browser does not decompress the file automatically, double-click MacPython23full.bin to decompress the file with Stuffit Expander. 3. Double-click the installer, MacPython23full. 4. Step through the installer program. 5. AFter installation is complete, close the installer and open the /Applications folder. 6. Open the MacPython-OS9 2.3 folder. 7. Double-click Python IDE to launch Python. 4

18 Chapter 1 The MacPython IDE should display a splash screen, and then take you to the interactive shell. If the interactive shell does not appear, select Window->Python Interactive (Cmd-0). You'll see a screen like this: Python 2.3 (#2, Jul , 11:45:28) [GCC (prerelease)] Type "copyright", "credits" or "license" for more information. MacPython IDE >>> Python on RedHat Linux Installing under UNIX-compatible operating systems such as Linux is easy if you're willing to install a binary package. Pre-built binary packages are available for most popular Linux distributions. Or you can always compile from source. Download the latest Python RPM by going to and selecting the highest version number listed, then selecting the rpms/ directory within that. Then download the RPM with the highest version number. You can install it with the rpm command, as shown here: 5

19 Chapter 1 Example 1.2. Installing on RedHat Linux 9 localhost:~$ su - Password: [enter your root password] [root@localhost root]# wget Resolving python.org... done. Connecting to python.org[ ]:80... connected. HTTP request sent, awaiting response OK Length: 7,495,111 [application/octet-stream]... [root@localhost root]# rpm -Uvh python pydotorg.i386.rpm Preparing... ########################################### [100%] 1:python2.3 ########################################### [100%] [root@localhost root]# python Python (#1, Feb , 19:13:11) [GCC (Red Hat Linux )] on linux2 Type "help", "copyright", "credits", or "license" for more information. >>> [press Ctrl+D to exit] [root@localhost root]# python2.3 Python 2.3 (#1, Sep , 10:53:56) [GCC (Red Hat Linux )] on linux2 Type "help", "copyright", "credits", or "license" for more information. >>> [press Ctrl+D to exit] [root@localhost root]# which python2.3 /usr/bin/python2.3 Whoops! Just typing python gives you the older version of Python -- the one that was installed by default. That's not the one you want. At the time of this writing, the newest version is called python2.3. You'll probably want to change the path on the first line of the sample scripts to point to the newer version. This is the complete path of the newer version of Python that you just installed. Use this on the #! line (the first line of each script) to ensure that scripts are running under the latest version of Python, and be sure to type python2.3 to get into the interactive shell. Python on Debian GNU/Linux If you are lucky enough to be running Debian GNU/Linux, you install Python through the apt command. 6

20 Chapter 1 Example 1.3. Installing on Debian GNU/Linux localhost:~$ su - Password: [enter your root password] localhost:~# apt-get install python Reading Package Lists... Done Building Dependency Tree... Done The following extra packages will be installed: python2.3 Suggested packages: python-tk python2.3-doc The following NEW packages will be installed: python python2.3 0 upgraded, 2 newly installed, 0 to remove and 3 not upgraded. Need to get 0B/2880kB of archives. After unpacking 9351kB of additional disk space will be used. Do you want to continue? [Y/n] Y Selecting previously deselected package python2.3. (Reading database files and directories currently installed.) Unpacking python2.3 (from.../python2.3_ _i386.deb)... Selecting previously deselected package python. Unpacking python (from.../python_ _all.deb)... Setting up python ( )... Setting up python2.3 ( )... Compiling python modules in /usr/lib/python Compiling optimized python modules in /usr/lib/python localhost:~# exit logout localhost:~$ python Python (#2, Sep , 11:39:14) [GCC (Debian prerelease)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> [press Ctrl+D to exit] Python Installation from Source If you prefer to build from source, you can download the Python source code from ftp/python/. Select the highest version number listed, download the.tgz file), and then do the usual configure, make, make install dance. 7