ISA5575 Web Techniques and Applications The Web Server Instructor: Yi-Shin Chen Office: EECS 3201 Email: yishin@cs.nthu.edu.tw Office Hour: Tu. 1-2PM, Th. 3-4pm Client/Server Architecture Model Terminals PCs MACs XWindows Multiplatform Browsers (Clients) Addressing Scheme (URL) + Protocol (HTTP, FTP, etc.) + Format Negotiation (MIME) Mechanisms HTTP server FTP server Gopher server NNTP server Server for each protocol 2007/3/13 Web Techniques and Applications 2 1
What Does the WWW Server Do? Enables browser requests Provides Support for retrieving hypertext documents Manages access to the Web site Supports clickable image maps (SSIM) Provides several mechanisms for executing server-side scripts Common Gateway Interface (CGI) Application Programmers Interface (API) Produces log files and usage statistics 2007/3/13 Web Techniques and Applications 3 What to Look for in a Web Server Main features Platform they run on Complete support for HTTP 1.1 Multithreading, load balancing Server side includes image mapping, analysis tools Security IP address restriction Domain name restriction Secure transactions: SSL or S_HTTP Proxy server 2007/3/13 Web Techniques and Applications 4 2
How Servers Handle Multiple Requests For each request, a complete copy of the server is made and executed Initially a parent process waits for requests; for each new request a child process is spawned Or, a single server program handles many requests simultaneously (multithreaded) The server must keep track of all requests and switch between them as needed Writing multithreaded programs is easier in a language that supports multiple threads, e.g. Java 2007/3/13 Web Techniques and Applications 5 Some Available Servers Product Apache Sun Java System Web Server WebSphere Application Server IIS 6.0 Apache http://wwws.sun.com/software/products/appsrvr/home_appsrvr.h tml IBM www.ibm.com/software/webservers/appserv/was/ Microsoft Publisher http://httpd.apache.org/ Sun MicroSystems Win32/UNIX Solaris/Win32/Linux Win32/UNIX/Linux Win32 Platform http://www.microsoft.com/windowsserver2003/iis/default.mspx For a comparison of web servers see http://www.serverwatch.com/ 2007/3/13 Web Techniques and Applications 6 3
Internet Server Usage Source from http://www.netcraft.com Apache currently dominates Internet servers Statistics on intranets are difficult to determine 2007/3/13 Web Techniques and Applications 7 Server Features Document Root Server Side Includes Authentication Proxy Servers Caching CGI Scripting Application Program Interface 2007/3/13 Web Techniques and Applications 8 4
Document Root The Web server has access to a tree of files that it can deliver The files are created by content providers Files may contain text, images, sound, video, other programs, etc. The document tree is organized by the web site administrator The root of the document tree is given to the web server when it starts 2007/3/13 Web Techniques and Applications 9 Managing the Document Tree http://domain/cs/index.ht ml The document root is /serverhome/htdocs <A HREF=page.html> /serverhome <A HREF=images/banner.gif> other htdocs <A HREF=../ee/index.html> user ee cs <A HREF=/serverhome/other/usr/public/h ome.html> public home.html index.html pages.html index.html images banner.gif http://domain/cs is mapped to /serverhome/htdocs/cs/index.html 2007/3/13 Web Techniques and Applications 10 5
Directory Listing When the URL path resolves to a directory, not to a file, a web server can be configured to return an error a special default file or a directory listing 2007/3/13 Web Techniques and Applications 11 Server-Side Includes A server-side include is an action taken by the server before it delivers the document to the client Unfortunately, this feature requires the server to examine and parse the document, looking for the special commands Here, the server needs to understand HTML Most servers supporting this feature come with it disabled as the default Typically, parsable HTML files are denoted by a different suffix; e.g., shtml General form: <!--#command tag1="value1" tag2="value2" --> 2007/3/13 Web Techniques and Applications 12 6
Some Server-Side Include Directives Command config include fsize flastmod exec errmsg timefmt sizefmt file echo var file file cmd Tag Explanation Controls what messages are sent back to the client if an error occurs Provides a new format for expressing dates Provides a format for expressing file sizes Gives a pathname relative to current directory Prints the value of an environment variable Values for var tag are: DOCUMENT_NAME The current filename DATE_LOCAL Current date using local time zone DATE_GMT Current date using Greenwich Mean Time LAST_MODIFIED Last date/time file was touched Prints size of the file Prints last modification date of the file Executes a shell command or CGI script 2007/3/13 Web Techniques and Applications 13 Basic User Authentication Basic authentication is supported by all HTTP servers The server administrator creates secure directories accessible via password files maintained by the server Client requests a resource from a secure directory; e.g., GET /secure/test.html HTTP/1.0 Server responds with authentication request to the client; e.g., HTTP/1.0 401 Unauthorized Browser requests user name and password, scrambles them, and retries the request GET http://domain/secure/test.html HTTP/1.0 Authorization: Basic 0<V%L:EB.G-E8W))$J Server decodes name and password and checks it against its password file 2007/3/13 Web Techniques and Applications 14 7
Creating Server-Side Applications Servers offer several mechanisms Application Programming Interface (API) Common Gateway Interface (CGI) J2EE /.NET interfaces Microsoft IIS server-side supports: API applications compiled as DLL Active Server Pages (written in VBScript) ASP.NET (written in C++, VB.NET, Jscript or C#) Client machine Response Server machine Browser Request IIS server page.dll 2007/3/13 Web Techniques and Applications 15 Application Programmer s Interface API extensions, like CGI scripts, produce dynamically generated web pages; API filters can pre- and post-process requests, e.g. for authentication, custom logging, security API extensions or filters on Windows are compiled DLLs; they are loaded into the same address space as the web server Hence they start up much faster, but they can crash the web server Like CGI programs *.dlls must be placed in a directory for which the server has execute permission API extensions/filters are server specific *.dlls are invoked by naming them directly in a URL, e.g. http://domain/script/myprog.dll 2007/3/13 Web Techniques and Applications 16 8
Configuring a Server No matter which operating system or server, you will need to define the location of the server (server root) location of documents to deliver (document root) location of CGI scripts or server-side components to execute You may also wish to define Access restrictions Fancy indexing Server-side includes Other features 2007/3/13 Web Techniques and Applications 17 Port Numbers A typical computer will have thousands of ports System ports: 0-1023, are reserved for certain functions, typically assigned by IANA (Internet Assigned Number Authority) See http://www.iana.org/numbers.html Programs running on these ports must be started under root Registered ports: 1024-49151, many are assigned by IANA, no need to be root and generally unused Dynamic ports 49152-65535, ports that are freely available 2007/3/13 Web Techniques and Applications 18 9
Standard Port Numbers Port 20 21 23 25 53 70 79 80 88 110 113 119 Protocol ftp ftp telnet SMTP DNS gopher finger http kerberos pop3 ident nntp Purpose ftp data connection ftp control connection telnet session simple mail transfer protocol domain name service gopher protocol finger protocol hypertext transfer protocol kerberos authentication protocol post office protocol remote identity service network news transfer protocol 2007/3/13 Web Techniques and Applications 19 The Apache Web Server 2007/3/13 Web Techniques and Applications 20 10
Apache Web Server Apache is based on code and ideas developed at NCSA in httpd version 1.3 (early 1995) The name Apache comes from the fact that it is A set of PAtCHEs to NCSA s httpd Apache is free and is downloadable from http://www.apache.org The latest version is 2.2.4 2007/3/13 Web Techniques and Applications 21 Apache Home Page Example of open source software Available for many platforms: Win32, Win9x, Solaris, Linux, etc. 2007/3/13 Web Techniques and Applications 22 11
How does Apache Work? Apache runs under a multitasking operating system, e.g. UNIX, OS/2, Windows the binary is called httpd Key directories include: conf contains configuration files htdocs contains html files logs contains logging data cgi-bin contains executable scripts icons set of *.gif files used e.g. in directory listings src contains source files Apache idles, listening to an IP address and port 2007/3/13 Web Techniques and Applications 23 Configuring Apache - conf Directory Three files are provided with model configuration parameters httpd.conf-dist srm.conf-dist access.conf-dist These files must be edited and versions created without the -dist Partial contents of the mime.types file text/html text/plain text/richtext video/mpeg video/quicktime video/x-msvideo x-world/x-vrml html htm txt rtx mpeg mpg mpe qt mov avi wrl vrml 2007/3/13 Web Techniques and Applications 24 12
Apache Settings - httpd.conf Directives are keywords followed by their value,e.g. Port 9637 ServerAdmin horowitz@usc.edu ServerRoot /home/scf-03/cscixxx/webserver/apache_1.3.19 ErrorLog logs/error_log the above are ones that must be set Note: A complete set of instructions for how to install apache can be found on the class web page under Special Tools category. 2007/3/13 Web Techniques and Applications 25 Apache Settings - httpd.conf (Cont.) # Timeout: The number of seconds before receives and sends time out Timeout 300 # KeepAlive: Whether or not to allow persistent connections (more than one request per connection). Set to "Off" to deactivate. KeepAlive On # MaxKeepAliveRequests: The maximum number of requests to allow during a persistent connection. Set to 0 to allow an unlimited amount. They recomend you leave this number high, for maximum performance. MaxKeepAliveRequests 100 # KeepAliveTimeout: Number of seconds to wait for the next request KeepAliveTimeout 15 2007/3/13 Web Techniques and Applications 26 13
Apache Settings - httpd.conf (Cont.) Server-pool size regulation. Apache dynamically adapts to the load it sees --- it tries to maintain enough server processes to handle the current load, plus a few spare servers to handle transient load spikes. It does this by periodically checking how many servers are waiting for a request. If there are fewer than MinSpareServers, it creates a new spare. If there are more than MaxSpareServers, some of the spares die off. MinSpareServers 2 MaxSpareServers 3 StartServers 2 #number of servers to start MaxClients 3 2007/3/13 Web Techniques and Applications 27 Apache Settings - srm.conf # DocumentRoot: The directory out of which you will serve your documents. DocumentRoot /home/cscixxx/webserver/apache/htdocs # UserDir: The name of the directory which is appended onto a user's home directory if a ~user request is received. UserDir public_html # DirectoryIndex: Name of the file or files to use as a prewritten HTML DirectoryIndex index.html # FancyIndexing is whether you want fancy directory indexing or standard IndexOptions FancyIndexing 2007/3/13 Web Techniques and Applications 28 14
Authentication in Apache There are two methods for controlling access to directories The file access.conf in the conf/ directory can be used Per-directory access rules can be set by a file placed in a specific directory The name of the file is set by the directive AccessFileName.htaccess is the default name When the server attempts to retrieve a document, it looks for an access control file in the directory or the parent directory. The file it looks for is set by the directive AccessFileName 2007/3/13 Web Techniques and Applications 29 Some Sectioning Directives The server configuration files use sectioning directives to define special access rights within a section or subdirectory Example: <Directory> specifies directory to which directives apply Options controls which server features are available None All FollowSymLinks follow symbolic links ExecCGI execution of CGI scripts allowed Includes server-side includes are enabled Indexes will automatically return a list of files in the directory 2007/3/13 Web Techniques and Applications 30 15
Limit Sectioning Directive <Limit> controls which clients can access a directory; directives within LIMIT include: order in which deny and allow are evaluated deny from host1, host2,... allow from host1, host2,... require named-users or group-users or AuthUserFile referer allows access only from this directory satisfy all or any 2007/3/13 Web Techniques and Applications 31 Using LIMIT Host Filtering is used to limit document trees to certain machines Example: to limit access to the u95xxxxx public_html documents to NTHU only <Directory /usr/~u95xxxxx/public_html/> <Limit GET> order deny,allow deny from all allow from.nthu.edu.tw </Limit> </Directory> If someone tries to access documents in this directory from outside of usc they get a 403 Forbidden message 2007/3/13 Web Techniques and Applications 32 16
Using LIMIT (Cont.) Example2: to limit documents so NTHU people CANNOT access them <Directory /usr/~~u95xxxxx/public_html/> <Limit GET> order allow,deny allow from all deny from from.nthu.edu.tw </Limit> </Directory> 2007/3/13 Web Techniques and Applications 33 Using LIMIT (Cont.) Example of a directive used to limit access to only NTHU domain <limit GET> order deny, allow deny from all allow from 140.114 </Limit> 2007/3/13 Web Techniques and Applications 34 17
Example Example 3: Suppose you want to restrict files in a directory called secure/ to the user named student1 and password XXXYYY Step 1. create a file called.htaccess in directory secure/ that contains these 7 lines: AuthUserFile /otherdir/.htpasswd must be a full UNIX pathname AuthGroupFile /dev/null there is no group file AuthName ByPassword any name is OK here AuthType Basic Basic or Digest is OK <Limit GET> require user student1 </Limit> 2007/3/13 Web Techniques and Applications 35 Example (Cont.) Step 2. Next create the password file /otherdir/.htpasswd using the htpasswd program distributed with apache. Type htpasswd -c /otherdir/.htpasswd student1 and enter the password twice. You are done. Step 3. To add other names and passwords run htpasswd without the -c option. 2007/3/13 Web Techniques and Applications 36 18
Example (Cont.) To generalize this example to include several students, Step 4. create a group file called /otherdir/.htgroup which looks like: my-users: student1 student2 student3 student4 Alter the.htaccess file to look like AuthUserFile /otherdir/.htpasswd AuthGroupFile /otherdir/.htgroup AuthName ByPassword AuthType Basic <Limit GET> require group my-users </Limit> Now it points to the group file 2007/3/13 Web Techniques and Applications 37 Contents of.htaccess The.htaccess file contains pointers to the users file, the group file, and defines the name and type of authorization nunki.usc.edu(10): more.htaccess AuthUserFile /cs/u95xxxxxx/pwd/.htpasswd AuthGroupFile /cs/u94xxxxxx/pwd/.htgroup AuthName OnlineCourse AuthType Basic 2007/3/13 Web Techniques and Applications 38 19
Contents of.htpasswd This file contains a set of names and passwords cs25.cs.nthu.edu.tw(13): more.htpasswd admin:kjpokufbldrxu chungw:2ds5qooeqt1hm isa5575:ykgzc.bmvv8sy cs4710:7814vi6ojtq9c cs6710:t.osxbunnhigg isa5651:nukyncclvglh. nunki.usc.edu(13): more.htgroup admin: admin isa5575 cs4710 student: admin isa5651 cs6710 2007/3/13 Web Techniques and Applications 39 Authentication Directives AuthType specifies the type of authorization control, basic or digest Digest uses MD5 hashing If AuthType is used, then AuthName, AuthGroupFile and AuthUserFile must be defined AuthName gives the name of the realm in which the user s names and passwords are valid. For requests in the realm, the server requests a name and password AuthGroupFile gives the name of a file that contains group names and members AuthUserFile a file of usernames and encrypted passwords Both AuthGroupFile and AuthUserFile should be stored outside the document tree of the web-server; do not put it in the directory that it protects. Otherwise, clients will be able to download it. 2007/3/13 Web Techniques and Applications 40 20
Using htpasswd This program is included in the Apache distribution as htpasswd.c try htpasswd to get instructions htpasswd [-c] passwordfile username [-c] creates a new file called passwordfile and prompts for a password for username 2007/3/13 Web Techniques and Applications 41 Adding a Smart Error Page When someone requests a page or directory that does not exist it is better to return an error page than a 404 Not Found message Look at /conf/httpd.conf find the section that describes the properties of your domain, e.g. <VirtualHost www.test.com> ServerAdmin webmaster@test.com DocumentRoot /web/test ServerName www.test.com ErrorLog /log/test/error_log transferlog /log/test/access_log AgentLog /log/test/agent_log RefererLog /log/test/refer_log </VirtualHost> 2007/3/13 Web Techniques and Applications 42 21
Adding a Smart Error Page Insert the line ErrorDocument 404 /error-page.html place the error page at DocumentRoot/ErrorDocument or in this case /web/test/error-page.html Contents of error-page.html: <HTML><HEAD><TITLE>Test Error Page</TITLE></HEAD> <BODY>Error. You have requested a page that cannot be found. You may wish to return to the <A HREF=http://www.test.com/>home page by clicking here.</a> </BODY></HTML> 2007/3/13 Web Techniques and Applications 43 Analyzing Web Server Log Files Each visit to your site produces a record of the request in log file log files in Apache are typically stored in the logs directory there are several different formats Common Log Format Extended Common Log Format Extended-2 Log Format 2007/3/13 Web Techniques and Applications 44 22
Common Log Format Host ident usr time req s1 c1 client hostname or IP address remote identity, often a dash authenticated user name date/time request is received HTTP request server s HTTP response status code Content-length Example entries from access log www.cs.nthu.edu.tw - - [17/Feb/1999:23:21:09-0800] "GET / HTTP/1.0" 200 1316 140.114.229.195 - - [23/Feb/1999:10:32:09-0800] "GET / HTTP/1.1" 200 1316 2007/3/13 Web Techniques and Applications 45 23