PolyServe High-Availability Server Clustering for E-Business 918 Parker Street Berkeley, California 94710 (510) 665-2929 www.polyserve.com Number 000217 White Paper Configuring Apache and IIS for High Availability Web Server Clustering Many service providers must support a large number of web sites on each of their web servers. Doing this can be a challenge in that if any of the servers or services goes down, possibly many users, across a number of web sites, can lose access to the web site and its information. Configuring web servers to handle multiple web sites may also be a challenge, in that it may take many IP addresses to uniquely address the individual web sites on each of the servers. This paper specifically addresses how to configure and manage web servers to handle large numbers of web sites on a set of servers in a high-availability web cluster, using only a single (or a few) IP addresses. We focus on two of the more popular web servers: Apache and Microsoft IIS. Both now contain features to allow named based virtual hosting (or host header support) rather than the older IP based virtual hosting. These new features allow a minimum number of IP addresses to be used to manage multiple web sites on a set of servers. Further, we show how to configure Understudy and its virtual host capability, together with Apache and IIS, to achieve highly available web server clusters. We also discuss using these configurations to setup load balancing across the cluster using a minimum number of IP addresses. In addition, we discuss the use of data synchronization techniques to dynamically replicate web content across servers in a cluster, to minimize maintenance costs and guarantee consistent web data. Using Understudy with Apache and IIS, a highly available set of web servers can be defined to host multiple web sites. These web sites can serve large numbers of client requests without paying the high cost for a sophisticated traffic management system or high-end switch. This allows ISP/ASPs.com s, and other service providers, to cost effectively manage web server clusters by minimizing their costs, both in equipment, maintenance, and setup time. This paper contains the following sections: Configuring Apache & Microsoft Internet Information Server (IIS) for Multiple Web Site Hosting o Configuring Apache Web Server for Named-based Virtual Hosts o Configuring Microsoft IIS to use Host Header Names Using Understudy to Achieve Highly Available Web Server Clusters with Apache & IIS o Configuring Web Servers for Primary/Backup Operation o Configuring Web Servers for Load Balancing Operation Static Load Balancing Web Sites across a Server Cluster Dynamically Replicating Web Site Content Configuring Apache & Microsoft Internet Information Server for Multiple Web Site Hosting HTTP version 1.0 did not support the ability to uniquely identify web sites by name and therefore required a unique IP address to access an individual web site. This forced service providers to use blocks of IP addresses to uniquely access each web site. If the service provider needed to provide multiple web sites per server, the result was that a lot of IP addresses were burned up in the process. To solve these and other related problems, HTTP version 1.1 provides support for host header names to be used to host multiple web sites using a single IP address. These changes are required to be supported by both the web server and browser to be effective. The following versions of Apache and IIS web servers, and Netscape and Internet Explorer browsers, are HTTP 1.1 compliant. Apache Server Version 1.3+ Microsoft Internet Information Server (IIS) Version 4.0+ Netscape Navigator 3.x+ Internet Explorer 3.x+ PolyServe Tech Tip #000217 Configuring Apache & IIS for High Availability Clustering
Both Apache and IIS have traditionally supported a virtual host concept. This ability was known as IP-based virtual hosting by Apache, but applied in the same way to IIS. With IP-based virtual hosting, the server must have a different IP address for each IP-based virtual host. In order to support multiple web sites on a server, this generally meant configuring multiple IP aliases on the network interface(s) for the servers. A description of IP-based virtual hosting is available at the Apache organization web site. With Apache Server 1.3 and IIS 4.0, it is now possible to configure the new named-based virtual hosting feature to support multiple web sites with a single IP address. The following set of documents go into detail on how to setup and configure these web servers for named-based virtual hosts: Apache 1.3 User s Guide : Named-Based Virtual Hosting Microsoft IIS: Using Host Header Names to Host Multiple Sites from One IP Address Both documents describe how to support browsers that are not HTTP 1.1 compliant via the use of workarounds. Although most browsers are now HTTP 1.1 compliant, there are older versions out there, so be aware of this. We provide an overview of the configurations for Apache and IIS with named-based virtual hosts. Please refer to the vendor documents above, together with the appropriate users guides for both Apache and Microsoft IIS. Configuring Apache Web Server for Named-based Virtual Hosts Apache version 1.3 now has a new directive known as the NameVirtualHost directive which specifies an IP address that should be used as a target for name-based virtual hosts. An example makes this more clear: Let s assume we would like to access both www.mysite.tld and www.yoursite.tld using the same IP address 150.1.1.100. By adding to one of the Apache configuration files (httpd.conf or srm.conf) code such as follows, this is possible: NameVirtualHost 150.1.1.100 <VirtualHost 150.1.1.100> ServerName www.mysite.tld DocumentRoot /www/mysite <VirtualHost 150.1.1.100> ServerName www.yoursite.tld DocumentRoot /www/yoursite Other directives can also be placed into the <VirtualHost> section as well. The most important point is that both virtual hosts must be pointing to the same IP address 150.1.1.100. It is important to also note that the main server will never be served from the virtual hosts IP address. If you employ the virtual host feature, you should stop using the main server as an independent server and instead use it as a location for directives that are common for every virtual host. These and other related issues are discussed in the appropriate Apache document. Compatibility with older (non-http 1.1 compliant) browsers are also discussed. The Apache User s Guide also contains a set of virtual host examples which are instructive in assisting web server managers in configuring their Apache web servers correctly. The main advantage in using Apache named-based virtual hosts is that a single IP address can manage an entire set of web sites on a single server. As we will show later, this can be used with Understudy s virtual host capabilities to allow a single IP address to access multiple web sites across a high-availability server cluster. - 2 -
Configuring Microsoft IIS to use Host Header Names (named-based virtual hosts) Microsoft Internet Information Server version 4.0 is HTTP 1.1 compliant and as a result can support host header names to host multiple web sites from a single IP address. Microsoft has a technical article (Using Host Header Names to Host Multiple Sites from One IP Address) that clearly spells out how to configure IIS to support what is termed named-based virtual hosting by the internet community. We will provide a snapshot of these directions in this section. But we recommend reading the article and following the instructions in the appropriate Microsoft NT Option Pack documentation. Also note that what Apache refers to as a named-based virtual host, Microsoft refers to as host header name support (they are referring to the host header in HTTP 1.1), but in fact they are the same concept. Host header names can be used to host web sites from a single IP address. The following steps allow IIS to be configured to support this feature: 1. Start the Internet Service Manager (ISM). This loads the plug-in for the Microsoft Management Console (MMC). 2. Click the right mouse button on the web site that should use a host header name (named-based virtual host), and click Properties on the shortcut menu. 3. On the tab for the Web Site, select the IP address that the site should use (Select All Unassigned if you only have a single IP address on the server), and set the TCP Port that should be used (this usually defaults to port 80). 4. Click on the Advanced button. 5. In the list Multiple Identities for this Web Site, click the identity that you want to use. 6. Select Edit, and add the host header name that is desired. Also, if this site should respond to more than one host header name, select the Add button to add other identities to this list. You may specify a different host header name for each identity, but you must use the same IP address and port. 7. Apply the changes and re-start the Web site. 8. Register the host header names with your domain naming system. If the web server is on a private LAN or a Virtual Private Network, register it with your intranet name resolution system (such as WINS). If the computer is on the Internet, register it with your Domain Name Service (DNS). 9. After you have registered the host header name, attempt to access the site by using a browser with the host header name. Make sure the browser is HTTP 1.1 compliant. Warning Don t use a host header name with the default web site. This may result in strange behavior with packages for IIS (such as Microsoft Proxy Server) that expect the default web site to use the All Unassigned IP Address, TCP port 80, and have no host header name. When IIS web server gets a request for a Web page, it looks at the information sent by the browser. If the browser is HTTP 1.1 compliant (IE 3.x and later, or Netscape Navigator 3.x and later), the HTTP header contains the domain name requested. IIS uses this to determine which site should answer the request. For additional information, see the following topics in the Windows NT Option Pack documentation: Microsoft Internet Information Server o Server Administration o Name Resolution o About Name Resolution Microsoft Internet Information Server o Server Administration o Web and FTP Sites o Naming Web Sites If the browser is non HTTP 1.1 compliant, there is a workaround to allow host header support to function properly. You can program a core page to display the sites available and the user can select which site to use. For instructions on how to do this, refer to the following documentation: - 3 -
Microsoft Internet Information Server o Server Administration o Web and FTP Sites o Supporting Host Header Names in Older Browsers For more information on these and other related subjects, use the Microsoft Knowledge Base to access: Q186810 Event ID 115 When Using Host Headers with IIS The main advantage with using IIS host header names is that a single IP address can manage an entire set of web sites on a single server. As we will show, this can be used with Understudy s virtual host capabilities to allow a single IP address to access multiple web sites across a high-availability server cluster. Using Understudy to Achieve Highly Available Web Server Clusters with Apache & IIS Understudy allows servers to be grouped together into clusters for high-availability and load sharing. Understudy does this by allowing virtual hosts to be defined in a way that is similar to how Apache and IIS define them. But, whereas virtual hosts in Apache and IIS allow multiple web sites to be managed on a single server with a single IP address, Understudy s virtual hosts allow client requests to be mapped to a set of clustered servers. Understudy insures that these client requests are properly mapped to a server and service that is up and operational. Understudy does this by monitoring servers and services (such as HTTP) and if a server or service failure is detected, Understudy can switch client requests from one server to another in the cluster. To review Understudy and its operation, an overview exists on the Polyserve product web site and in the Understudy datasheet. These documents describe in more detail Understudy s features, configuration, and operational characteristics. It is important to understand an Understudy Virtual Host and its meaning. An example best illustrates this. Suppose there are two servers: Server A and Server B. Server A is known to the naming service as server A and has as its IP address 192.168.100.1; while Server B is known as server B and has an IP address of 192.168.100.2. Understudy runs on both servers and allows multiple virtual hosts to be defined for high availability purposes. Let s assume that a virtual host named virtual_server exists in Understudy. The virtual host virtual_server has its own IP address (it must be unique) of 192.168.100.100. Understudy then asks the server manager which server will be the primary and which will be the backup server for this virtual host. The configuration would be as follows: Virtual Host Primary Server Backup Server virtual_server servera serverb Understudy operates in the following way: clients that access virtual_server are sent to server A (the primary). As long as server A is operating correctly, it handles all client requests. If server A goes down, or any service monitor on server A reports a failure, then server B takes over the IP address associated with virtual_server. It does this by using a gratuitous ARP message. As long as server A or the monitored service is down, server B will continue to receive client requests because server B is now managing the IP address for virtual_server. If server A or the monitored services come back up, it takes over control of the IP address again, and it handles all client requests. Understudy can manage many virtual hosts per cluster, this allows many web sites to be supported per cluster. And using Apache & IIS host header support, this can be done in a way that minimizes the use of multiple IP addresses, yet still achieving high availability. Understudy can be configured to manage virtual hosts in an active-passive configuration (one host is the primary, the rest are backups) or in an active-active cofnguration (load balancing). Load balancing can be done by combining Understudy with DNS round robin. A detailed white paper exists to assist users in setting up Understudy with DNS round robin load balancing. This paper will show web server managers how to configure Apache and IIS to take advantage of Understudy s highavailability features in both an active-passive and active-active configuration for support of multiple web sites. We will focus on configuring Understudy with Apache & IIS using the new HTTP 1.1 features to minimize the use of IP addresses. - 4 -
Configuring Web Servers for Primary/Backup Operation This section will show how to configure either Apache or IIS on a pair of web servers running Understudy so that if the primary server fails, the backup will immediately take over operation of any and all client requests. We focus on managing this configuration for multiple web sites with a single IP addresses. Let s use an example to illustrate this setup and make the following assumptions: There are two servers (Server A 192.168.100.1; Server B 192.168.100.2) Multiple web sites are to be hosted. Their names are: o www.site1.com o www.site2.com o www.siten.com (representing N web sites for perhaps N different customers) Each of the servers has a duplicate copy of the web site To begin this example we need to configure Understudy by building a virtual host. The advantage with HTTP 1.1 support in Apache 1.3 and IIS 4.0 is that only a single IP address is required to support multiple web sites. With previous versions of Apache & IIS (that only supported HTTP 1.0) multiple virtual hosts, with an IP address for each, would have to be defined to manage multiple web sites. Now, only a single virtual host is required. Here is how to do it. First, Understudy must be configured with a single virtual host that will be used to access all web sites on each server. Using our example above, we configure the virtual host as follows. Note that an HTTP service monitor is also defined for the virtual host. This allows HTTP to be monitored to validate that the web server software is serving pages correctly. Virtual Host Primary Server Backup Server Service Monitor 192.168.100.100 servera serverb HTTP As we shall see, the single virtual address 192.168.100.100 will allow clients to access any web site on the servers, and Understudy will ship the client requests to the primary or the backup depending upon the state of the cluster. The following entries are also required in the DNS Server or internal name server: www.site1.com 192.168.100.100 IP address matches Understudy virtual host IP Address www.site2.com 192.168.100.100 www.siten.com 192.168.100.100 Note that each of the web sites has the same IP address (the address of the virtual host defined by Understudy). If you recall how Understudy works, requests will be sent to the primary server (Server A) unless it or HTTP goes down, in which case the requests will be sent to the backup server (Server B). We can now configure Apache and IIS to show how their support of HTTP 1.1 host header names allows a highavailability server cluster to be operated using a single IP address. - 5 -
Configuring Apache to Support the Primary/Backup Cluster Operation We configure Apache to support our primary/backup high-availability configuration by altering the httpd.conf file on each of Server A and Server B to include the following: NameVirtualHost 192.168.100.100 <VirtualHost 192.168.100.100> ServerName www.site1.com DocumentRoot /www/site1 Matches DNS entry & Understudy virtual host IP address Matches NameVirtualHost entry above <VirtualHost 192.168.100.100> ServerName www.site2.com DocumentRoot /www/site2 <VirtualHost 192.168.100.100> ServerName www.siten.com DocumentRoot /www/siten Note that each VirtualHost statement list the virtual address defined by Understudy (and in DNS) for each web site. Configuring IIS to Support the Primary/Backup Cluster Operation In the same manner as Apache, we can configure Microsoft Internet Information Server to support our primary/backup cluster operation as well. From the section on configuring IIS, we perform the following tasks to configure IIS: 1. Start the Internet Service Manager (ISM). This loads the plug-in for the Microsoft Management Console (MMC). 2. Click the right mouse button on the web site that should use a host header name (www.sitex.com), and click Properties on the shortcut menu. 3. On the tab for the Web Site, select the IP address that the site should use (Select All Unassigned if you only have a single IP address on the server), and set the TCP Port that should be used (this usually defaults to port 80). 4. Apply the changes and re-start the Web site. 5. Register the host header names (the correct DNS entries are shown above for www.sitex.com) with your domain naming system. If the web server is on a private LAN or a Virtual Private Network register it with your intranet name resolution system (such as WINS). If the computer is on the Internet, register it with your Domain Name Service (DNS). 6. After you have registered the host header name, attempt to access the site by using a browser with the host header name. Make sure the browser is HTTP 1.1 compliant. Note that these instructions differ from the ones for IIS shown in the previous section as we have eliminated the need to support the Multiple Identities for this Web Site option. If you need to support this, then simply run through those commands as well. - 6 -
Understudy and Apache/IIS Primary/Backup Web Server Clustering in Operation Let s review the configuration and its operation. Understudy is configured for server A to be the primary and server B the backup. Apache or IIS are configured to use a single IP address (the virtual IP address defined by Understudy for the cluster of server A and Server B). DNS points each web site at the virtual IP address. Perhaps most importantly, each web site must have a duplicate copy of each sites web data. ( Alternatives to this approach are discussed in the last section of this paper). The operation of this web server cluster shows how it guarantees high availability even when one of the servers fails. Each client that accesses one of the web sites is sent to the virtual IP address 192.168.100.100 this resolves to Server A as long as Server A is up and operational, (and the HTTP service monitor on Server A is validating that Apache/IIS is serving pages from the web sites). Regardless of which web site is accessed by the client, Apache/IIS are configured to serve pages from the correct web site based on the HTTP 1.1 host headers. If Server A can no longer respond to client web requests (because the server goes down or the HTTP service monitor is no longer seeing responses to its queries), Understudy on Server B sends a gratuitous ARP indicating that it is now taking over the virtual host defined by IP address 192.168.100.100. Now, requests to any of the multiple web sites are sent to Server B (the backup server). Since Apache/IIS on Server B are configured to accept all web sites on the virtual host IP address 192.168.100.100 all client requests for any of the multiple web sites are correctly served. When Server A becomes operational, it takes back over control of the virtual host IP address. With this configuration of Understudy, Apache or IIS, and DNS, as long as at least one of the web servers in the cluster is operating correctly, the clients of all these web sites will be correctly served with just a single IP address. Configuring Web Servers for Load Balancing Operation In the previous section we showed how to configure Understudy and Apache or IIS for Primary/Backup operation. What if you would like load balance client requests across multiple web servers, but if any one of them failed, the others would take over operation for the failed server? Again, using Understudy, Apache or IIS, and DNS, this is possible using only a single different IP address per server. The configuration below is only valid for host header/named-based virtual host support (HTTP 1.1). Let s make the same assumptions and use the same example as in the previous section. There are two servers: Server A and Server B, with the same IP addresses as before (192.168.100.1, 192.168.100.2). Again there are multiple web sites, www.site1-siten.com. Each of the servers has an identical copy of each of the web sites to be served. First let s show how Understudy should be configured. We are going to play a trick using DNS round robin and Understudy to configure load balancing. Please refer to our white paper on DNS round robin to understand how this setup operates. The basic setup of Understudy is as follows: Virtual Host Primary Server Backup Server Service Monitor 192.168.100.100 servera serverb HTTP 192.168.100.101 serverb servera HTTP We now have two virtual host IP addresses defined rather than one. Note that the first virtual host s primary server is Server A, the second virtual host s primary server is Server B. Now we must setup DNS to support the round robin feature that is inherent in Bind. Again, this is analyzed in more detail in the white paper DNS round robin, but the basic concept is that since DNS can support round robin by placing multiple IP addresses in the name server entry for a particular access site, we replace the real IP addresses with the virtual IP addresses we have defined with Understudy. DNS round robin allows DNS to alternate between IP addresses when it resolves names. In our example above, for www.site1.com, we would normally put the IP addresses of the real servers in the appropriate DNS files. This would round robin requests to - 7 -
these servers. By putting the virtual IP addresses defined by Understudy in the DNS files, in case of failure the servers have appropriate backups. The required DNS entries for the example are: www.site1.com. 60 IN A 192.168.100.100 Understudy Virtual IP addresses www.site1.com. 60 IN A 192.168.100.101 www.site2.com. 60 IN A 192.168.100.101 www.site2.com. 60 IN A 192.168.100.101 www.siten.com. 60 IN A 192.168.100.101 www.siten.com. 60 IN A 192.168.100.101 Note the. at the end of the host name. This is important. See our DNS white paper for more details on setting up DNS round robin. Each of the web sites is given two IP addresses to round robin client requests. These IP addresses are the virtual IP addresses defined for Understudy to manage. An example of this in operation will help to understand how this works. Client requests for www.site1.com are round robin rotated between 192.168.100.100 and 192.168.100.101 which are in fact virtual IP addresses. As long as Server A and B are both up, 192.168.100.100 resolves to Server A and 192.168.100.101 resolves to Server B. Assume that Server A goes down then Understudy on Server B realizes the failure and takes over the virtual address 192.168.100.100. So all requests to this IP address now go to Server B. When Server A comes back up, it takes control of 192.168.100.100 again. By using Understudy and DNS round robin, a high-availability, load sharing cluster can be configured. Now that we have defined how to configure round robin load sharing with Understudy, let s complete the picture by showing how to configure Apache or IIS for named-based virtual hosting of these web sites. The basic concept is that since each web site has two IP addresses that it could be accessed from, the Apache NameVirtualHost statements and the IIS web sites need to be given for each one. Let s go through this for both Apache and IIS web servers. Configuring Apache to Support Load Balancing Cluster Operation In our example, the configuration for Apache would look as follows (we show the two VirtualHost records side by side, on the same lines to emphasize that you need two VirtualHost statements per web site). This configuration allows Apache to be used with DNS round robin and Understudy to achieve a high-availability, load sharing, web server cluster for multiple web sites using only one IP address per server in the load balanced cluster. In our example there are two servers in the cluster, therefore 2 IP addresses are needed for load balancing operation. The Apache configuration looks like: NameVirtualHost 192.168.100.100 NameVirtualHost 192.168.100.101 Corresponds to Understudy Virtual Hosts <VirtualHost 192.168.100.100> <VirtualHost 192.168.100.101> ServerName www.site1.com ServerName www.site1.com DocumentRoot /www/site1 DocumentRoot /www/site1 <VirtualHost 192.168.100.100> <VirtualHost 192.168.100.101> ServerName www.site2.com ServerName www.site2.com DocumentRoot /www/site2 DocumentRoot /www/site2 <VirtualHost 192.168.100.100> <VirtualHost 192.168.100.101> ServerName www.siten.com ServerName www.siten.com DocumentRoot /www/siten DocumentRoot /www/siten - 8 -
Note that these entries above for the appropriate Apache files are required on each server in the cluster. Configuring IIS to Support Load Balancing Cluster Operation Configuring IIS to support load balancing is actually a little easier than Apache. The main thing is to use the All Unassigned feature so that any of the virtual IP addresses that access the web servers will be accepted by IIS and point to the correct web pages. The instructions to carry this out are as follows: 1. Start the Internet Service Manager (ISM). This loads the plug-in for the Microsoft Management Console (MMC). 2. Click the right mouse button on the web site that should use a host header name (www.sitex.com), and click Properties on the shortcut menu. 3. On the tab for the Web Site, select All Unassigned and set the TCP Port that should be used (this usually defaults to port 80). 4. Apply the changes and re-start the Web site. 5. Register the host header names (the correct DNS entries are shown above for www.sitex.com) with your domain naming system. If the web server is on a private LAN or a Virtual Private Network register it with your intranet name resolution system (such as WINS). If the computer is on the Internet, register it with your Domain Name Service (DNS). 6. After you have registered the host header name, attempt to access the site by using a browser with the host header name. Make sure the browser is HTTP 1.1 compliant. You may also attempt to access it from another browser that is HTTP 1.1 compliant to validate the load balancing operation (both servers should be active). Understudy and Apache/IIS Web Server Load Balancing in Operation Load Balancing of Apache and IIS web servers is fairly straightforward given the configuration we have just defined. Understudy uses a pair of virtual hosts to round robin requests between Server A and Server B. In case any of the hosts or services on those hosts fail, the virtual host corresponding to that primary server fails over to its backup server. In our example, if Server A failed, the virtual IP address that points to Server A as the primary would fail over to Server B. Therefore, round robin requests to the first virtual IP address (that normally go to Server A) would go to Server B; the second virtual IP address would go to Server B as well. When Server A was operational again, the first virtual IP address would again point back to Server A. This is all accomplished for both Apache & IIS with only one IP address per server in the cluster. This is a tremendous saving versus using the old IP-based virtual hosting inherent in HTTP 1.0. Static Load Balancing Web Sites across a Server Cluster There is another level of load balancing that can be achieved in your web server cluster. Unlike the previous section, this is static in nature. The basic idea can best be illustrated with an example. Say you have web sites that you would like to cluster on a pair of servers. You setup the following static cluster configuration: Web Site Primary Server Backup Server www.site1.com Server A Server B www.site2.com Server B Server A www.site3.com Server A Server B www.site4.com Server B Server A Alternating which web site has its primary on A and which has its primary on B allows you to statically balance the web requests across the servers in the cluster. Obviously this isn t as effective as load balancing using DNS round robin, but there are cases where DNS round robin is difficult to implement. For instance, if the DNS server is outside of your control, the necessary DNS table changes may be difficult or impossible to make. If some web sites are more heavily used than others, this static load balancing method will not be as effective. - 9 -
Dynamically Replicating Web Site Content Throughout this document we have made one big assumption, which can be difficult to achieve. That assumption is that if failover occurs on any server in the web cluster, the data will be the same on all web servers in the cluster. In practice, this synchronization can be a challenge to insure. Procedures to do this manually, or homegrown methods to do this automatically, are often employed. In either case, this process can be labor intensive. PolyServe understands that this is a problem and our next release of software will provide data replication services for automatic web content distribution, email synchronization, and file replication. In the meantime, manual or automatic mechanisms must be put in place to guarantee that the web content is the same on each server. Another alternative is to use a file or database server that is shared amongst the web servers in the cluster. Although, this can lead to single points of failure and performance bottlenecks. By integrating the data replication and clustering technology into the same software package, we feel that we can significantly reduce the maintenance costs normally associated with web servers, while still insuring high availability. Data synchronization technologies can also assist in replicating configuration data as well application data. We intend to provide a variety of application data replication services to enhance our clustering technology. - 10 -