DNS and email issues in connecting UNINET-ZA to the Internet Alan Barrett November 2011 Abstract This article describes some of the activities and configuration relating to the Domain Name System (DNS) and email handling before and shortly after the South African academic network, UNINET-ZA, was connected to the Internet in 1991. 1 A very brief introduction to the DNS As those of you with a background in Internet technology will know, the Domain Name System, or DNS, is a distributed directory or database that can answer questions like what is the numeric IP address that corresponds to the machine name, or which machines handle email for this domain. For example, it can tell you that incoming email for users at ru.ac.za is handled by four machines, and it can tell you the names and IP addresses of those machines (in both the old IPv4 and the new IPv6). Names in the DNS are hierarchical, with dots separating the levels. So ru.ac.za has three levels: ru for Rhodes University at the third level, ac for the academic community at the second level, and ZA for South Africa at the top level. Above the top level domain name ( ZA in this example) is an an even higher level called the root, which often goes un-noticed, but which is critical to the functioning of the system. Presented at Rhodes University, Grahamstown, to mark the 20th anniversary of the connection of the South African academic network to the Internet. 1
At each level (technically, at each zone, which is not quite the same thing), there is a computer called the primary name server, and a few more called secondary name servers. Collectively, the primary and secondaries are called authoritative name servers. The authoritative name servers for the root know the names and IP addresses of the authoritative name servers for.com,.net,.za, and all the other top level domains. The authoritative name servers for ZA know the names and IP addresses of the authoritative name servers for co.za, ac.za, and so on, until we get to the authoritative name servers for ru.ac.za knowing a lot of things about the machine names and IP addresses and email routing within Rhodes University. There s another class of name server called a recursive name server. Such a name server is not necessarily authoritative for anything, but it is willing to receive DNS queries and figure out the answers by following chains of referrals of the form I don t know, but try asking one of this list of servers instead. For example, if you send a question about ru.ac.za to the root servers, it can refer you to the ZA servers, and the ZA servers can refer you to the ac.za servers, or maybe directly to the ru.ac.za servers. Recursive name servers know how to use these referrals to figure out the answer to just about any DNS query. The last part of the picture is called a resolver. This is a component of the operating system or library that knows where to find a recursive name server and how to send queries to it. When you configure your computer to use a particular name server, you are actually configuring your resolver to use a particular recursive name server. 2 DNS in an isolated network The previous information is all well known to anybody with experience on the technical side of Internet operations. But it gets tricky when you have a network that can t reach the Internet at all, yet uses the same technology. Before UNINET-ZA was connected to the Internet, we were using internet protocols, including the basic TCP/IP, SMTP for email, and the DNS. We had Internetcompatible domain names, like ru.ac.za and und.ac.za. But we were not connected to the Internet, and so we had no way of sending DNS queries to the Internet s root DNS servers. So we set up our own internal root zone, with our own primary and secondary root DNS servers. 2
We also needed two versions of the ZA zone. An internal version for use inside the South African network, and having information about all the connected institutions; and an external version for use in the Internet. 3 External ZA zone The external version of the ZA zone was very simple. It specified the names and IP addresses of a few name servers (run by volunteers on the real Internet), and some wildcard MX records directing all email for *.ac.za to an email gateway at psg.com. 4 Internal root and ZA zones The internal version of the root zone said that we had three root name servers: one at the University of Natal, one at Rhodes, and one at the University of Cape Town. It also knew about the internal ZA name servers (which were actually the same three machines), and it had wildcard MX records directing email for *.edu, *.com, *.net, *.uk, etc., to a gateway machine. The internal version of the ZA zone knew about all the third level domains like ru.ac.za, und.ac.za, uct.ac.za, wits.ac.za, frd.ac.za. (We kept ac.za in the same zone as za.) Some of those third level domains had not yet fully connected to the UNINET-ZA TCP/IP network, and we just gave them wildcard MX records directing their email to a gateway machine. Other third level domains belonged to institutions that were fully connected to the UNINET-ZA TCP/IP network, and the internal version of the ZA zone knew the names and IP addresses of the name servers for those institutions. The institutions name servers, in turn knew the details of what went on inside each institution. When changes were made to the information in the internal zones, the changes would be loaded into the primary name servers, and then copied to the secondary name servers using the standard zone transfer protocol. We did encounter some bugs in the DNS software, and I fixed a few of them, but what we did was almost exactly the same as what would have had to be done by people managing the root zone or other top level domains in the real Internet. 3
5 Configuration required to make use of the internal DNS Once we had an internal version of the DNS root zone, we naturally wanted machines on the internal network to use the internal root zone. That would also make them follow referrals from the internal root zone to the other internal zones. There were two parts to this: configuring resolvers, and configuring recursive name servers. All Internet-connected computers need to have a resolver that knows the IP address of a nearby recursive name server. What typically happens is that each university, department, business, or ISP, provides a few recursive name servers, and provides instructions for configuring all the other computers to use those recursive name servers. The same applies on a network that uses internet protocols without being connected to the Internet. Nowadays this sort of configuration is almost always automated, but in 1990 it was almost always manual. In order to help smaller institutions that did not have the equipment or expertise needed to set up their own recursive name servers, a few of the larger universities connected to UNINET-ZA allowed their recursive name servers to be used by people at the smaller institutions. All recursive name servers need to know the IP addresses of one or more of the authoritative root name servers. On the real Internet, the list of authoritative name servers and their IP addresses changes so slowly that the list can be included with the operating system and still be usable several years later, so most people never need to worry about it. On our internal network, we had a completely different set of root name servers, so the operators of all recursive name servers needed to manually configure them. We distributed a file with the information in the format used by the then most common name server software, BIND, and instructions for how to install the file and instruct BIND to use it. 6 Typical use before connection to the Internet Before the connection of UNINET-ZA to the Internet, computers inside UNINET- ZA would use recursive name servers that knew about the internal root DNS zone, so they would see only the internal DNS system. DNS queries about destinations within the ZA domain would be answered from the internal DNS zones. DNS 4
queries for MX records (used for routing email) outside the ZA domain would be answered with wildcard records directing all email to gateway machines at Rhodes and the University of Natal, which were able to send email to the outside world using non-internet transports, such as UUCP and Fidonet. Meanwhile, on the real Internet, more or less the opposite was the case. They could get proper answers to DNS queries about names other than ZA, but the only information for ZA domains was in the form of wildcard MX records directing email to gateways at psg.com. 7 Changes to the ZA zones after connection to the Internet After UNINET-ZA was connected to the Internet, it became possible to send DNS queries to both the internal and external versions of the root name servers and the ZA name servers. Of course you would get different results depending on which servers you queried. We needed to make this consistent. An easy first step was to make the external version of the ZA zone identical to the internal version. We edited the internal version of the ZA zone to include NS records for both the internal and external authoritative name servers, and we started referring to it simply as the ZA zone, not the internal ZA zone. On the machine that had previously been the primary name server for the external version of the ZA zone, we changed the configuration to make it a secondary server for the new ZA zone. Now the same version of the ZA zone was being distributed to all the internal and external authoritative name servers. 8 Phasing out use of the internal root servers We changed the internal version of the root zone to be a copy of the real, or external, root zone. Now internal machines that sent queries to the internal root servers would get the same answers as if they had sent queries to the external root servers. We also distributed configuration files and instructions for how to reconfigure all recursive name servers to start using the external root name servers instead of the internal root name servers. Many months later, we ceased operating the internal root name servers. 5
9 Changes to email routing Even though it was technically possible to send email directly using the SMTP protocol from a computer inside UNINET-ZA to a computer on the Internet outside South Africa, or vice versa, the connection had such low bandwidth and was so heavily congested that this did not work well. To help with incoming email (from the outside world to UNINET-ZA sites), the internal sites could (with or without permission) create backup MX records sending their email to a gateway at psg.com. When network congestion made the site s primary mail server unreachable from the outside world, the backup MX record would direct email to the gateway, which has special configuration to deal with such email. To help with outgoing email (from UNINET-ZA sites to the outside world), the sites could specially configure their email systems to send such mail to a gateway machine. I operated one such gateway at the University of Natal. 10 Email transport across the international link It s not enough for email to collect on a gateway machine at one side of the international link. We wanted the email to actually be transported from one gateway, across the international network link, to a gateway at the other side, and then to be delivered to its final destination. We discovered that the SMTP protocol did not work very well over the highly congested link. Connections would often time out and fail, and when they worked, they would be very slow. We wanted to use compression, and a batch protocol instead of an interactive protocol. The email protocol we chose for use between psg.com and und.ac.za was compressed batch SMTP over UUCP over TCP. We already had experience with essentially the same thing over dialup modems instead of over TCP, so it was easy to set this up. Compressed batch SMTP means that the email messages are encapsulated in a file format called bsmtp or batch SMTP, which looks like one direction of the SMTP protocol (just sending the commands and data, without waiting for lineby-line replies); then multiple email messages were batched together in each file, up to some configured size limit, and the files are compressed. 6
UUCP over TCP was much more robust that SMTP when dealing with the severely congested international link. However, we found that the unix rcp program worked better than UUCP at transferring files over the congested link. We created a hybrid transport that involved creating compressed batches using UUCP, and then transferring the batch files using rcp instead of UUCP. We were able to gain even more efficiency by transporting only the data files, not also the control files that UUCP uses. The UUCP control files are very small (probably less than 100 bytes), but each control file could take tens of seconds to transmit due to the severe congestion. 11 More DNS issues Somehow, incorrect IP addresses for some computers at psg.com had been entered as glue records in DNS zone files for some third level domain under.ac.za. These incorrect IP addresses propagated in unexpected ways until they polluted most of the name servers in South Africa, and many name servers elsewhere. The reason for the propagation was traced to bugs in the name server software (technically, failure to keep glue records separate from authoritative records), combined with the way organisations often provide secondary name service for each other. At one point, I replaced the zone transfer program with a version that specifically checked for and removed this invalid IP address. We also sent many email messages to the operators of name servers known to have the incorrect record, giving instructions for how they could remove it. There were also problems with many sites listing machines at other sites as backup mail servers (MX records) or secondary name servers (NS records) without first getting permission, leading to lost email, or error messages in log files, and to attempts at education for the miscreants. 7