NSFNET: The T1 The Internet Comes of Age November 29, 2007 Eric Aupperle, Rick Boivie, Paul Bosco, Hans-Werner Braun, Mathew Dovens, Mark Knopper, Yakov Rekhter, Walter Wiebe, Jessica Yu
Remarks on the NSFNET T1 Backbone Hans-Werner Braun University of California-San Diego
Thank you, Eric. I guess you did not expect to share a stage in Washington, DC, with me now, when you hired me almost 25 years ago? I would also like to express my appreciation to the DoD Advanced Research Projects Agency for developing the underlying framework that made the NSFNET and the Internet possible. Special thanks also to Dave Mills for being my mentor on Internet matters, at a time when I knew nothing about it. In the mid-1980s, before the NSFNET, access to the Internet was rather limited. At that time then ARPA-Internet was basically authorized only to the US Department of Defense and its contractors. The NSFNET changed this at a time when using the Internet Protocol was frowned upon, due to a new mandate by the US government towards using OSI protocols instead.
The original 56 kilobits per second NSFNET backbone, using Fuzzball-based LSI11 nodes, became operational in mid-1986 between the five new NSF supercomputing centers and the National Center for Atmospheric Research. Those formally responsible got the links and equipment in place, but ran into difficulties making the system work. I stepped in and turned the backbone into a functional state, with access via NCAR's University Satellite Network project, and basically ran the backbone from then on, until the new T1 network came online in mid-1988. The rather open-access nature of the NSFNET and new routing paradigms constituted a challenge for the Internet, which until then was basically a hierarchical structure centered around the Arpanet. With the NSFNET, there were suddenly multiple national backbones with very different administrations and modes of operation.
I was the Co-Principal Investigator for the new NSFNET award to Merit at the University of Michigan. With Eric Aupperle as the Project Director having Merit commitments beyond the NSFNET project, responsibility to make the system and its architecture work was on my shoulders, with help from many people working on the components. I also had to spend lots of challenging time with the Internet community, trying to make them believe in our concepts, and that we would make things work, and I had to argue with those who could just not believe that we would be able to make it work at all. We were working hard with Merit's dedicated staff, and our partners IBM and MCI, to make the project succeed. There was a lot of new ground to cover. The backbone nodes, which IBM built, were basically architected from scratch, and were really a distributed system of nine small computers, plus various other parts.
MCI thought we were pretty crazy, wanting unchanneled 1.5 megabits per second T1 links, instead of multiplexing them down to 56 kilobit per second voice-like circuits. When we told them that within a few years we need unchanneled 45 megabits per second DS3 links, they thought we are completely insane. But, both IBM and MCI worked with us through all the issues, and we made it all work in time and within NSF's budget, and they were generally excellent project partners and a real pleasure to work with.
Let us also not forget that an integral partner, as important as any other one on this project, was the National Science Foundation, as part of the NSFNET cooperative agreement. Steve Wolff and his staff were very involved, continually part of the decision making process, and pleasant to work with. Beyond the award itself, the National Science Foundation most certainly deserves as much credit as anyone else for the success of the NSFNET.
I think a lot of the initial Internet explosion was due to NSF's openness for usage of the NSFNET. Suddenly many more people, especially universities, came to the table. A meeting Scott Brim and I had with Pentagon officials resulted in opening up the Internet Engineering Task Force to the NSFNET and beyond. New routing complexities required innovative approaches, resulting in things like the creation of the still-in-use Border Gateway Protocol, to hold a meshed Internet of many autonomous constituents together. All this laid the ground work for an Internet industry to eventually step in, especially around the mid-1990s, to create an all but globally ubiquitous cyberinfrastructure that would have a profound world-wide societal impact.
By the time I left Merit and the University of Michigan near the beginning of 1991, to work on research projects at the San Diego Supercomputer Center, we had already made great progress towards upgrading the NSFNET backbone to a 45 megabit per second T3 capability. We had demonstrated this in December 1990 to the NSFNET community: cross country and across several T3 nodes. Another clear success of the partnership. I also continued to work with the National ScienceFoundation towards new network infrastructure architectures. I certainly did not expect the degree of success NSFNET had, and how many people and projects would quickly come to depend on it. But those kind of things are still happening, as the Internet gets brought towards even more remote areas and to new uses. If the underlying concept is sound, like it is for the Internet, and it addresses a real need, it should come as no surprise to see heavy growth.
In closing, I would like to point out that the Internet should not be viewed as a static "build it and they will come" environment. It is more like a living organism, that has to constantly evolve, build new components, replace or abandon old ones, and so on. Including always anticipating the need to accommodate new applications, new users, new environments, new technologies, and a seemingly inexhaustible hunger for ubiquity and performance.
Pioneering New Routing Technology and Architecture New routing architecture: Separation of Interior routing from exterior routing New routing protocol: IS-IS Evolution from EGP to BGP
Tackling The Routing Scaling Challenges Routing scaling challenges with the fast growth of NSFNET Routing table doubled every 10 months Classful routing does not scale Classless Internet Domain Routing (CIDR) developed
Registered IPv4 Address Allocation History Graph source: online
Summary The NSFNET backbone directly contributed to the advancement of routing technology in the Internet: First deployed IS-IS and BGP CIDR development and deployment It was a very unique experience to work on the NSFNET from the very beginning I am very proud of having been part of the team
NSFNET: The T1 IBM Overview 11-29-07 Walter Wiebe
Partner Appreciation & Background IBM ACIS & Communications Joint Studies: CMU Andrew File Server & Campus LAN U of WI VM TCP/IP U of MD DOS TCP/IP Yale LAN/ASCII Gateway Cornell Theory Center / Supercomputer Brown Hypermedia/Computer aided Learning U of M Institutional File Server ACIS related projects: BSD 4.2 Unix for IBM 370 including TCP/IP stack AOS 4.2 and 4.3 (BSD Unix for the IBM RT/PC) o Including TCP/IP stack and h/w and s/w for ethernet and token ring TCP/IP training workshop 4-86 TCP/IP workshop at U of MD in fall of 86 Laureate Series Announced April 1987 VM/TCPIP DOS/TCPIP 8232 Channel to LAN LAN ASCII Gateway Almaden, Yorktown, Kingston, Poughkeepsie, Manassas, Raleigh - R & D collaboration teams
T1 NSFNET Key Reviews & Dates NSFNET Project Solicitation 6-15-87 IBM reviews: Gomory, Krowe, Goldberg, Lucente Proposal submitted by Merit 8-15-87 IBM & MCI as Joint Study partners Simulated backbone network running in Milford lab 10-87 Contract to develop and manage the NSFNET awarded to Merit MCI & IBM 11-24-87 Test network lines planning 1-88, initial HW 02-87, test plan 03-88, order parts 3-88(3488 total) Sys test 5-88, test tools at 13 nodes 5-88, 3,488 parts at AA assm. Depot 05-88. NSS installs at 13 nodes 6-88 NSFNET goes live -7-88 New T1 network designed, developed, tested, deployed and cutover in 7.5 months. We have seldom seen a major networking project come off so smoothly and never one of such magnitude despite being new, complex, innovative line speeds an order of magnitude faster & heavy loads 8-88 Letter from FARNet to E. Aupperle
T1 NSFNET Key Reviews & Dates (cont.) 9-88 Interop NSS connects show net to NSFNET. 1 st. First demo of packet video Upgrade to full T1 & NSF Program review rates the performance of the network and partnership Outstanding 7-89. CA*net(the Canadian national backbone largely patterened on the NSFNET comes on line 10-90 NSFNET named Foundation for NREN - High Performance Supercomputing Act 12-91 Upgrade to all Smart cards & full T3 & FDDI 5-92 NSFNET decommissioned 05-95 An amazing set of experiences & important example of what can be accomplished with cooperation & collaboration between Academia, Gov t and Industry focused on overlapping goals that allowed everyone access & services of the global network.
The Nodal Switching System Rick Boivie November 29, 2007
June, 1987 It is anticipated that over the next 5 years, NSFNET will reach more than 10,000 mathematicians, scientists and engineers at 200 or more campuses and other research centers. From the 1987 NSF Project Solicitation
T1 NSS (1.5 Mb/sec links) IBM RT/PC s w AOS 4.3 (BSD 4.3) Token rings T1 DSU/CSU PSP RCP T1 DSU/CSU PSP T1 DSU/CSU PSP E-PSP Ethernet Flexibility, Scalability, Handled 20%/month traffic growth
T1 NSFNET 1988-1990 Seattle, WA Ann Arbor, MI Ithaca, NY Palo Alto, CA Salt Lake City, UT Boulder, CO Lincoln, NE Pittsburgh, PA Champaign, IL Princeton, NJ College Park, MD San Diego, CA Atlanta, GA Houston, TX We have seldom seen a major networking project come off so smoothly and never one of such magnitude -- FARnet, 1988
EASInet & CAnet NSFNET routers were also used in EASInet (Europe) and CA*Net (Canada) En Fevrier 1990, la premiere liason Internet transatlantique fonctionnant a un debit de 1,5 million de bits par seconde etait mise en service entre le CERN et l'universite de Cornell sur la cote est des Etats-Unis. -- Histoire Europeenne du WEB
T3 NSS (45 Mb/sec links) IBM RS/6000, AIX, Deep Adapters Microchannel T3 DSU/CSU RS/960 CPU T3 DSU/CSU RS/960 T3 DSU/CSU RS/960 RS/960 FDDI Disruptions on one link or route should not impact - Autonomous network adapters unaffected routes - Direct card-to-card transfers - New systems, new chips, new adapters, same architecture -Able to handle the 50,000,000 user, 50,000 net Internet of 1995 (far exceeeding NSF s original goals)
T3 NSFNET 1991-1995 Seattle, WA Palo Alto, CA Salt Lake City, UT Boulder, CO Argonne, IL Lincoln, NE Ann Arbor, MI Ithaca, NY Pittsburgh, Urbana PA Champaign, IL Cambridge, MA Princeton, NJ College Park, MD San Diego, CA Atlanta, GA 50,000,000 users! 50,000 CIDR networks! 93 countries! Far exceeding original requirements Houston, TX IBM did much of the engineering for NSFNET and performed superbly in that activity. We have thought deeply about whether there is another company which we would trust to do this very important job, and we have come up empty. - Back on Track to the NII?, Directors of WestNet, 1996
De Tocqueville on Communications & Communications in Michigan I know but one single means of increasing the prosperity of a people that is infallible in practice and that I believe one can count on in all countries as in all spots. This means is naught else but to increase the ease of communications between men... America, which is the country enjoying the greatest sum of prosperity ever yet accorded a nation, is also the country which, proportional to its age and means, has made the greatest efforts to procure the easy communications I was speaking of. In the Michigan forests there is not a cabin so isolated, not a valley so wild, that it does not receive letters and newspapers at least once a week; we saw it ourselves. -- Democracy in America, Alexis de Tocqueville, 1835
A More Recent Quote It s clear that the Internet is nothing less than the single most powerful tool we have ever seen for driving business, economic and societal change. Lou Gerstner, former Chairman of IBM
NSFNET Intelligent Network Adapters and Reflections Paul Bosco November 29, 2007
Reflections on NSFNET Public/Private Partnering NSF/University/Corporate Teamwork Regional/National/International Teamwork Time to Scale/Market Acceleration Research & Development Teams Operational & Management Teams Innovation/Standards Leadership Innovation (IP Routing, Systems,...) Standards (IETF Process, ) Technology/People Transfer Vendors (Cisco, Juniper, ) Operators (AOL, Comcast, )
Reflecting on NSFNET 1. Extraordinary Growth Could we possibly need memory for 500 routes someday? Phone system traffic grows at 8%/year, let s plan for 20%/year 2. Application Changes From Remote Access (ex: Super-Computer) To Collaboration (ex: Email & ftp) 3. Standards Challenges IP vs OSI & ATM 4. Foundation for Success The Web IP Everywhere
NSFNET Inter-domain routing Yakov Rekhter
In the beginning (1989) January 1989, 12 th IETF TNP ( three napkins protocol ) Produced over lunch by Y. Rekhter (at that time with IBM Research) and K. Lougheed (Cisco) Spring 1989 two interoperable implementations NSFNET/IBM Cisco June 1989 RFC1105 A Border Gateway Protocol (BGP)
In the beginning (1989)
In the beginning BGP design goals Overcome limitations of EGP-2: eliminate restriction on inter-domain topology to be spanning tree (with ARPANET as a root) eliminate problems caused by IP fragmentation of EGP-2 updates Support few thousand classful IPv4 routes Replace EGP-2 in the NSFNET Backbone Was positioned as a short-term solution, to be (eventually) replaced by a long-term solution
18 years later (2007) Four versions (and numerous extensions) later BGP remains the sole inter-domain routing protocol in the Internet Supports Classless Inter-Domain Routing (CIDR) Carries ~240,000 IPv4 routes (as of November 2007) Supports IPv6 inter-domain routing BGP usage extends well beyond inter-domain routing in the Internet: BGP/MPLS VPNs (aka 2547 VPNs), BGP-based VPLS, etc BGP is a widely successful protocol, as it far exceeds its original goals both in terms of scale (being deployed on a scale much greater than originally envisaged), and in terms of purpose (being used in scenarios far beyond the initial design).
Lessons learned Short-term solutions tend to stay for a long time; long-term solutions tend to never happen Good Enough solutions are sufficient; Perfect solutions may not be necessary Focus on solving practical problems in real time emphasis on engineering Meet market needs and accommodate technical progress by focusing on flexibility and extendibility Tightly couple evolution and further development with the operational experience and feedback from the service providers