Value of NERO and Commercial Alternatives

Transcription

1 Value of NERO and Commercial Alternatives A report to: Curt Pederson Vice Provost for Information Services Oregon State University And Shay Dakan Director of Network Services Oregon State University December 16, 2004 By Jon Dolan Associate Director of Network Services Oregon State University

2 EXCUTIVE SUMMARY OSU gets redundant and highly reliable connections to the Internet, Internet2, OSU Cascades, Extension Offices and Experiment Stations, DAS (State Agencies), OPEN (K-12), Some City and Counties in Oregon, and the other OUS institutions from NERO currently based on a consortium membership with little or no administrative overhead. These connections foster collaboration in-state and globally and mean that faculty, staff, and students can share electronic information very easily between connected partners. Replacement of the Internet, Internet2, OSU Cascades, and Extension Offices and Experiment Stations through commercial providers will cost at least 3 times what we current spend on NERO because we will not be sharing resources or cost. Strategic investments OSU has already made in regional dark fiber will have limited potential return without leveraging similar regional dark fiber owned by the local municipalities and run by NERO. Other OUS institutions benefit similarly from NERO and have told us through the OUS ETS assessment that it is their most important partnership. Problems do exist in terms of assurances that each partner does get what they pay for, and by extension questions arise over who controls the scope of membership and contributions. Our conclusion is that an oversight board, accountable to the Chancellor and members, should be established to first sort out the current finances and then be left in place to govern NERO in a fair and democratic way. We do think that an assessment of the situation will make it clear that some administrative overhead is required.

3 BACKGROUND OSU s initial connection to the Internet was through a consortium of NorthWest schools who banded together under the name of NorthWest Net, to first connect regionally, then nationally, and eventually internationally. This was in fact the way that nearly all of the Internet was run prior to 1995, when the Clinton Administration moved the management and administration of the Internet from the National Science Foundation to private interests. After the privatization of the Internet in general, NorthWest Net agreed to disband and to sell assets to private interests. Between 1995 and late 1996, OSU connected to the Internet via a commercial Internet Service Provider, MCI. In late 1996 OSU joined the Network for Research and Education in Oregon (NERO). This network had originally been built as a test bed in the Willamette Valley for high end networking. With the fundamentals in place it made sense to use NERO to connect the larger higher education institutions in Oregon together to leverage an economy of scale to bulk purchase Internet bandwidth. OSU saw an immediate cost savings, as did the other institutions, and then gained a new benefit in being able to connect to the other schools directly. The Oregon University System (OUS) at the time, had been paying for centralized Information Technology for the regional Universities who could not afford large scale IT systems on their own. As part of providing these services, OUS also interconnected those institutions and ran a state wide data network. This network was merged into the NERO backbone to aggregate circuits, and to share bandwidth and costs. Again, by banding together all institutions were able to buy and share larger data circuits at a lower per megabit per second (Mbps) cost. The consortium then agreed to first contribute the costs they were already incurring for in-state data circuits, and then share the Internet costs on a usage basis. Financial contributions to the consortium were then based on a Mbps cap of Internet traffic and members could have unlimited bandwidth to each other provided the capacity existed with the aggregated circuits we each contributed. The economies of this circuit aggregation made sense for more than just the higher education institutions. The more consortium members involved, the lower the per megabit cost each of us had to contribute. By sharing bandwidth and costs everyone got a better deal, so no one objected when the Department of Administrative Services (DAS) and the Oregon Public Education Network OPEN) also joined the consortium. As long as all members were public institutions, the more the merrier (and cheaper). Using this collective buying power, NERO has been able to steadily drop the Mbps membership rate which has allowed members to either increase Internet bandwidth or realize cost savings. CURRENT NERO CONNECTIONS FOR OSU OSU currently gets its commodity Internet bandwidth (capped at 65Mbps,) a Gigabit per second (Gbps) circuit to the Oregon Internet2 Gigapop in Eugene, a shared 100Mbps circuit to our branch campus in Bend (OSU Cascades,) a shared 45Mbps circuit for our ~50 statewide Extension Offices and Experiment Stations, various shared

4 connections to the other OUS institutions, and various shared connections to state and local government agencies. To provision all of this, NERO maintains a network point of presence (POP) on campus at OSU in Corvallis. NERO s POP currently is connected to POPs in Eugene (at the UofO), Salem (in a DAS facility,) and Portland (in the Pittock carrier hotel.) Those connections are made by connecting network equipment to fiber optic cables owned in part by OSU, in part by UofO, and in part by the municipalities along the I5 corridor who worked with the Lane Council of Governments (LCOG) to obtain fiber from LEVEL 3 communications in exchange for rights of way. In the spirit of the original consortium, each of these pieces have been provided by the respective owners to be used by NERO for the good of all. Using existing assets that alone have limited usefulness, NERO is able to provide a regional dark fiber network which can be leveraged by all consortium members and considerably lowers the operating costs for all. This also allows for considerable future growth with only modest investments by members. In addition to the circuits provided over the fiber infrastructure, NERO s Corvallis POP also has redundant circuits going north to Portland and South to Eugene for reliability. In fact the core NERO network has been exceptionally well designed with redundancy and reliability as non-negotiable prerequisites. Given our extreme dependence on being connected to the outside world electronically, this is a requirement for the day-to-day business of OSU and the rest of the consortium members. OSUs extension offices connect to the nearest NERO POP, saving money in distance charges on circuits and using the existing NERO network backbone to get back to OSU. Similarly, we have joined forces with Central Oregon Community College (COCC) to get OSU Cascades in Bend connected, along with COCC, to the NERO network backbone. The collaboration required with the main campus in Corvallis would not be possible with out these high speed connections, and these connections would not be financially feasible without sharing costs within the NERO consortium. In fact the ease of connecting within the state is now assumed and numerous collaborations happen without even our specific knowledge that would otherwise be impossible or at the very least problematic. For example, OSU s public safety department had been connecting to the Law Enforcement Database system in Salem by a dedicated circuit and gateway server. This was done as a separate connection under the assumption that it would be the only way to meet the required uptime for any system with health and safety implications. This separate connection was plagued by problems and our public safety department tried simply connecting via our standard NERO connection. The result was that since the change there have been no connection problems, and the separate connection has been retired. This is a cost savings, but that benefit is actually a side benefit. The requirement that the connection be absolutely reliable came first and foremost.

5 COST OF COMMERCIAL ALTERNATIVES We have looked at what it would cost us to replace these connections with circuits provisioned commercially. Given our partnership with NoaNet Oregon, we would be able to replace much of what we currently get from NERO but it would come at a significant cost. If we were to buy commodity Internet bandwidth, a 100mb/s connection to Bend, and a dedicated Gbps connection to Eugene for Internet2 traffic from NoaNet we could round out a few other circuits from Qwest that would give us at least all of the basic connections required. In addition we would likely need to hire an additional Network Engineer. Those annual costs would look something like this: Commodity 100Mbps from NoaNet: $102,420 *Carrier diverse redundant circuit from Qwest: $ 72,000 Dedicated Gbps for I2 from NoaNet: $105,852 *Extension tail circuit from Qwest: $ 36,000 *100Mbps connection to Bend from NoaNet: $ 36,000 *Additional FTE plus OPE $ 80,000 *Estimated costs. Total Estimated Annually: $432,272 FY04 NERO dues (not counting value of fiber) $156,750 Using commercial providers to approximate the services that we currently receive from NERO would increase our costs nearly threefold. This does not take into account the lost opportunities to use our regional fiber to increase our connections as needed and to seamlessly connect to the other NERO partners statewide. It is very hard to put a dollar value on those lost opportunities, but they would likely far outweigh the tangible ones presented above. It is reasonable to ask why NERO can be so much cheaper. The answer to that question is in the sharing of costs and resources. I described above an arrangement wherein members of the consortium had various segments of dark fiber at their disposal and were able to combine them for the good of all. Those sort of collaborations and contributions of resources constitute the real value proposition for working collectively within the NERO consortium and is where these cost savings come from. NERO BENEFITS FOR REGIONALS It is worth noting that it is not just the large users of bandwidth and data connections who benefit from NERO and realize these sort of cost savings. For example, as part of our ETS assessment, Jason McKerr interviewed each of the regional IT Directors. One of the first questions he asked was What are your most important partnerships? He reported that in every single case, without exception, the first response (and sometimes only response) was the service that NERO offers. For the regional

6 Universities, connecting at comparable speeds to what they are doing now would simply not be possible without NERO and would severely negatively impact their operations. An example of this is Eastern Oregon University. I believe Eastern has as many as one third of their entire education base on distance/electronic education. As the tools to provide this education come to rely more and more heavily on services on the Internet, they could loose a significant student base from inadequate Internet connectivity. I m sure the other institutions would have similar problems. IS THERE A PROBLEM WITH NERO? While it is unquestionably clear that NERO is a very good thing technically for all involved, there is a growing sense that it has grown beyond its structure. First, there are several new members such as Oregon Public Broadcasting, the Oregon Symphony, Oregon Community Health Information Network, and others whose membership does not seem to fit the original scope of NERO/OWEN and who s membership was not subject to any sort of broad based review. Coming from the more the merrier perspective, their membership may be entirely appropriate but when questions of adequate staffing and funding arise the issue of who exactly pays for what and who might be getting more than they pay for is inevitable. Second, it does seem that NERO is woefully understaffed. We are asking for an awful lot from 2 people to run a network of this size. While the technical details are being attended to, long range visioning, strategic planning, adequate financial reporting, and information sharing in general are taking a back seat if they are being addressed at all. The current staff has done a phenomenal job of providing a very high level of service, but additional staffing seems warranted. Third, the lack of formal written agreements is setting us up for a crisis when the current employees move-on. The details of most of the arrangements are not written down and are known only to the Director of Engineering as far as we know. We have been able to get an incredible amount done on a handshake and mutual trust, however, at some point someone other than the Director of Engineering will need to know who agreed to what, and it will need to be in writing. Case in point, with the current turn over at the chancellor s office and subsequent review of budgets, some $500,000 that NERO was counting on from OUS did not appear to be in the current budget. Given that there is no paper trail on what was committed to and why, a crisis developed over how to fill the budget gap. This would not be an issue if we had agreements in writing. RECOMMENDATIONS First and foremost there needs to be clear accountability. We would like to see regular financial reports, a formal process for evaluating new members, a review of the mission and scope of NERO, a review of the staffing levels and organizational structure of NERO, and of course we need NERO to stay financially solvent so the current budget gap needs to be dealt with quickly. To accomplish this we recommend that a governance

7 board be established that is comprised of the CIOs from PSU, UofO, and OSU. This board should be charged with the following: 1. Prepare an accurate financial picture of NERO. Who pays what, for what. 2. Clearly define the scope and mission of NERO. 3. Organize NERO s administration such that regular financial reports are readily available and current services provided to whom and why are clearly articulated. We have clearly accomplished an amazing amount with relatively little resources on the basic premise that we all should work together for the common good. David Crowe in particular is to be commended for virtually single handedly building this network and growing it to where it is today. Like all organizations, however, there does come a time when growth and maturity must be reconciled. NERO needs to mature to continue to be successful and some more formal structures and agreements will accomplish that.