Perry T. Eidson Technical Operations Communications Engineer Emory University Technical Services

Transcription

1 Perry T. Eidson Technical Operations Communications Engineer Emory University Technical Services Internet2 Fall 2008 Member Meeting October 14, 2008

2 Founded in 1836 at Oxford, Georgia by the Methodist Episcopal Church Expanded to Atlanta Campus in 1915 Through Generosity of Asa Candler, Founder of Coca-Cola 2008 Enrollment 12,755 Ranked 18 th 2008 U.S. News and World Report Overall Individual Schools May Rank Higher

3 Nine Schools Emory College 1836 Oxford College 1836 School of Medicine 1854 School of Nursing 1905 Candler School of Theology 1914 School of Law 1916 Roberto C. Goizueta Business School 1919 Graduate School 1919 Rollins School of Public Health 1990 Affiliates Include Carter Center and Yerkes National Primate Research Center

4 Total Employees, Emory and Emory Healthcare 23,733 (largest private employer in Atlanta area) 2008 Combined Budget - $3.0 billion Endowment as of August 31, $5.6 billion (17 th among U.S. universities) $411.2 million in sponsored research for fiscal 2008

5 Emory Healthcare Largest Health Care Provider in Georgia 4 Owned Hospitals With Major Presence at Grady Memorial Hospital (Fulton- DeKalb Hospital Authority), VA Medical Center Atlanta and Children s Healthcare of Atlanta-Egleston Major and Minor Emory Clinic Locations Across City - Expanding Across State

6 Major Data Center Supporting Academic and Healthcare Needs IBM z/890 Mainframe Running z/os IBM P590 Upgrading to P595 AIX Multiprocessor Partitionable Machines 6 Egenera Bladeframes HP C-class Blade Servers VAX/VMS Machines 750 Terrabyte SAN Network Growing to 1+ Petabyte Cytrix Environment for Healthcare Numerous Sun and Windows Servers

7

8 Several Schools and Affiliates Have Their Own Networks/Servers That Connect to University Backbone Dual 10G Backbone Cube Core Topology 44k+ data ports 32K+ voice ports Virtual Core Networks and Routers Provisioned Through Large Routers/ Firewalls

9 Metro WAN Megabits to Major Locations Mix of T-1, 10 Meg NMLI, 100 Meg NMLI Additional Locations Not Included in Fiber Project Due to Minimal Requirements or Excessive Distance Not Cost Effective Mixed Vendors, Routers Everywhere Evolved From Previous 5 Legacy Organizations Separate Billing/Contracts Availability, Cost, Latency Issues

10 Network Communications Created in 2000 Combining 5 Separate Entities University Telecom University Network Operations Emory Clinic Telecom Hospital Telecom Healthcare IS Networking Combined Entity Had Size, Scope, Expertise and Financial Wherewithal to Tackle DWDM/Fiber Project

11 2001 Began Evaluation of Equipment and Fiber Vendors 2002 Tested Systems in Lab, Identified Dark Fiber Vendor 2002 Awarded Contracts Multi-month Fiber Construction 2003 Fiber Complete/Equipment Installed First Circuits Cut-over March 2003

12 First Cut-over Due to NMLI Failures Added 2 Hops per Leg to Enable Cut to 1 Gig DWDM Circuit Prior to Core Upgrade From 100 Meg to 1 Gig Latency Dropped Enough So End User Customers Called NetCom to Ask What Changed Much Better Terminal Response Visible!

13 First Design Called for Three Separate Rings Traffic Separated by Function Core Traffic HIPPA and FERPA Regulations Border Traffic Access to Outside Entities Rings Took Separate Paths SONET Ring for TDM Links Final Design All Rings Took Same Path and Met SOX at Telecom Hotel (56 Marietta)

14 Main Campus Briarcliff Campus North* Cox Hall CMONT* NDB ECLH *Not on DWDM Rings Campus Fiber 56 Marietta Grady

15 At Cutover, Amortized Monthly Fiber Ring Cost Less Than Cost of Leased Circuits Bandwidth Increased From 527 Meg to 17 Gig Latency Decrease and Bandwidth Increase Resulted In Remote Locations Moving From WAN to LAN Environment

16 Moved From Routers at Most Locations to Central Routers and Switches Everywhere Else Moved From Physically Separate Cores to Virtual Cores Utilizing 10 G Duplicated Physical Pipes Large Core Routers Providing VRF Moved From Multiple Physical Firewalls to Virtual Firewalls on Large Platforms

17

18

19 Address Power, Space and Cooling Constraints in Primary Data Center Issues With Off Site Tape Backups Extended Period of Time to Recover Support of Healthcare and Research Require Immediate Recovery Timeliness of Data Backups

20 Multi-year Effort to Identify Appropriate Location Location Relative to Main Campus Cost Network Availability Multi-Phase Project Secure Site and Mirrored Critical Data Create Business Continuity Infrastructure Hot-Standby Modify Apps For Parallel Use

21 White ST Main Campus

22 Former A&P Meat Packing Warehouse Bunker Like Construction Dual Commercial Power Feeds N+1 Diesel Generator Sets For Backup Power Redundant Cooling With Well as Primary Source and City Water as Backup Dual Fiber Entrance

23 Multiple Air Conditioning Units Multiple Power Distribution Units Fiber Paths to Leased Space Kept Separate by at Least 8 Feet Separate DC Plants in Emory Leased Space Separate DWDM Nodes 1 Each on Core and Border Rings Dual Master Switches Cross Linked via Fast Spanning Tree for Immediate Recovery in Event of Link Failure

24 Core Fiber Ring DC-A DC-B DWDM Core DWDM Border AC-DC Data Switch AC-DC Data Switch Storage Arrays Fiber Channel Switch Border Fiber Ring

25 Current Network 30 Gig to SDC Dual 10G IP Links From Separate Core Routers Separate Rings Opposite Directions A & B DC Feeds From Separate DC Plants Dual 4G Fiber Channel Transparent Transponders Separate Rings Opposite Directions Bonded Into Single 8G Link Seamless Fallback/Recovery If One Side Impaired Private 1G Link for PACS System Special Need for Flat Non-Routed Network Spare 1G circuit

26 IP Latency Well Under 1 Millisecond Fiber Channel Latency Microseconds Calculated Latency Based on Speed of Light Actual Response Slightly Slower But Only Just Slightly SRDF Read/Writes Look Same at Remote Center 5-10 Minute Job Less Than 5-10 msecs Difference

27 150 Terabytes Storage Currently Online 130 Terabytes IP Based 20 Terabytes Fiber Channel PACS System Active-Active Between Campus Data Center and SDC Sun Hardware Multiple Servers 40 Terabytes in Each Location

28 Near Future (end of year) EMC DMX4 at Campus Current DMX3s Moved to SDC P595 Upgrade for Campus Current P590 Moved to SDC Integration to Storage Grid Complete 2 Locations on Campus for Storage Plus SDC Schools Independent IT Departments Using Storage Facilities

29 Major Outpatient Clinic New Hospital Link to Oxford Campus 50 Mile Link From 56 Marietta Telecom Hotel Internet Links 10 G to Internet2 1 G Link to AIX Public Peering Point 2 1 G Links to Separate Commercial Internet Providers

30 59 Exec Park Cox Hall NDB

31 59 Exec Park Northlake Cox Hall NDB

32 Combination of DWDM and Campus Fiber Enables CDP DWDM Bandwidth and Latency Brings Remote Locations On Campus Criticality of Data Sets Multi-Generational Studies Both Human and Animal Irreplaceable Predictive Healthcare Initiative Data Intensive

33 DWDM Growth Has Been Explosive 17 Gig at Install 32X Displaced Circuits 114 Gig Currently 6X Installed Capacity 216X 2003 MAN Capacity

34