Paul Schopis Thomas Skill Bruce Burton Roger Bielefeld OARnet/Ohio State University University of Dayton University of CincinnaA Case Western Reserve Univ pschopis@oar.net tskill1@udayton.edu BURTONBE@UCMAIL.UC.EDU roger.bielefeld@case.edu Advanced Networking: Regional Collaboration Through CC- NIE/IIE Awards Sparks Partnerships & Success - The Ohio Experience Internet2 Technology Exchange 2015 Cleveland, OH October 5
Paul Schopis Background OARnet is a project of the Ohio Department of Higher EducaAon, created by the Ohio Legislature and uses The Ohio State University as the program s fiscal agent. It is a consoraa organizaaon comprising of 90 insatuaons of higher educaaon, members include Aer 1 research insatuaons, state universiaes, small privates colleges and community colleges In recent years membership has expanded to k- 12, Public BroadcasAng and government State Government has recognized potenaal and invested in 100G architecture Science Drivers in the region Ohio has huge medical research faciliaes; significant aerospace and materials research efforts OARnet/OSU has had two NSF projects that have helped drive regional parecipaeon and success
CC- NIE InnovaEons to TransiEon a Campus Core Cyberinfrastructure to Serve Diverse and Emerging Researcher Needs Science DMZ construcaon with advanced technologies 100Gbps connecavity, perfsonar, OpenFlow, RoCE/iWARP, Bro Connect OSU at 100Gbps to OARnet- Internet2 peered network Aggregate research VLANs arising from internal OSU departments Define and establish role of a Performance Engineer on Campus App development to help operaaons, policy development, funding model
CC- NIE Wide- area experimentaaon case studies with Co- PIs OSU MU experiments: Brain imaging, Soybean translaaonal genomics OSC experiments: adopaon of cloud- based technologies for big- data import, storage and collaboraaon, as well as related analyacs OSU HFCMPL experiments: foster mula- physics research collaboraaon and high- resoluaon simulaaon steering; graduate capstone project Outreach and support for Co- PI experiments in Year 1 other campus researchers in Year 2
OSU Science DMZ (Logical Diagram) 5
Equipment Purchase and Locations 100 Gbps border router connect to OARnet- Internet2 peered network Cisco Nexus 7009-100Gpbs Carries all R&E traffic and DMZ OpenFlow switch configure VLANs to remote sites (e.g., MU, GENI) NEC PF5820F- 64XW OpenFlow controllers Open Daylight - POX perfsonar measurement points collect end- to- end performance metrics One at border next to OpenFlow switch; instrument at 2 or 3 depts. Data transfer nodes wide- area RDMA- based, GridFTP integrated technologies Pinnacle 2X2601H8 Policy- directory server enforces researchers project- specific policies Dell R710 Containers running Policy, controllers and personar Bro Cluster invesagate the tradeoffs to be balanced between researcher flow performance and campus security pracaces Arista 7150S switch Juniper EX3200 24 port 10/100/1000BaseT Myricom Myri- 10G- network adapter Silicon Mechanics Rackform iserv R304.v3 "10Gb Bro IDS Cluster Worker Sniffer10G 9v.2.0) 6
Personnel Hiring and Collaboration Logistics Performance Engineer: Keeper and Help- desk for Science DMZ Reports to Paul Schopis; Interacts closely with John Heimester, Jay Young and Steve Romig LocaAon: OSC/OARnet Graduate Student - 1: RoCE/iWARP experiments Reports to: Dhabaleswar Panda; Interacts closely with Prasad Calyam LocaAon: OSU CSE Graduate Student - 2: Imaging Analysis experiments Reports to: Umit Catalyurek; Interacts closely with Prasad Calyam LocaAon: OSU BMI Graduate Student - 3: OpenFlow experiments Reports to: Dhabaleswar Panda LocaAon: OSC/OARnet 7
OSU- MU Experiments Common testbed setup tasks FederaAon of OSU and MU Science DMZs user accounts/roles perfsonar instrumentaaon and measurement Establishment of VLAN extensions and IPv6 Set up opamized end- to- end data transfers with RoCE and iwarp Brain imaging analysis MU Brain Explorer experiments with OSU for data- intensive brain imaging modeling, analysis and visualizaaon funcaonaliaes Data Cuser at OSU BMI to be leveraged to improve computaaon speed Soybean translaaonal genomics and breeding (Scalability experiments) MU Soybean KB database experiments with OSU for set up of GENI slices to dynamically change user load paserns from remote campuses Service response Ame analysis of distributed databases, web- services for remote user access scalability 8
The Curious Case of CEMAS Center for Electron Microscopy and AnalysiS Project with AFRL and AFIT at WPAFB Renderer needs L2 and 1 Microsecond PDV WPAFB needs to do connecavity thru OARnet/DREN peering at L3 Requirements: Must buffer and alter Amestamps at receiver Must encapsulate in IP at sender, pass through security inspecaon and decapsulate before receiver POC completed 9
CC*IIE NSF Project Title: Transforming a Regional Network and the Regional Community to Serve Diverse and Emerging Research Needs Three Key Goals #1 Host a series of workshops focusing on enabling Science and Science DMZ, workshop series include perfsonar, OperaAng InnovaAon Networks and a workshop creaang OARperf, a regional forum to support advanced networking efforts #2 Deploy perfsonar widely throughout OARnet at both backbone and campus locaaons #3 Build a network of a CoaliAon of the Willing to support ongoing community support of one another; OARperf First goals have been met and as second year gets under way we are working on goal three
Leveraging the Investment Building on success of OSU s CC*NIE infrastructure grant Shared knowledge with others in community Wrote lesers of support Used posieon of regional network to act as convener and advocate for members
Thomas Skill CC*NIE - NSF Project Title: Network 10Gb Upgrade & Science DMZ ImplementaDon to Support Science & Engineering Research and Enhance Outreach for High School STEM EducaDon Three Keys to our Success: Diverse Team of Research Faculty as Co- PIs with Great Science Stories Compelling stories on how we will leverage the technology Outstanding CollaboraDon Partners Beyond Campus Community College (Sinclair), HBCU University (Central St), Major Research University (OSU), and Ohio Super Computer Center. OARnet as our state- wide technology deployment partner Excellent Technology Plan with Strong Team to Execute Catalyst for our High Performance Network & Research CompuDng Strategy Moved us to full Internet2 member status in 2013
Support world-class materials characterization research by enabling remote access and manipulation of instrumentation located in Columbus, Ohio Collaboration Partner: Goal: Efficiently and productively share highly scarce scientific instruments via an ultra high speed/low latency network connection between UDayton and OSU. Dr. Charlie Browning, Materials Engineering 13
Cross-institutional Nanotechnology STEM Education and Workforce Training Collaboration Partner: Dayton, Ohio Goal: Bring high-end laboratory environments to high school and college students via alternative delivery methods that provide interactive nanotechnology laboratory experiences. Dr. Andrew Sarangan, Electro-Optics Program 14
Investigating patterns of precipitation change as a result of global climate change, and projecting future precipitation changes Collaboration Partner: Goal: Rapidly and accurately move massive climate study data to the Ohio Supercomputer Center in Columbus Ohio from the University of Dayton Campus for remote processing and analysis. Dr. Shuang-ye Wu, Geology 15
Research on glacier melting processes, hydrological modeling and earthquake induced landslides. Collaboration Partner: Goal: Analyze multi-sensor and multi-temporal satellite data to study glacier dynamics and supraglacial lake development. Also, map high altitude glacierized basins for earthquake-induced landslides and river blockages a humanitarian issues. ASTER satellite images showing the development of Tasman Glacier lake in New Zealand and acceleration in the terminus velocity from (a) 29 Jan., 2002, (b) 31 Dec., 2002, (c) displacement map from (a) and (b). White dashed line shows centerline distance downglacier from the Rudolph con[luence. Dr. Umesh Haritashya, Geology 2015 Nepal earthquake induced landslide blocking Tom Khola River 16
Research on Human Centric Intelligence, Surveillance & Reconnaissance (ISR) Sensors. Collaboration Partner: Wilberforce, Ohio Goal: Develop signatures for human activity recognition and threat detection by analyzing data captured by multiple sensors that will distinguish normal and abnormal gait behaviors caused by additional weight placed on the body. Dr. Vijay Asari, Electrical Engineering 17
Technology Strategy
Collaboration Bene[its Key Benefits Science Strengthening collaboraaon among the research community throughout the state Leveraging use/availability of scarce research instruments via remote access Robust access to senior researchers in support of High School and Community College STEM EducaAon Enhancing Public- Private Graduate & Undergraduate EducaAon and Training Key Benefits Technology CollaboraAon with OARnet and Ohio- based UniversiAes speeds adopaon of advanced networking via: Centrally- shared experase on implementaaon of Science DMZ circuits & perfsonar monitoring Shared training & skill building via pooled resources & centralized support/coordinaaon Community outreach to other state- wide insatuaons in sharing new experase on high performance networking
Bruce Burton NSF Project Title: UCScienceNet (UCSN) A High Bandwidth Science DMZ to Enable STEM Discovery, CollaboraDon, and EducaDon STEM Drivers: Remote Sensing and Geographic InformaAon Science NSF NaAonal Snow and Ice Data Center, NASA, JPL, Alaska SAR Facility. Center for Intelligent Maintenance Systems (IMS) University of Michigan, Missouri University of Science and Technology, University of Texas at AusAn. ComputaAonal Fluid Dynamics Ohio SuperCompuAng Center, ORNL, P&G, XSEDE Supercomputers. High Energy Physics CERN, Fermilab, Pacific Northwest NaAonal Laboratory Project Goals 100 Gbps to OARnet/Internet2 Science DMZ, perfsonar, Research CollaboraAon 10 Gbps connecavity to five research- intensive locaaons
UCScienceNet
Key Bene[its Science, Technology, and RelaEonships Enabling naaonal and internaaonal STEM collaboraaon. Increasing research opportuniaes with large data transfers. Expanding knowledge and skill sets (SDN, HPC, ScienceDMZ) Strengthen relaaonships with OARnet and other Ohio colleagues ExtraProject CollaboraEon Business ConAnuity hosang at SOCC Campus CyberInfrastructure Engineer CC*DNI proposal OARperf
If You Build It.. Building on UCScienceNet CC*IIE infrastructure grant Partnership with UC Office of Research 250K Developed a dialogue with Ohio SuperComputer Center Establish a shared vision and long term menu of services for High Performance CompuAng (HPC) resources for researchers.
Roger Bielefeld - CWRU Case Western Reserve University Cleveland, Ohio Private research university (Carnegie: Very High Research AcEvity) Case InsEtute of Technology (established 1877) Western Reserve University (established 1826) Only Ohio school with winning football record versus Ohio State! Case Western Reserve University (federaeon 1967) Applied Social Science, Arts and Sciences, Dental Medicine, Engineering, Graduate Studies, Law, Management, Medicine, Nursing Enrollment: 10771 (Undergraduate: 4911, Graduate and Professional: 5860) Faculty: 3149 full- Eme Staff: 3113 full- and part- Eme
CWRU Research External research funding: $311.4 million in 2014 59% ($182.9 million): Department of Health and Human Services 30% ($92.3 million): non- federal 5% ($16.2 million): NaEonal Science FoundaEon 7% ($20 million): Departments of Defense, Energy, and other federal sources Cleveland Clinic Lerner College of Medicine of Case Western Reserve University $90 million in research funding
CWRU CC- NIE NSF Project Title: Science DMZ and Research Network Upgrade for CWRU Award 1340938 Project start: 1 October 2013 PI: Bielefeld, Co- PIs: Gonick, Risen Science Drivers Experimental Physics: large underground xenon experiment with Sanford Underground Research Facility (South Dakota) and 15 universiaes in the U.S., U.K. and Europe Dr. Daniel Akerib, Professor of Physics, currently at Stanford Linear Accelerator Center Dr. Thomas Shus, Professor of Physics, currently at Stanford Linear Accelerator Center Membrane and Structural Biology: projects in cryo- electron microscopy and molecular dynamics small angle X- ray scasering protein footprinang mass spectrometry Dr. Phoebe Stewart, Professor of Pharmacology, Director of the Center for Membrane and Structural Biology Imaging: advanced MRI applicaaons and real- Ame MRI with collaborators at NIH, Northwestern, and Siemens (Germany) Dr. Mark Griswold, Professor of Radiology
CWRU CC- NIE Project Goals NSF Project Title: Science DMZ and Research Network Upgrade for CWRU Project Goals Connect CWRU at 100 Gbps to OARnet- Internet2 peered network Completed in November 2013 First Ohio university connected to Internet2 at 100 Gbps Science DMZ Deployed in July 2014 IniAally one data transfer node, one perfsonar node Ongoing tweaking and expansion Campus network upgrades Five research- intensive campus buildings upgraded to 10 Gbps capability by February 2015 Network distribuaon hub for the engineering quad upgraded to 10 Gbps in April 2015 Leverages single- mode fiber in our network cable bundle
CWRU Network Upgrades NSF Project Title: Science DMZ and Research Network Upgrade for CWRU Building Upgrades Glennan Building Mechanical and Aerospace Engineering, Electrical Engineering and Computer Science, Materials Science and Engineering, Center for Surface Analysis of Materials White Building Materials Science and Engineering Olin Building Electrical Engineering and Computer Science, Mechanical and Aerospace Engineering, NaAonal Center for Space ExploraAon Research Rockefeller Building Physics Wood Building School of Medicine Bingham Building Civil Engineering, Mechanical and Aerospace Engineering, Cleveland FuncAonal Electrical SAmulaAon Center, Center for Micro and Nano Processing Network distribuaon hub for the enare engineering quad
CWRU Science DMZ Cancer Genomics Hub at UC Santa Cruz Globus Aspera (NCBI) GENI NAS deployment (future 1.8 PB) Server hoseng (future) 40 Gbps data transfer nodes 40 Gbps HPC network
NaAonal Science FoundaAon CC- NIE Award 1340938 (Bielefeld, PI).
Bene[its and Challenges Benefits Science and Technology Enabling naaonal and internaaonal collaboraaon (e.g. physics and imaging) Reducing network fricaon for co- located research centers (e.g. Center for Imaging Research and Center for Membrane and Structural Biology) AcceleraAng the move away from SneakerNet FacilitaAng very high volume movement of geneac/genomic data dbgap UCSC Cancer Genomics Hub Increased ability to leverage Ohio- based resources Ohio Supercomputer Center Deeper relaaonships with OARnet and other Ohio colleagues Poised to paracipate in Cleveland s 100 gigabit Health- Tech Corridor
Bene[its and Challenges Challenges Staff Resources Complexity in sewng up, maintaining, and expanding the Science DMZ Lack of network engineering staff to support the Science DMZ Outreach to research community for awareness and opamal ualizaaon CC*DNI Cyberinfrastructure Engineer proposal is pending
Plug Today at 9:00 AM in this room! Using Fixed ConfiguraEon Data Center Switches in the Campus Backbone Dan Majhews, CWRU Digging deeper into some of our technical choices for CC- NIE network upgrades that allowed us to do more than we promised!