globus online The Research Cloud Steve Tuecke Computation Institute University of Chicago and Argonne National Laboratory

Transcription

1 globus online The Research Cloud Steve Tuecke Computation Institute University of Chicago and Argonne National Laboratory

2 Cloud Layers Software-as-a-Service (SaaS) Platform-as-a-Service (PaaS) Infrastructure-as-a-Service (IaaS) 2

3 Big science has achieved big successes with advanced community services LIGO: 1 PB data in last science run, distributed worldwide OSG: 1.4M CPU- hours/day, >90 sites, >3000 users, >260 pubs in 2010 SubstanRal teams Sustained effort Leverage common technology ApplicaRon- specific solurons ProducRon focus ESG: 1.2 PB climate data delivered to 23,000 users; 600+ pubs Community services built on Globus Toolkit so4ware

4 But small and medium science is suffering Data deluge Ad- hoc solurons Inadequate so4ware, hardware & IT staff

5 Medium science: Dark Energy Survey Blanco 4m on Cerro Tololo Every night, they receive 100,000 files in Illinois They transmit files to Texas for analysis then move results back to Illinois and make them available to users Process must be reliable, routine, and efficient The cyberinfrastructure team is not large! Image credit: Roger Smith/NOAO/AURA/NSF

6 The challenge of staying competitive "Well, in our country," said Alice "you'd generally get to somewhere else if you run very fast for a long time, as we've been doing. "A slow sort of country!" said the Queen. "Now, here, you see, it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!"

7 Time-consuming Tasks in Research Run experiments Collect data Manage data Move data Acquire computers Analyze data Run simulations Compare experiment with simulation Search the literature 7 Communicate with colleagues Publish papers Find, configure, install relevant software Find, access, analyze relevant data Order supplies Write proposals Write reports

8 Why SaaS? Software-as-a-Service (SaaS) Platform-as-a-Service (PaaS) Deliver advanced functionality that: Requires no user software installation or operation Minimal IT proficiency required Can be cheaply and incrementally adopted Usage-based subscription pricing; no big up-front costs Consolidates troubleshooting and support An expert group can proactively detect and correct problems Utilizes an efficient software delivery lifecycle Updates developed, tested and deployed quickly Dominates commercial & consumer markets What about the research market? Infrastructure-as-a-Service (IaaS)

9 Time-consuming Tasks in Research Run experiments Collect data Manage data Move data Acquire computers Analyze data Run simulations Compare experiment with simulation Search the literature 9 Communicate with colleagues Publish papers Find, configure, install relevant software Find, access, analyze relevant data Order supplies Write proposals Write reports

10 What is Globus Online? Reliable file transfer. Easy fire-and-forget transfers Automatic fault recovery High performance Across multiple security domains No IT required. Software as a Service (SaaS) No client software installation New features automatically available Consolidated support & troubleshooting Works with existing GridFTP servers Globus Connect solves last mile problem >3500 registered users, >3PB moved Recommended by XSEDE, NERSC, Blue Waters, and many campuses 10

11 Case Study: Lattice QCD Fast: Reduced transfer times Easy: Fire-and-forget transfers Automated retry No file pre-staging No complex infrastructure Convenient CLI and GUI interfaces Indiana University researcher moved ~6 TB from Tennessee to Texas in 2 days Globus Online frees up my time to do more creative work than typing scp commands or devising scripts to initiate and monitor progress to move many files. I moved GB files tonight in about 1.5 hours. I am very impressed. I also like the new commands and help system. 11

12 What s Next? Run experiments Collect data Manage data Move data Acquire computers Analyze data Run simulations Compare experiment with simulation Search the literature 12 Communicate with colleagues Publish papers Find, configure, install relevant software Find, access, analyze relevant data Order supplies Write proposals Write reports

13 Globus Online Vision Goal: Accelerate discovery and innovation worldwide by providing research IT as a service Leverage software-as-a-service to: provide millions of researchers with unprecedented access to powerful tools reduce research IT costs dramatically via economies of scale Civilization advances by extending the number of important operations which we can perform without thinking of them Alfred North Whitehead,

14 Potential economies of scale Small laboratories PI, postdoc, technician, grad students Estimate 10,000 across US research community Average ill-spent/unmet need of 0.5 FTE/lab? + Medium-scale projects Multiple PIs, a few software engineers Estimate 1,000 across US research community Average ill-spent/unmet need of 3 FTE/project? = Total 8,000 FTE: at ~$100K/FTE => $800M/yr (If we could even find 8,000 skilled people) Plus computers, storage, opportunity costs,

15 PaaS for Research Software-as-a-Service (SaaS) Platform-as-a-Service (PaaS) No one SaaS provider can deliver it all Must create ecosystem that: Infrastructure-as-a-Service (IaaS) Allows any SaaS provider to easily participate Dramatically reduces the cost of creating and operating service within the ecosystem Provides seamless user experience across services Agnostic to / works across any cloud IaaS provider Sustainable Ecosystem requires Platform as a Service Target the unique needs of the research community 15

16 Globus Online PaaS Software-as-a-Service (SaaS) Platform-as-a-Service (PaaS) Platform as a Service (PaaS) that allows other applications to integrate into the Globus Online ecosystem Horizontal platform services Make it easier to create new services in ecosystem Currently identity, profile, and group management REST APIs to integrate with Globus Online vertical services Currently file transfer Others to be announced at Globus World (April) Globus Connect integrates resources 16 Infrastructure-as-a-Service (IaaS)

17 Future of IaaS in Research Software-as-a-Service (SaaS) Platform-as-a-Service (PaaS) Prediction Small- to mid-scale campus-based clusters will disappear over next decade Will be provided by public cloud IaaS providers, and by consolidated research cloud IaaS providers Why? Supercomputing centers will continue to exist, based on specialized exascale computing architectures Researchers will increasingly use SaaS SaaS will make use of cloud IaaS easy Economies of scale will make IaaS cost effective Mobility will make location unimportant 17 Infrastructure-as-a-Service (IaaS)

18 Virtual private laboratories Provide as an outsourced service the hardware, so,ware, and processes required for successful research Secure, efficient, and cost- effecrve 18

19 Virtual private laboratories Research challenges IdenRficaRon of processes Design of key building blocks Scalability to massive data Human computer interface Security IntegraRon of collaboraron Economic models Success metrics Accelerated discovery Enhanced collaboraron Reduced costs Poten6al advantages Powerful capabilires IntuiRve interfaces Economies of scale Professional operarons ElasRc capacity Easier collaboraron OpRmizaRon 19

20 For More Information GlobusWorld: April Chicago Visit to: Get a free account and start moving files Visit for: Tutorials, FAQs, Pro Tips, Troubleshooting Papers, Case Studies Contact support@globusonline.org for: Help getting started & using the service Follow us on Twitter and Globus Online on Facebook 20