James P. Doherty AOG 2016

Transcription

1 Combing MATLAB with Amazon Web Services to produce highly scale cloud based applications James P. Doherty AOG 2016

2 Overview MATLAB is the world s leading scientific computing software. Amazon Web Services is the world s leading cloud computing platform. Combining the two enables applications written in MATLAB to access almost unlimited computational power and durable data storage. This enables us to: solve extremely large computational problems rapidly, without the cost of setting up and maintaining a super computer or a cluster. deploy highly available MATLAB based web applications that can be accessed by one or one-million users. centralise data storage and streamline workflows. By using the MATLAB Complier and MATLAB SDK toolbox, this can be done without re-writing any MATLAB code.

3 Outline What is the cloud? AWS Why cloud is useful to engineering design? Example application Architecture behind the application Conclusions

4 What is the Cloud? Somewhere at the other end of your internet connection a place where you can access apps and services, and where your data can be stored securely. The cloud is a big deal for three reasons: No effort on your part to maintain or manage it. You can access cloud-based applications and services from anywhere all you need is a device with an Internet connection. It's effectively infinite in size, so you don't need to worry about it running out of capacity and scales on demand so you only pay for what you use. Traffic/compute power Victorian Racing Club Melbourne Cup Provision for peak Wasted $$ Scale on demand Jan Dec Some of the maor cloud computing companies

5 AWS Every day, AWS adds enough new server capacity to support all of Amazon s global infrastructure when it was a $7B retailer

6 What role can cloud computing play in engineering design? Access virtually unlimited computing power Specific machines for RAM, storage, or GPU For highly parallel tasks, like Monte Carlo or optimisation problems you can spread work across 1000s of machines Highly durable data storage Enhanced QA/QC Desktop applications are hard to control and maintain. With cloud you have one fully QA ed copy of an application. Access from anywhere. Improved proect management and collaboration. Secure intellectual property

7 The application: design foundations for offshore oil and gas infrastructure Source: Norske Hydro Particular challenges Weak soils Complex load history Capacity depends on load history Must simulate many cases Pazfloor, Angola. Image courtesy of Subsea 7

8 Acknowledgements Ali Amodio Carl Erbrich Dave Williams Fraser Bransby Henry Krisdani Marcelo Silva Mike O Neill Phil Watson Susan Gourvenec Dave White Mark Randolph

9 Offshore foundation design Process Input soil properties Consolidation analysis to get change in effective stress under dead loads Change in effective stress leads to change in soil strength Using modified strength, conducted a series of H-V capacity calculations for different effective widths and CLR values Compare with 1000 s of load cases Document H seabed V Image courtesy of Susan Gourvenec

10 Job details

11 Paste input into excel like tables

12 Solve the consolidation problem

13 H-V capacity problem

14 Example output Computationally intensive; up to 30 minutes per run

15 More output from HV

16 Paste in 1000s of load cases

17 Compute FoS

18 System architecture All AWS regions have at least two physically separate availability zones, with a guaranteed uptime of 99.95%. Design your system so that if one zone goes down, the system still works as normal

19 System architecture All AWS regions have at least two physically separate availability zones, with a guaranteed uptime of 99.95% Some services reside in specific availability zones Computer servers

20 System architecture All AWS regions have at least two physically separate availability zones, with a guaranteed uptime of 99.95% Others automatically work across zones

21 System architecture User fills in a web form and hits run and send a package of data to the web server

22 System architecture The web server sends the package to a data base for storage Job queueing Web server is in constant communication with data base to keep user informed of status of the ob

23 System architecture The web server sends meta data to a queue Status=queueing Job queueing

24 System architecture An application server reads the meta data from a queue Status=queueing Job queueing

25 System architecture Using this meta data, and the MATLAB Database Toolbox the application server can get the full input data obect from the database and can then run the ob Job Running

26 System architecture; more than serving up a web page! After the ob finishes and app server generates output data and writes this to the database using the MATLAB Database Toolbox. Images are stored in S3 Status=Complete Job Complete

27 System architecture The computational engine(s) EC2 instances with the following MATLAB Runtime Complier Library (MCR) Your complied MATLAB code as a dll A wrapper function that can read from SQS and call the dll Then from this, create an Amazon Machine Image (AMI). Can then spin up as many EC2 instances based on this AMI

28 Example with eight App servers

29 Example with sixteen App servers

30 Conclusions Cloud computing offers opportunities to access massive compute power and gives flexibility and freedom to a workforce. It also assists with QA and QC and helps secure valuable IP. This has been demonstrated by developing the Fugro AG shallow foundation design package, BearCon. The web application combines several design processes into a single workflow, resulting in a large reduction in manual data processing, making design activities more efficient. The scaling capability of cloud based computing will allow large volumes of foundation design calculations to be conducted. This will allow a range of design options to be explored, resulting in optimised foundation solutions. i.e. better engineering design. Reliability based design can be developed on the back of this Monte Carlo simulations of a calculations that takes 5 minutes, would take 1 week on a single machine. Or, using 200 machines (say AWS M3 instances at 15 cents per hr), would get the ob done in around 48 minutes for $30.

31 Conclusions By compiling MATLAB into dll using the Complier and SDK toolbox, MATLAB can be seamlessly combined with AWS. The MATLAB Database Toolbox simplifies data storage and integration of the data into MATLAB algorithms. Found to be fast and reliable. Even though MATLAB was one of about 8-10 different technologies used it was probably the most important in enabling an application such as BearCon to be written.