UQ pipeline implementa,on and so0ware integra,on

Transcription

1 UQ pipeline implementa,on and so0ware integra,on michael aivazis psaap review october 2009

2 Table of contents 1. Introduc,on people, computa,onal resources 2. Overview of the UQ pipeline problem scope pipeline architecture the ingredients: pyre, mys,c, VTF, eureka capabili,es, verifica,on and valida,on 3. Status and outlook summary of work in progress planned ac,vi,es for the remainder of the year tasks that specifically address recommenda,ons from the last review assessment 2

3 introduc,on people computa,onal resources resource u,liza,on so0ware engineering 3

4 introduc,on: people The so0ware integra,on team From CSE: Michael Aivazis, Sharon BruneS, Julian Cummings, Jan Lindheim, San,ago Lombeyda, Mike McKerns, Mark Stalzer, Leif Strand From Solid Dynamics and Materials: Michael Or,z, Anna Pandolfi, Bo Li From UQ: Tim Sullivan Expect the group to grow with representa,on from Experimental Science and CFD 4

5 introduc,on: computa,onal resources Computa,onal resources 12,800 opteron cores 512 opteron cores 2144 opteron cores 810 opteron cores 5

6 introduc,on: computa,onal resources Computa,onal resources details cores cpu type memory (GB/core) interconnect os compilers shc (CACR) 810 AMD Opteron 2.{2,4,5} GHz 2 Infiniband RHEL Pathscale coyote (LANL) ubgl (LLNL) lobo (LANL) hera (LLNL) cerrillos (LANL) 2,580 82,000 4,352 13, AMD Opteron 2.6 GHz IBM PowerPC2 700 MHz AMD Opteron 2.2 GHz AMD Opteron 2.3 GHz AMD Opteron Cell 4 Infiniband RHEL Pathscale 1/2 BG torus+tree SuSE IBM 2 Infiniband CHAOS Pathscale 2 4 1/4 Infiniband Infiniband between Opteron cores CHAOS Modified RHEL Pathscale IBM/Intel 6

7 introduc,on: resource u,liza,on Resource u,liza,on CPU cycles used by Caltech Jan 1 to Oct 20, ,564,069 6,024,505 hrs , ,523 hera coyote lobo shc 7

8 introduc,on: so0ware engineering Complexity management Sources: Project size: asset complexity: number of lines of code, files, entry points dependencies: number of modules, third party libraries run,me complexity: number of objects types and instances Problem size: number of processors needed, amount of memory, cpu,me Project longevity: life cycle, duty cycle cost/benefit of reuse managing change: people, hardware, so0ware, technologies Locality of needed resources compute/persist: where, how, when, who Usability: access, interfaces, security, etc Risk mi,ga,on: Promote and ensure key prac,ces weekly mee,ngs coding and documenta,on standards uniform builds, tes,ng: regression, benchmarks, verifica,on, valida,on so0ware process: svn, trac, wiki, and doxygen, epydoc, tes,ng: unit, component, applica,on; muta,on, regression; soon to be automated Flexible so0ware architecture 8

9 the UQ pipeline problem scope pipeline architecture the ingredients: pyre, mys,c, VTF, eureka capabili,es, verifica,on and valida,on 9

10 the UQ pipeline: scope The physical system High fidelity modeling of ballis,c impact requires non linear kinema,cs advanced material models a method for handling the extreme deforma,ons caused by the penetra,on fracture and fragmenta,on erosion robust contact detec,on and resolu,on algorithms 10

11 the UQ pipeline: scope Problem scope Our methodology involves global op,miza,ons that require efficient explora,on of a huge parameter space D F is the largest devia,on in performance when each input parameter is allowed to vary over its en,re range D F-G is very similar Need to stage, monitor and analyze the output of thousands of simula,ons against the backdrop of constantly shi0ing computa,onal resources Managing such a complex computa,onal environment requires a sophis,cated so0ware infrastructure Pyre, the Caltech ASC component framework, has par,al support for much of the required infrastructure 11

12 the UQ pipeline: scope Workflow overview Evalua,ng the model diameters involves analyzing the available datasets from previous runs preparing a collec,on of interes,ng input decks iden,fying appropriate computa,onal resources for the new set of runs preparing the computa,onal environment for each run shipping the input decks verifying that ancillary requirements such as connec,ons to the monitoring and journaling agents, are sa,sfied scheduling the job with the machine queue manager monitoring the runs wait for a job to start, monitor its progress, collect output (including debugging/performance info) collec,ng partly digested results from remote machine and archiving them along with their input deck for later analysis See sketch of the architecture on next slide 12

13 the UQ pipeline: architecture Architectural overview machine 1 job manager monitor queue manager optimizer population generator convergence? job manager archiver journal job 1 job 2 job n job manager machine N queue manager job 1 job 2 job n 13

14 the UQ pipeline: ingredients Early pipeline implementa,on DAKOTA + ABAQUS Limita,ons file I/O based interac,ve diagnos,cs for progress/state of UQ or op,miza,on progress not built in the UQ, Dakota, and model codes must be separately compiled and tested on each machine simulated annealing not available (to complement gene,c algorithms) deployment limited to single machines adding func,onality to intelligently tap resources from remote machines tailoring op,miza,on algorithms for specific needs is difficult some algorithms only find infima user must negate objec,ve func,on to do supremum op,miza,on not a standard package already installed on most lab systems diagnos,cs involving visualiza,on are clumsy for large variable counts 14

15 the UQ pipeline: ingredients Ingredients of the UQ pipeline Our implementa,on: mys,c, a distributed op,miza,on framework fully deployed on lab machines managed calcula,ons using the SPHIR surrogate on thousands of cores a large number of op,miza,on algorithms scipy: community supported/maintained see hsp:// local varia,ons that are well suited to our problem a simula,on archiving subsystem PostgreSQL database back end web based user interface a distributed simula,on monitoring subsystem diagnos,cs, probes custom simula,on viewer for debugging and post mortem analysis computa,onal engines for modeling the impact VTF next genera,on codes from our center a surrogate for the SPHIR gun 15

16 the UQ pipeline: ingredients Pyre Pyre is a soaware architecture: a specifica,on of the organiza,on of the so0ware system a descrip,on of the crucial structural elements and their interfaces a specifica,on for the possible collabora,ons of these elements a strategy for the composi,on of structural and behavioral elements Pyre is mul, layered flexibility complexity management robustness under evolu,onary pressures Pyre is a component framework applicabon specific applicabon general framework computabonal engines 16

17 the UQ pipeline: ingredients Using components Component based solu,ons are ideal for complex systems encourage the decomposi,on of the problem into manageable func,onal units expose the interac,on mechanisms between these units enable the nearly independent evolu,on of the parts Component frameworks enable an incremental and evolu,onary approach exis,ng codes can start producing results immediately new services can be incorporated incrementally The goal is to encapsulate and deploy F component core input ports properbes Component control name output ports 17

18 the UQ pipeline: ingredients Services for computa,onal engines Normal engine life cycle: deployment staging, instan,a,on, sta,c ini,aliza,on, dynamic ini,aliza,on, resource alloca,on launching input delivery, execu,on control, hauling of output teardown resource de alloca,on, archiving, execu,on sta,s,cs Excep,onal events core dumps, resource alloca,on failures diagnos,cs: errors, warnings, informa,onal messages monitoring: debugging informa,on, self consistency checks Parallel processing Distributed compu,ng 18

19 the UQ pipeline: ingredients Simula,on services Problem specifica,on components and their proper,es Solid modeling overall geometry model construc,on topological and geometrical informa,on Boundary and ini,al condi,ons high level specifica,on access to the underlying solver data structures in a uniform way Materials and cons,tu,ve models materials proper,es database strength models and EOS associa,on with a region of space Computa,onal engines selec,on and associa,on with geometry solver specific ini,aliza,ons Simula,on driver ini,aliza,on appropriate,mestep computa,on orchestra,on of data exchanges checkpoints and field dumps Ac,ve monitoring instrumenta,on: sensors, actuators real,me visualiza,on Full simula,on archiving 19

20 the UQ pipeline: ingredients Simula,on archiving Produce a fully repeatable execu,on by recording scripts user choices sources (cvs/svn tags or even the files themselves) build procedure required third party libraries version of as many run,me components as can be determined generated data sets (urls, actual files) Implementa,on meta data in PostgreSQL HDF5 embed XML meta data parsed for deducing the layout of the file as format evolves can be extracted for easy indexing 20

21 the UQ pipeline: ingredients mys,c: key components The job manager stages and launches new jobs broadcasts execu,on control direc,ves maintains a registry of submised jobs The iterator adjusts the cost func,on parameters reacts to control direc,ves The mapping strategy provides an algorithm to distribute the workload among available resources The launcher knows how submit jobs on the current execu,on environment 21

22 the UQ pipeline: ingredients User interface for the simula,on archive 22

23 the UQ pipeline: ingredients Custom simula,on viewer 23

24 the UQ pipeline: ingredients Simula,on capability: the VTF Ini,al capability built using adlib, the center s Lagrangian solver finite kinema,cs parallel explicit dynamics with excellent scalability flexible, scalable meshing elements: ten noded quadra,c, ten noded composite material models: power law, and J2+vinet contact: smooth and non smooth surface based contact pyre integra,on PSAAP extensions contact: volume based: billiard ball element erosion: geometric: in radius, element quality varia,onal: compares the deforma,on energy to the fracture energy 24

25 the UQ pipeline: ingredients Status VTF simula,on capability Components: new contact and element erosion algorithms in place paralleliza,on complete: large runs on all plauorms Verifica,on: in the process of collec,ng and organizing the historical verifica,on tests into a coherent test suite element types, material models, contact: in place element erosion: in progress Valida,on: building valida,on applica,ons for all major components materials: uniaxial tests complete, shear tests need revival contact algorithms are being validated against Molinari[2002] for impact speeds below 500m/s with good ini,al agreement comparison with our experiments is in progress 25

26 the UQ pipeline: ingredients Perfora,on using the VTF 26

27 the UQ pipeline: ingredients Perfora,on using the VTF II 27

28 the UQ pipeline: ingredients Valida,on of the contact algorithm The billiard ball contact algorithm has three free parameters: p: the frac,onal interpenetra,on volume k: the s,ffness of the contact restoring force b: a dampening factor Comparison with experiments [Molinari 2002] of spherical steel projec,les on thick steel plates v = 200m/s v = 400m/s v = 600m/s 28

29 the UQ pipeline: ingredients Simula,on profiling Time consuming por,ons of the code revealed by profiling tools Speedshop, gprof, etc. Outliers shown in load balancing reports Erosion computa,on is responsible for ~85% of the,me in simula,ons explicit integra,on checking is currently done every step costly residual_general log_mulss assemble: loop unrolling possibili,es! Compu,ng and upda,ng correctors and restoring forces, ~15% Need to keep exploring fracture based erosion scheme less costly element erosion computa,on SpeedShop profile of a run with 465K elements, 64 MPI tasks on 4 hera nodes 29

30 the UQ pipeline: ingredients Scaling of the contact algorithm avg. Bme to perform contact detecbon (sec) proxyball scaling hera log(2) cores avg. time per step (sec) K elements in ini,al mesh, grows with core count Contact occurs in a rela,vely small region of the plate Execu,on,me increases with core count, as some cores handle larger contact regions For larger core counts, the contact region begins to be be effec,vely distributed full application - vtf with proxyball weak scaling 22K element base mesh log(2) cores shc 30

31 the UQ pipeline: ingredients Load balance issues on small core counts MPI_allreduce balance % of MPI Bme MPI task 3.2M element run on 32 shc cores 31

32 the UQ pipeline: ingredients Applica,on scaling avg Bme/step (sec) Log(2) CPUs hera (LLNL) lobo (LANL) shc (Caltech) 32

33 the UQ pipeline: ingredients Next genera,on lagrangian code eureka: a new solid dynamics capability object oriented finite element and meshfree framework highly flexible and extensible finite deforma,ons, visco elas,city, viscoplas,city, thermal coupling, contact, fracture and fragmenta,on extensive material model library OTM (op,mal transporta,on meshfree) based on op,mal transporta,on theory with material point sampling both solid and fluid flows exact essen,al boundary condi,on enforcement exact linear and angular momentum conserva,on free from tensile instabili,es contact provably convergent energy based material point erosion algorithm 33

34 the UQ pipeline: ingredients SPHIR surrogate Model of the SPHIR response: MICHAEL AIVAZIS PSAAP REVIEW OCTOBER 2009 perfora,on diameter as a func,on of projec,le speed and plate thickness α = 0 6 α =

35 status and outlook summary of work in progress planned ac,vi,es for the remainder of the year tasks that specifically address recommenda,ons from the last review assessment 35

36 status and outlook In progress Complete valida,on of the new contact algorithm Integrate new erosion criterion into the simula,on drivers Valida,on against our experimental data Paralleliza,on: contact element erosion Conduct preliminary runs using Ta(j2+vinet) for projec,le, target Manage the deluge of informa,on from our runs db schema almost complete data harves,ng techniques job tracking: both programma,c and interac,ve Deploy the prototype distributed op,miza,on framework pyre driven applica,ons explore security issues on lab machines integrate with job tracking 36

37 status and outlook Planned ac,vi,es So0ware test suites automa,on coherent verifica,on strategy Simula,on capability lagrangian solver: contact valida,on improve simula,on capability and validate against our experiments eureka: framework integra,on paralleliza,on eulerian code: repeat our process with the new code UQ framework simula,on archiving explora,on of op,miza,on algorithms and their effect on the methodology VTF driven by mys,c recas,ng of exis,ng simula,on drivers as pyre applica,ons for beser integra,on with the UQ framework 37

38 status and outlook Review recommenda,ons Verifica,on: in previous years, verifica,on and valida,on was the responsibility of the research groups and the results were published in the literature we have embarked on a systema,c construc,on of thorough regression, benchmark, verifica,on and valida,on test suites tes,ng strategy is documented on our wiki tes,ng will be integrated with the simula,on launching and archiving facili,es so tests can be submised anywhere the code runs, at any,me Computa,onal requirements: we believe we understand the capabili,es necessary to model ballis,c impact performance modeling is underway: we collect data from every run and we are building resource predictors Iden,fica,on, quan,fica,on and reduc,on of the major sources of uncertainty individual variable sub diameters are excellent metrics uncertainty reduc,on is now a high priority task 38

39 status and outlook Assessment Assessment from last year: good con,nuity from ASC to PSAAP management structure so0ware development process VTF extensions to handle the new applica,on are well underway Since then: completed deployment of UQ pipeline on lab machines ini,ated the construc,on of our test suites instrumented simula,on to collect data for performance modeling new contact algorithm implementa,on, verifica,on; valida,on underway; paralleliza,on new erosion criterion implementa,on, verifica,on and valida,on tests to be constructed; paralleliza,on preliminary design and implementa,on of the pyre based distributed op,miza,on framework see poster for details Deferred: Ta/Ta impacts: un,l experiments are available 39

40 end of presenta,on 40