An Analytical Framework for Particle and Volume Data of Large-Scale Combustion Simulations Franz Sauer 1, Hongfeng Yu 2, Kwan-Liu Ma 1 1 University of California, Davis 2 University of Nebraska, Lincoln
Introduction Detailed combustion simulations Essential for developing high efficiency engines S3D by Sandia National Laboratories Two different representations of the flow Eulerian specification (vector field data) Lagrangian specification (particle data) Study data from either the Eulerian or Lagrangian viewpoints The ability to collate these results can be extremely useful Big data issues 2
Outline Framework overview Single data processing and analysis Topological Feature Extraction (Eulerian) Particle Query and Analysis (Lagrangian) Joint data processing and analysis Feature-based particle query Particle-based volume feature query Results Performance tests Example analyses Conclusion 3
Overview Black arrows represent traditional processing steps Red arrows represent feature-based particle query Blue arrows represent particle-based volume feature query Topological Flow Classification Vector Field Data Classified Voxel Data Feature Extraction Segmented Feature Particle Data Extracted Particles Particle Extraction Analysis 4
Topological Flow Classification Use a method proposed by Chong et al. 1 Compute a local rate-of-deformation tensor Categorize into one of 27 fundamental types Only a few dominated patterns present in simulation flows Classification Topological Description 2 Node / node / node, unstable (NNN/U) 11 Node / saddle / saddle, stable (NSS/S) 12 Node / saddle / saddle, unstable (NSS/U) 18 Focus / stretching, stable (FS/S) 19 Focus / stretching, unstable (FS/U) 20 Focusing / compressing, stable (FC/S) 21 Focusing / compressing, unstable (FC/U) 1 M. S. Chong, A. E. Perry, and B. J. Cantwell. A General Classification of Three Dimensional Flow Fields. Physics of Fluids, vol. 2, pp. 765-777, 1990. 5
Topological Flow Classification FC/U regions shown in green FS/S regions shown in yellow 6
Topological Feature Extraction High turbulence leads to features that are heavily interwoven Growing regions based on connectivity will span the entire dataset Need a way to pinch off features of interest Use a modified version of standard region growing techniques Measure a voxel s connectivity strength User defined threshold 7
Topological Feature Extraction Modified region growing 1. Users select a feature of interest by placing a seed point 2. Neighboring voxels of like topotype are added to a queue 3. Iterate through the queue a) Check connectivity strength by counting like neighbors b) Add to region if the count exceeds threshold c) Add like neighbors to queue 4. Growing finishes when queue is empty Increasing Threshold 8
Topological Feature Extraction Alternate extraction method using sub-classifications Divide classifications into 4 sub-types Grow each sub-region separately Count number of bordering voxels Connect according to a threshold Adds an extra level of control Original classification Sub-topo classification Increasing Threshold 9
Topological Feature Extraction Parallelize via master-worker paradigm Master process views an entire slice 3D domain is split among worker processes Grow a 2D region in serial on master node Treat each voxel as a seed point and distribute to worker nodes for growing Growing must continue across boundaries Send message to neighboring node Add necessary voxels to its queue 10
Particle Query and Analysis Extract subsets of particles based on its properties (temperature, mixture fraction, etc.) Embarrassingly parallel Each worker node can extract independently Requires a single pass Visualized as point-sprites Each node renders its subset of particles separately Combined on master node by checking depth buffers 11
Feature-Based Particle Query Study the properties of features using particle data Identify and extract particles encapsulated by a feature of interest Extend the particle query to accept voxel data 3D bitmask represents the feature Minimize communication cost Map the spatial location of the particle to voxel space Check against bitmask 12
Feature-Based Particle Query FS/S FC/U 13
Particle-based Volume Feature Query Study flow classifications based on particle data Map each extracted particle to voxel space Generate a 3D bitmask describing the location of particles Direct comparison to volume data Use as a set of seed points for region growing Trajectory assisted feature tracking Assemble particle data into trajectories Use as a correspondence between features at different timesteps 14
Results Real simulation data of a turbulent lifted ethylene jet Vector field data (2025 x 1600 x 400 grid) Particle data (~40 million particles) National Energy Research Scientific Computing Center (NERSC) Hopper - 6,384 node Cray XE6 system Each node consists of two AMD MagnyCours 2.1-GHz processors 15
Performance Tests Region growing time dependent on feature size Tests involve a feature at a scale of interest to scientists Approximately 10,000 voxels Separate tests for feature and particle extraction phases Do not reflect I/O times (both the particle and volume data have already been distributed to all nodes) 16
Performance Tests Particle Extraction Region Growing 17
Performance Tests Particle Extraction Region Growing 18
Sample Analyses Feature-based particle query Dataset represents a non-premixed jet Fuel and oxidizer are injected separately Mixing and burning in some portions of the jet Just mixing in other portions Mixture fraction becomes an important variable Look at relationship with temperature to determine if burning occurs 19
Sample Analyses Non-linear correlation (burning) Linear correlation (mixing) 20
Sample Analyses B A 21
Sample Analyses Feature A (burning) Feature B (mixing) 22
Sample Analyses Feature A (burning) Feature B (mixing) 23
Sample Analyses Particle-based Volume Feature Query Range query on temperature Extract the hottest/coldest parts of the jet Look at the flow classifications Hot portions: 35.9% FS/S and 23.2% FC/U Cold portions: 32.6% FS/S and 21.6% FC/U Similar breakdown for mid range temperatures 24
Conclusion and Future Work Present a framework that performs parallel data analyses on particle and volume data Modifications to region growing to aid in extracting turbulent flow features Parallelization leads to large speedups Particle extraction scales very well Region growing portion can still be improved Generalize to other datasets Explore trajectory assisted feature tracking In situ analysis and visualization 25
Acknowledgments Sandia National Laboratories Jackie Chen and Ray Grout National Science Foundation through grants OCI- 0905008, OCI-0850566, OCI-0749227, CCF-0811422 Department of Energy through grants DEFC02-06ER25777, DE-CS0005334, DE-FC02-12ER26072 with program managers Lucy Nowell and Ceren Susut- Bennett 26
Thank You Questions?