High-performance computing of wind farms in the atmospheric boundary layer

High-performance computing of wind farms in the atmospheric boundary layer VSC User Day Antwerp, 30 November 2015 Johan Meyers Mechanical Engineering, KU Leuven

Acknowledgements Research team Dries Allaerts, Pieter Bauweraerts, Vahid Bokharaie (postdoc), Juliaan Bossuyt, Ali Emre Yilmaz, Thomas Haas, Wim Munters, Cornelia Nita, Athanasios Vitsas, Liang Fang HPC support team @ KU Leuven Martijn Oldenhof, Jan Ooghe, Leen Van Rentergem Funding ERC Active Wind Farms (Grant no 306471), FWO G.0376.12, IDO/11/012 (BOF/KU Leuven) Computing: VSC TIER 1 and TIER 2 (KU Leuven)

London Array, UK 175 turbines 630 MW Source: London Array

Alta Wind Energy Center, USA 490 turbines 1,320 MW Source: http://en.wikipedia.org/wiki/alta_wind_energy_center

Gansu Wind Farm Project, China = cluster of wind farms Phase one (2010) More than 3500 turbines Installed capacity of 5,160 MW Total project (2020) Planned capacity of 20,000 MW Construction rate of 36 turbine/day One of six national megaprojects... Source: https://en.wikipedia.org/wiki/gansu_wind_farm

Rotor Diameter and Rated Power IPCC (2011), Special report on renewable energy

Vestas V164 8MW (2014) Overall height of 220 m Rotor diameter of 164 m Source: http://www.centralcarbonfiber.com/application/index.html

Horns Rev 1, DK 12 February 2008 Source:Vattenfall

Power deficit in wind farms Horns Rev Barthelmie et al., J. Phys.: Conference Series 75 (2007) 012049 Lillgrund Dahlberg JÅ, Vattenfal Tech Report, 2009

Outline 1. Single turbine: operation and Betz theory 2. Wind-farm simulations & the atmospheric boundary layer 3. SP-Wind: parallelization details 4. Perfomance issues and improving parallel communications 5. Summary and outlook

Single turbine: Betz theory Betz A. Das Maximum der theoretisch moglichen Ausnutzung des Windes durch Windmotoren. Zeitschrift für das gesamte Turbinenwesen 1920; 26: 307 309. Joukowsky NE. Windmill of the NEJ type. In Transactions of the Central Institute for Aero-hydrodynamics of Moscow, 1920 (in Russian).

Single turbine: Betz theory /. % Wake mixing /. % Dissipation / % /. % /. %

Outline 1. Single turbine: operation and Betz theory 2. Wind-farm simulations & the atmospheric boundary layer 3. SP-Wind: parallelization details 4. Perfomance issues and improving parallel communications 5. Summary and outlook

Some physical aspects of ABLs Various ABL types Source: Stull, 1988, An introduction to Boundary Layer Meteorology Earth s rotation - Coriolis forces linear with velocity - Motions to pole are deflected to the right (northern hemisphere) Thermal stability - Stable stability surpresses vertical motions

Conventionally neutral ABL 3D view Free atmosphere Ground level

Wind-farm boundary layer modeling Classical approach Pressure-driven boundary layer Assumptions Turbines in inner layer Outer layer dynamics negligible Neglect Coriolis forces and thermal effects (for neutral ABL)

LES of wind-farm boundary layer Horns Rev Code: SP-Wind Mesh: 1536 x 768 x 192 Turbine model: ADM

Wind-farm boundary layer modeling Classical approach Pressure-driven boundary layer Assumptions Turbines in inner layer Outer layer dynamics negligible Neglect Coriolis forces and thermal effects (for neutral ABL) Does this approach still hold for Large turbines? Very large wind farms? What is the importance of outer layer effects?

LES of wind-farm boundary layer Outer layer effects z [km] z [km] y[km] x [km] z [km] y[km] x [km] Numerical details Wind farm: 20 x 9 turbines Resolution: 30 x 15 x 5 m Domain: 28.8 x 4.8 x 5 km Grid points: ca. 165M Simulated time: 4h y[km] x [km] High Performance Computing ca. 10 days on 512 cores

Wind-farm boundary layer: flow behaviour 1 km 500 m 200 m

Wind-farm optimal control Consider a wind-farm with N turbines AIM, min Subject to ;,, disk velocity F, 0 0 Degrees of freedom in control space, 15000 USE ADJOINT EQUATIOS TO CALCULATE GRADIENT (continuity) (momentum)

Forward and adjoint simulations Forward simulation Adjoint simulation

Optimal control of wind-farm boundary layer Power per Row Overall gain power output: 7% (preliminary further gains possible) Coordinated Optimal Control Hub height axial velocity Coordinated Optimal Control Greedy ( ) Greedy ( )

Outline 1. Single turbine: operation and Betz theory 2. Wind-farm simulations & the atmospheric boundary layer 3. SP-Wind: parallelization details 4. Perfomance issues and improving parallel communications 5. Summary and outlook

Numerical discretization Navier Stokes equations 0 Discretization in SP-Wind is using: o Fourier-spectral method in x, y o fourth-order energy-conserving finite differences in z

Numerical discretization In Fourier space (,, ) 0 Problem: convolution sum in Fourier space computational effort ~ Instead: Cost

Fourier transforms and parallelization Two-dimensional Fourier transforms (in x, y) Successive 1D (I)FFTs: first in x-direction, then in y-direction Use of FFTW Code parallelization (until end 2013): slab decomposition IFFT in y-dir MPI Communication IFFT in x-dir FFT in y-dir MPI Communication FFT in x-dir

New parallelization SP-Wind Pencil decomposition

New parallelization SP-Wind Fourier Space Real Space

Outline 1. Single turbine: operation and Betz theory 2. Wind-farm simulations & the atmospheric boundary layer 3. SP-Wind: parallelization details 4. Perfomance issues and improving parallel communications 5. Summary and outlook

Strong scaling initial tests Cores Challenging for parallelization

Strong scaling speed up Cores

Communication vs. no communication No communication Cores

Intra node versus internode communication First observations Important bottleneck: internode communication Internode communication << intranode communication

Intranode versus internode communicatoin Rearrange communication order Intranode Internode Internode Intranode Intranode

Improved communication arrangement Cores

Improve internode communication 1400 256 cores CASE: 512 768 192 Probability density function 1200 1000 800 600 400 200 MPI_ALLTOALL MPI_ALLTOALL_packed MPI_IALLTOALL FFTW_MPI 10000 samples for good convergence 0 0 0,1 0,2 0,3 0,4 0,5 0,6 Internode Communication time

Improve internode communication 1000 CASE: 512 768 192 1024 cores Probability density function 900 800 700 600 500 400 300 200 100 0 MPI_ALLTOALL MPI_ALLTOALL_PACKED MPI_IALLTOALL FFTW_MPI 0 0,05 0,1 0,15 0,2 0,25 0,3 0,35 0,4 0,45 0,5 Internode Communication time

Improved performance 35 30 25 20 15 10 5 SPEEDUP vs. 32 CORES CASE: 512 768 192 0 0 200 400 600 800 1000 1200 Met intra node opt Beste versie Ideaal zonder intra node opt

HPC Case planning

Summary and outlook HPC simulations of wind-farm boundary layers o o Improve understanding of relevant physics (towards new generation of design models for industry) Change wind-farm control Supercomputing and parallelization o o Intranode communication much faster than internode Packing of communication can be advantageous (not always) Further improvements: towards 1000+ cores per run

THANK YOU