1

Observations of an Accidental Computational Scientist SIAM/NSF/DOE CSME Workshop 25 March 2003 David Keyes Department of Mathematics & Statistics Old Dominion University & Institute for Scientific Computing Research Lawrence Livermore National Laboratory

2

Academic and lab backgrounds86-02: ICASE, NASA Langley74-78: B.S.E., Aerospace and Mechanical/Engineering Physics 78-84: M.S. & Ph.D., Applied Mathematics 84-85: Post-doc, Computer Science 86-93: Asst./Assoc. Prof., Mechanical Engineering 93-99: Assoc. Prof., Computer Science 99-03: Prof., Mathematics & Statistics 03- : Prof., Applied Physics & Applied Mathematics99- : ISCR, Lawrence Livermore 03- : CDIC, Brookhaven

3

Computational Science & EngineeringA “multidiscipline” on the verge of full bloom Envisioned by Von Neumann and others in the 1940’s Undergirded by theory (numerical analysis) for the past fifty years Empowered by spectacular advances in computer architecture over the last twenty years Enabled by powerful programming paradigms in the last decade Adopted in industrial and government applications Boeing 777’s computational design a renowned milestone DOE NNSA’s “ASCI” (motivated by CTBT) DOE SC’s “SciDAC” (motivated by Kyoto, etc.)

4

Niche for computational scienceHas theoretical aspects (modeling) Has experimental aspects (simulation) Unifies theory and experiment by providing common immersive environment for interacting with multiple data sets of different sources Provides “universal” tools, both hardware and software Telescopes are for astronomers, microarray analyzers are for biologists, spectrometers are for chemists, and accelerators are for physicists, but computers are for everyone! Costs going down, capabilities going up every year

5

Simulation complements experimentationEnvironment global climate wildland firespread Experiments expensive Experiments prohibited or impossible Scientific SimulationExperiments dangerousExperiments difficult to instrument

6

Example #1: wildland firespreadSimulate fires at the wildland-urban interface, leading to strategies for planning preventative burns, fire control, and evacuationJoint work between ODU, CMU, Rice, Sandia, and TRW

7

Example #1: wildland firespread, cont.Objective Develop mathematical models for tracking the evolution of wildland fires and the capability to fit the model to fires of different character (fuel density, moisture content, wind, topography, etc.) Accomplishment to date Implemented firefront propagation with level set method with empirical front advance function; working with firespread experts to “tune” the resulting model Significance Wildland fires cost many lives and billions of dollars annually; other fire models pursued at national labs are more detailed, but too slow to be used in real time; one of our objectives is to offer practical tools to firechiefs in the field

8

Example #2: aerodynamicsSimulate airflows over wings and streamlined bodies on highly resolved grids leading to superior aerodynamic designJoint work between ODU, Argonne, LLNL, and NASA-Langley

9

Example #2: aerodynamics, cont.Objective Develop analysis and optimization capability for compressible and incompressible external aerodynamics Accomplishment to date Developed highly parallel nonlinear implicit solvers (Newton-Krylov-Schwarz) for unstructured grid CFD, implemented in PETSc, demonstrated on a “workhorse” NASA code running on the ASCI machines (up to 6,144 processors) Significance Windtunnel tests of aerodynamic bodies are expensive and difficult to instrument; computational simulation and optimization (as for the Boeing 777) will greatly reduce the engineering risk of developing new fuel-efficient aircraft, cars, etc.

10

Example #3: radiation transportSimulate “flux-limited diffusion” transport of radiative energy in inhomogeneous materialsJoint work between ODU, ICASE, and LLNL

11

Example #3: radiation transport, cont.Objective Enhance accuracy and reliability of analysis methods used in the simulation of radiation transport in real materials Accomplishment to date Leveraged expertise and software (PETSc) developed for aerodynamics simulations in a related physical application domain, also governed by nonlinear PDEs discretized on unstructured grids, where such methods were less developed Significance Under current stockpile stewardship policies, DOE must be able to reliably predict the performance of high-energy devices without full-scale physical experiments

12

Example #4: fusion energySimulate plasmas in tokomaks, leading to understanding of plasma instability and (ultimately) new energy sourcesJoint work between ODU, Argonne, LLNL, and PPPL

13

Example #4: fusion energy, cont.Objective Improve efficiency and therefore extend predictive capabilities of Princeton’s leading magnetic fusion energy code “M3D” to enable it to operate in regimes where practical sustained controlled fusion occurs Accomplishment to date Augmented the implicit linear solver (taking up to 90% of execution time) of original code with parallel algebraic multigrid; new solvers are much faster and robust, and should scale better to the finer mesh resolutions required for M3D Significance An M3D-like code will be used in DOE’s Integrated Simulation and Optimization of Fusion Systems, and ITER collaborations, with the goal of delivering cheap safe fusion energy devices by early-to-mid 21st century

14

We lead the “TOPS” projectU.S. DOE has created the Terascale Optimal PDE Simulations (TOPS) project within the Scientific Discovery through Advanced Computing (SciDAC) initiative; nine partners in this 5-year, $17M project, an “Integrated Software Infrastructure Center”

15

Toolchain for PDE Solvers in TOPS* projectDesign and implementation of “solvers” Time integrators Nonlinear solvers Constrained optimizers Linear solvers Eigensolvers Software integration Performance optimization (w/ sens. anal.)(w/ sens. anal.)*Terascale Optimal PDE Simulations: www.tops-scidac.org

16

SciDAC apps and infrastructure4 projects in high energy and nuclear physics5 projects in fusion energy science14 projects in biological and environmental research10 projects in basic energy sciences

17

Optimal solversConvergence rate nearly independent of discretization parameters Multilevel schemes for linear and nonlinear problems Newton-like schemes for quadratic convergence of nonlinear problemsAMG shows perfect iteration scaling, above, in contrast to ASM, but still needs performance work to achieve temporal scaling, below, on CEMM fusion code, M3D, though time is halved (or better) for large runs (all runs: 4K dofs per processor)

18

We have run on most ASCI platforms…‘97‘98‘99‘00‘01‘02‘03‘04‘05‘06100+ Tflop / 30 TBTime (CY)Capability1+ Tflop / 0.5 TB30+ Tflop / 10 TBRed3+ Tflop / 1.5 TBBlue10+ Tflop / 4 TBWhite50+ Tflop / 25 TBNNSA has roadmap to go to 100 Tflop/s by 2006 www.llnl.gov/asci/platformsSandiaLos AlamosLivermoreLivermore

19

…and now the SciDAC platformsIBM Power4 Regatta 32 procs per node 24 nodes 166 Gflop/s per node 4Tflop/s (10 in 2003)IBM Power3+ SMP 16 procs per node 208 nodes 24 Gflop/s per node 5 Tflop/s (doubled in February to 10) BerkeleyOak Ridge

20

Computational Science at Old DominionLaunched in 1993 as “High Performance Computing” Keyes appointed ‘93; Pothen early ’94 Major projects: NSF Grand, National, and Multidisciplinary Challenges (1995-1998) [w/ ANL, Boeing, Boulder, ND, NYU] DoEd Graduate Assistantships in Areas of National Need (1995-2001) DOE Accelerated Strategic Computing Initiative “Level 2” (1998-2001) [w/ ICASE] DOE Scientific Discovery through Advanced Computing (2001-2006) [w/ ANL, Berkeley, Boulder, CMU, LBNL, LLNL, NYU, Tennessee] NSF Information Technology Research (2001-2006) [w/ CMU, Rice, Sandia, TRW]

21

CS&E at ODU todayCenter for Computational Science at ODU established 8/2001; new 80,000 sq ft building (for Math, CS, Aero, VMASC, CCS) opens 1/2004; finally getting local buy-in ODU’s small program has placed five PhDs at DOE labs in the past three years

22

Post-doctoral and student alumniLois McInnes, ANLLinda Stals, ANUDinesh Kaushik, ANLSatish Balay, ANLD. Karpeev, ANL

23

24

LLNL’s ISCR fosters collaborations with academe in computational science

25

ISCR’s philosophy: Science is borne by peopleBe “eyes and ears” for LLNL by staying abreast of advances in computer and computational science Be “hands and feet” for LLNL by carrying those advances into the laboratory Three principal means for packaging scientific ideas for transfer papers software people People are the most effective!

26

ISCR brings visitors to LLNL through a variety of programs (FY 2002 data)