TOPS Scalable Solvers for Complex Field Simulations

     
     
     
     
     
     
     
     
     
     
     
     
     
     
     


     
PIs: G. Biros5, X. Cai 8, E. Chow3, F. Dobrian6, M. Dumett 3, V. Eijkhout9, R. Falgout3 , O. Ghattas 4, B. Hientzsch5, D. Keyes 6 , M. Knepley 1, S. Li 2,7, T. Manteuffel 8, S. McCormick 8,  A. Pothen 6 , R. Serban 3, B. Smith 1, P. Vassilevski 3, O. Widlund 5, C. Woodward 3, H. Zhang 1, ISIC affiliates : D. Brown (TSTT), P. Colella (APDEC), L. Freitag (TSTT), P. Hovland (PERC), L. McIness (CCTTSS), Application affiliates : A. Bhattacharjee (CMRS), J. Breslau (CEMM), J. Chen (CEMM), E. D’Azevedo (TSI), K. Germaschewski (CMRS), S. Jardin (CEMM), B. Messer (TSI), A. Mezzacappa (TSI)
 
1Argonne National Lab, 2Lawrence Berkeley National Lab, 3Lawrence Livermore National Lab, 4Carnegie Mellon U., 5 New York U., 6 Old Dominion U., 7 U. California-Berkeley, 8 U. Colorado-Boulder, 9 U. Tennessee
  

Summary
 
In support of SciDAC fusion and astrophysical simulations, the Terascale Optimal PDE Simulations (TOPS) project is creating a new generation of solvers for PDE field problems.


 The wide variety of partial differential equations (PDEs) that arise in core DOE science missions defies a general-purpose computational approach.  For dense and sparse linear systems, ordinary differential equations, and problems of more regimented mathematical structure, DOE has developed very successful broad-purpose numerical libraries. For PDE systems, appropriately, it has instead developed special-purpose code groups at national laboratories. 
 
TOPS aspires to provide a general-purpose toolkit of scalable solvers for the systems that arise at inner loops of the vast majority of these special-purpose codes, in which PDE field equations are reduced to large systems of nonlinear or linear equations by local discretizations (finite differences, finite elements, finite volumes) on Eulerian grids. Solving such systems commonly consumes 50% or more of the execution time of PDE-based codes (it presently consumes 90% in one of our partner applications).
 
TOPS solvers are actually more general-purpose than a focus on the ubiquitous PDE-based problem class suggests. TOPS solvers are also effective for some systems with far less structure.  However, exploitation of an underlying field structure underlies TOPS’ eponymous quest for optimality.
 
TOPS has chosen to focus initially on SciDAC fusion (CEMM, CMRS) and astrophysics (TSI) applications in order to demonstrate success and to find stimuli to address end-to-end software issues that might be overlooked in models that abstract only the mathematical difficulties. These are multirate, multiscale, multicomponent, multiphysics applications.












Figure 1. Target applications for TOPS are multirate, multiscale, multicomponent, and multiphysics, e.g., supernovae (c/o TSI, left) and Hall reconnection (c/o CMRS, right).

Being multirate, they require implicit solvers to “step over” dynamically irrelevant but stability-limiting fast waves.  Multiscale implies fine grids, either uniformly or adaptively.  As multicomponent problems, they inherit natural blocking carrying special implications for data structures and cache locality, not to mention for algorithmic approaches.  As multiphysics problems, they are often approached through operator splitting in practice, and scalable software solutions should not ignore the investments embodied in legacy codes for them,but seek to incorporate them as preconditioners by keeping code as data structure-neutral as possible and using callbacks.
 
TOPS presents a multilayered interface to applications.  Users who wish to pass matrix elements and right-hand sides to the solver may do so, and still take advantage of the algebraic multigrid (AMG) solvers of the Hypre library to recover the discarded field structure and find optimal coarsenings. Users who linearize but can supply more elemental or grid information can take advantage of solvers at the next level.
 
For eventual purposes of sensitivity analysis and optimization, TOPS encourages users to migrate yet a level higher and interface with a Jacobian-free Newton-Krylov (JFNK) solver, to which they supply pointers to a subroutine that is called to evaluate a high-fidelity nonlinear residual and to a second subroutine to precondition its Jacobian (which need never be explicitly computed).  These subroutines, in turn, might be created using automatic differentiation. At the highest level, a user might submit a modified problem preconditioned with nonlinear Schwarz.  For time-dependent problems, TOPS solvers are callable by the user on each user-governed timestep, or the user may call an adaptive implicit integrator, which calls a rootfinder on each timestep.
 
Eventually, many SciDAC applications codes may not call TOPS solvers directly at all.  APDEC and TSTT software may perform the discretization and, in turn, call TOPS for implicit solves of subsystems, such as scalar Poisson problems on AMR grids, or multicomponent blocked nonlinear problems on overlaid grids.
 
In its first year, TOPS has supported the CEMM fusion code M3D through a matrix-element interface, using AMG-GMRES to solve linearized, operator-split unstructured scalar problems on 2D poloidal crossplane grids at each timestep. M3D’s previous domain-decomposed iteration was adequate for today’s problem sizes but does not scale to fine grids, as does AMG preconditioning. This exploits PETSc-Hypre interoperability. TOPS and APDEC have collaborated on Chombo-Hypre interoperability to provide CEMM a scalable adaptive mesh solver.
 
With TSI, TOPS collaborated at the level of coupled field linear problems from AGILE-BOLTZTRAN, using an operator-split preconditioner with separate phases of point-first and component-first orderings.  These problems, with large blocks due to multigroup radiation, are unidimensional. For the 2D and 3D future of TSI, TOPS’ parallel multigrid solvers are needed.
 
CMRS provided TOPS with a full time-dependent 2D Hall magnetic reconnection discretization, on which TOPS has demonstrated JFNK, and will evaluate the benefit of fully implicit temporal integration. TOPS expects eventually to take all three codes to 3D implicit formulations with high-order discretizations on adaptive grids.
The TOPS webpage may be found at http://www.tops-scidac.org .

To view this page as a pdf file, click here

For further information on this subject contact:
 Prof. David E. Keyes, Project Lead
 Old Dominion University
 Phone: 757-683-3882
dkeyes@odu.edu