Advances in Parallel Electromagnetic Codes for Accelerator Science and Development*

Over a decade of concerted effort in code development for accelerator applications has resulted in a new set of electromagnetic codes which are based on higher-order finite elements for superior geometry fidelity and better solution accuracy. SLAC’s ACE3P code suite is designed to harness the power of massively parallel computers to tackle large complex problems with the increased memory and solve them at greater speed. The US DOE supports the computational science R&D under the SciDAC project to improve the scalability of ACE3P, and provides the high performance computing resources needed for the applications. This paper summarizes the advances in the ACE3P set of codes, explains the capabilities of the modules, and presents results from selected applications covering a range of problems in accelerator science and development important to the Office of Science. 3D ELECTROMAGNETIC SOFTWARE The advent of 3D electromagnetic (EM) codes for accelerator applications can be credited to Thomas Weiland who developed the "Finite Integration Technique" (FIT) and implemented in the MAFIA code suite based on a finite difference (FD) structured grid, which is now the commercial package Microwave studio by CST GmbH. Also FD based is GdfidL which is a parallel code for time varying and resonant fields. Other common commercial codes for high frequency simulation include HFSS from Ansoft and ANSYS from ANSYS, Inc. using the finite element (FE) method on an unstructured mesh. All these codes are available on parallel platforms. This paper presents the advances in the ACE3P (Advanced Computational Electromagnetic 3D Parallel) software which is a higher-order FE code suite developed at SLAC, capable of using massively parallel computations (>10k CPUs) to model large accelerator structures with higher accuracy. PARALLEL CODE DEVELOPMENT AT SLAC The code development effort of ACE3P started with a PhD thesis research [1] more than a decade ago to explore the parallel FE approach for electromagnetics in accelerator modeling. This led to the support from DOE’s High Performance Computing (HPC) programs under the Accelerator Grand Challenge (1998–2001), followed by the Accelerator Science and Technology (AST) project under the Scientific Discovery through Advanced Computation SciDAC-1 program (2001-2007), and continuing as the Community Petascale Project for Accelerator Science and Technology under SciDAC-2 (2007-2012) [2,3,4]. The motivation to develop highly accurate codes for modeling complex accelerator structures originated from the machine R&D for the ILC for which SLAC is a principal proponent. To meet the requirements for beam stability, the dimensions of the ILC accelerator cavity shown in Fig. 1 need to be modeled to 0.01 % accuracy in frequency so that tuning in the fabrication process can be avoided to reduce cost. There are several challenges to this task: (1) complexity of the HOM coupler with fine feature versus the cavity size, (2) problem size of multicavity structure and cryomodule, (3) required accuracy of 10s of kHz mode in a GHz range, and (4) speed for fast turnaround time to impact design. Figure 1. The ILC SRF cavity. PARALLEL HIGHER-ORDER FE METHOD In view of the ILC cavity modeling requirements, the ACE3P codes are developed based on the parallel higherorder FE method. A key advantage of the conformal (tetrahedral) mesh over the FD structured mesh is geometry fidelity as shown in Fig. 2a for the example of a coupler cavity. Further accuracy can be obtained through the use of quadratic surface and higher-order elements (p = 1-6) resulting in reduced computational cost as seen in Fig. 2b. An equaling important advantage is the power of parallel processing which both increases memory and speed, allowing large problems to be solved in far less time through scalability. Finally but not least, the success of large-scale simulations through high performance computing (HPC) relies heavily on the computational science research funded by SciDAC to improve code scalability and produce the desired results. Figure 2. (a) Tetrahedral mesh of a coupler cavity; (b) Frequency convergence of a cavity mode in terms of computer memory usage using p = 1, 2 and 3 basis functions in elements. SLAC-PUB-14349