Download Title Seismic modeling with Discontinuous Galerkin Finite

Title
Seismic modeling with Discontinuous Galerkin Finite-Element method: application to large scale 3D
elastic media
Author
V. Etienne, J. Virieux, S. Operto
Abstract
We present a Discontinuous Galerkin (DG) finite-element formulation with convolutional perfectly matched
layer (CPML) absorbing condition suitable to large scale seismic modelling in 3D elastic media. The
issues of performance and load balancing of the proposed scheme are discussed in details. A special
attention is given to the last point, a critical issue in applications based on a parallel architecture with
MPI communications like most of geophysical applications. When dealing with large scale modelling, it
is of first importance to seek for an efficient and inexpensive absorbing condition. Thanks to the mix of
interpolation orders which is a key feature of the DG method, we get a good compromise by combining
the DG P1 scheme in the medium to get a acceptable accuracy and the DG P0 scheme in the absorbing
layers to reduce the whole numerical cost. A good load balancing is observed between the processes
without paying any effort on the subdomain decomposition.
71st EAGE Conference & Exhibition — Amsterdam, The Netherlands, 8 - 11 June 2009
Introduction
The simulation of wave propagation in complex medium has been efficiently tackled with finite-difference
(FD) methods and applied with success to numerous physical problems since the past decades. Nevertheless, FD methods suffer from some critical issues inherent to the underlying Cartesian grid such as
parasite diffractions in case of boundaries with complex topography. Therefore and thanks to the ever
increasing computation power, others methods have focused a lot of interests. Concerning the 3D elastic
seismic wave equation, we can mention the Spectral Element method (SEM) popularized by Komatitsch
and Tromp (1999) and more recently the Discontinuous Galerkin (DG) method (Dumbser and Käser,
2006) which has been proved to give accurate results on tetrahedral meshes.
Here, we present a DG formulation with the convolutional perfectly matched layer (CPML) absorbing
condition (Komatitsch and Martin, 2007) suitable to large scale 3D seismic simulations. In this context,
the DG method provides major benefits. Thanks to tetrahedral meshes, one can fit almost perfectly complex topographies or geological discontinuities and the discretisation can be adapted locally to media
properties. Moreover, the DG method is completely local and therefore suitable for parallelization. We
should also mention the possiblity to mix different interpolation orders. The main drawback of the DG is
its relatively high numerical cost due to the fact that elements do not share their nodal values contrary to
classical continuous finite-element methods. In the following, we take benefit of the mix of interpolation
orders to achieve an absorbing condition with a reduced numerical cost by using low order in the absorbing layer. In the next sections, we introduce the DG method with the CPML formulation, we discuss
the computing aspects regarding the proposed scheme and we illustrate the efficiency of the proposed
method with a target of the SEG/EAGE overthrust model.
DG formulation with CPML
We apply the DG method to the 3D elastodynamic velocity-stress first-order system in the time domain
for an isotropic medium in a conservative form,
X
X
−
→
ρ∂t~v =
∂α (Mα~σ ) +
Mα ψα (~σ ) + f~0
α∈{x,y,z}
Λ0 ∂t~σ =
X
α∈{x,y,z}
X
∂α (Nα~v ) +
−
→
Nα ψα (~v ) + Λ0 ∂t σ~0 ,
(1)
α∈{x,y,z}
α∈{x,y,z}
where ~v is the particle velocity vector, ~σ the stress vector, ρ the density and Λ0 a matrix containing the
physical properties of the medium. Mα and Nα are matrices with real factors. In the DG formulation, the
system (1) is multiplied by a test function and integrated in the volume. We solve so-called weak form of
the system using tetrahedral cells and an explicit leap-frog scheme in time. The evolution of the system is
governed by the exchange of numerical fluxes between adjacent cells at each time step. Following the approach of BenJemaa et al. (2007), we adopt the centred flux scheme for its non-dissipative characteristic.
Moreover, we have introduced in the system (1) the CPML formulation developped by Komatitsch and
Martin (2007) which improves the behaviour of classical PML absorbing conditions (Berenger (1994))
at grazing incidence. The CPML does not require to split the elastodynamic equations, instead it makes
use of memory variables ψα which are updated at each time step according to the recursive expression,
ψαn = bα ψαn−1 + aα (∂α )n
∀α ∈ {x, y, z},
(2)
with n the time step and aα , bα some parameters depending on the time step and the length of the
absorbing layer. One memory variable is required for each partial derivative in the system (1) making
22 variables per degree of freedom in the CPML layer. We adopt the nodal form on the DG formulation
(Hesthaven and Warburton, 2008), assuming that each component of the stress or velocity vector is
decomposed as,
ndof
ndof
~vi =
X
j=1
vij ϕ
~ ij
~σi =
X
σij ϕ
~ ij ,
j=1
71st EAGE Conference & Exhibition — Amsterdam, The Netherlands, 8 - 11 June 2009
(3)
where i is the indice of the tetrahedral cell, ndof the number of degrees of freedom (or nodes), ϕ
~ ij the
interpolating Lagrange basis function at node j and vij and σij are the velocity and stress wavefields at
node j. For instance, in DG P0 , there is only one degree of freedom (the stress and velocity are constant
per cell) while in DG P1 , there are four degrees of freedom located at the 4 vertices of the tetrahedral cell
(the stress and velocity are linearly interpolated). Numerical experiments have shown that the DG P0
does not provide an acceptable solution in the 3D case due to its strong dependency to the mesh structure
and therefore one need to consider P1 as the lowest possible interpolation order to perform accurate
seismic modelling.
Computing aspects
In this section, we discuss the issues of performance and load balancing, a critical point in our application
based on a parallel architecture with MPI communications like most of geophysical applications. In table
1, we have reported the memory requirement and computation cost for updating the wavefield values in
a single cell during one time step. These simulations were performed on the IBM Blue Gene/P machine
with Power PC 450 CPUs of the IDRIS/CNRS. These figures are quite stable regardless the number of
MPI processes due to the high and intrinsic scalability of the DG method.
Cell type
Time per cell for one step Memory per cell
Ordinary P0 cell
1.6 µs
175 bytes
P0 cell inside CPML
2.5 µs
284 bytes
Ordinary P1 cell
6.2 µs
294 bytes
TABLE 1: memory and computation cost for updating the wavefield values in one cell
at one time step depending on the cell type
We first remark that a P0 cell inside the CPML costs about 60% more in time and memory than an
ordinary P0 cell due to the extra computation and the storage of the memory variables. We should also
note that the cost of a P1 cell is about 4 times more expensive in terms of computation time than the
cost of an ordinary P0 cell (and 2.8 times compared to P0 CPML cell) and approximately equivalent
regarding the memory to the cost of a P0 CPML cell. Likewise, the storage of the memory variables
in a P1 cell would require additionnal 264 bytes meaning a increase of 90% which should also reflect
the extra cost of the computation time. When dealing with large scale seismic modelling, it is of first
importance to keep the memory requirements and the computation time as low as possible and hence we
should seek for an efficient and inexpensive absorbing condition. The figures in table 1 indicate that a
good compromise should be met by combining the P1 scheme in the medium to get a acceptable accuracy
and the P0 scheme in the absorbing layers to reduce the whole numerical cost.
Numerical experiment in complex medium
In order to illustrate the proposed scheme, we considered a target of the SEG/EAGE overthrust model.
The dimensions of this target are 8 km x 8 km x 3.5 km (figure 1). An absorbing layer of 1 km width
has been added at the periphery of the model except on the top where a free surface condition has been
applied. We placed an explosive source in the middle of the model at 20 m depth and used a Ricker
wavelet with a mean frequency of 3 Hz. We built a regular mesh by dividing cubes into 5 tetrahedra and
set the cube size to tenth of the shortest propagated wavelength to reach the required precision. With a
maximum frequency of 9 Hz and a minimum S-wave velocity of 1300 m/s, the spatial discretisation is
15 m and the mesh contains more than 677 millions of tetrahedra. The simulations were performed on
the IDRIS/CNRS cluster with 4096 CPUs and a regular subdomain decomposition (figure 2). Table 2
gives the figures observed with DG P0 and DG P1 scheme when using in both cases a P0 CPML layer.
Order Time/cell Min mem/proc Max mem/proc Tot. mem Time step Nb step Tot. time
DG P0
2.4 µs
27.6 MB
44.9 MB
148 GB
0.668 ms
7489
49 min.
DG P1
6.0 µs
44.9 MB
46.4 MB
187 GB
0.200 ms 24965 6h 52 min.
TABLE 2: memory and computation cost for the simulations on the portion of the Overthurst model
with DG P0 and P1 interpolation in the medium and P0 CPML absorbing layers
71st EAGE Conference & Exhibition — Amsterdam, The Netherlands, 8 - 11 June 2009
1
2
3
Inline (km)
4
5
6
7
8
9
10
1
6000
5000
2
4000
3
m/s
Depth (km)
0
0
3000
4
2000
Figure 1: Section of the SEG/EAGE model (P-wave velocity) in the source plane. The explosive source
is placed at the middle of the model at 20 m below the free surface. A CPML layer of 1 km width has
been added to the model, the limits of this layer are plotted with yellow lines.
Depth (km)
0
0
1
2
3
Inline (km)
4
5
6
7
8
9
10
1
2
3
4
Figure 2: Map of the domain decomposition in the source plane, each subdomain associated to one CPU
is plotted with a different color. The 4096 CPUs allow for a regular domain decomposition of 32 x 32 x
4 subdomains. With such decomposition, some of these subdomains are totally located in the absorbing
layers (the limits of these layers are plotted with black lines).
For the first simulation with only P0 cells, we note that the memory is not well balanced between the
subdomains and the average computation time per cell corresponds to the cost of the P0 cells located in
the CPML layers. In this case, all the processes located in the medium spent about 60% of time waiting
for the processes located in the CPML layers. This is the main bottleneck of parallel computing, the
slowest process penalises all the others. On the contrary, for the second simulation with a mix of P1 and
P0 cells, we observe a good memory balance between the processes and the average computation time
per cell corresponds to the cost of the P1 cells located in the medium. In the second case, the absorbing
layers do not penalise the whole simulation. A series of snapshots extracted from the second simulation
are illustrated in figure (3).
Conclusions and perspectives
We have proposed a DG formulation with a low cost absorbing condition which is a crucial issue to
perform large scale seismic modelling. We obtain a good load balancing between the processes without
paying any effort on the subdomain decomposition. Nevertheless, the figures in table 2 still show a
relatively high numerical cost considering the size of the model. This has to be analysed and we should
mention that the regular mesh in the presented simulations was only used for test purpose and clarity
of demonstration. Obviously, the mesh needs to be tuned locally to the medium properties in order to
decrease significantly the number of cells and hence the global cost. For instance there is a factor of 3
between the lowest and highest velocities in the SEG/EAGE Overthrust model. The proposed scheme
will still pay off on unstructured meshes as the load balancing is achieved naturally. Another option for
reducing the CPU time, is to explore higher orders of interpolation to seek for the optimum numerical
scheme. For a complete analysis, a comparison of the DG formulation should be done against methods
71st EAGE Conference & Exhibition — Amsterdam, The Netherlands, 8 - 11 June 2009
2
Inline (km)
4
6
8
10
0
2
4
0
2
2
Inline (km)
4
6
8
10
0
0
2
Inline (km)
4
6
8
10
2
0
10
0
2
4
0
2
Inline (km)
4
6
2
4
1.5 second
10
0
2
Inline (km)
4
6
8
10
8
10
2
4.0 second
8
10
0
Depth (km)
8
8
2
2.5 second
Depth (km)
Inline (km)
4
6
Inline (km)
4
6
4
1.0 second
2
2
3.5 second
4
0
0
4
Depth (km)
0
Depth (km)
Depth (km)
0
2.0 second
4
Depth (km)
10
4
2
0
8
2
.5 second
0
Inline (km)
4
6
Depth (km)
0
Depth (km)
Depth (km)
0
0
2
Inline (km)
4
6
2
4
3.0 second
4.5 second
Figure 3: Series of snapshots in the source plane of the velocity component vx computed with the mix
DG P0 /P1 scheme. No spurious reflection at the model boundaries can be observed (the limits of the
CPML layers are plotted with yellow lines).
like FD or SEM in terms of performance and accuracy, knowing that the DG method will show its
efficiency in case of highly heterogeneous medium with or without complex topography.
Acknowledgments
This research was funded by the SEISCOPE consortium sponsored by BP, CGG-VERITAS, Exxon Mobil, SHELL and TOTAL. Access to the high performance computing facilities of IDRIS/CNRS computer
center provided the required computer ressources.
References
BenJemaa, M., Glinsky-Olivier, N., Cruz-Atienza, V. M., Virieux, J., and Piperno, S. [2007] Dynamic
non-planar crack rupture by a finite volume method. Geophysical Journal International 171, 271–285.
Berenger, J-P. [1994] A perfectly matched layer for absorption of electromagnetic waves. Journal of
Computational Physics 114, 185–200.
Dumbser, M., and Käser, M. [2006] An arbiratary high-order discontinuous Galerkin method for elastic waves on unstructured meshes - II. The three-dimentional isotropic case. Geophysical Journal
International 196(2), 319–336.
Hesthaven, Jan. S., and Warburton, Tim. [2008] Nodal Discontinuous Galerkin Method. Springer.
Komatitsch, D., and Martin, R. [2007] An unsplit convolutional perfectly matched layer improved at
grazing incidence for the seismic wave equation. Geophysics 72(5), SM155–SM167.
Komatitsch, D., and Tromp, J. [1999] Introduction to the spectral element method for 3D seismic wave
propagation. Geophys. J. Int. 139, 806–822.
71st EAGE Conference & Exhibition — Amsterdam, The Netherlands, 8 - 11 June 2009