Automated hyper-parameter optimization for deep learning framework to simulate boundary condi-

tions for wave propagation

Harpreet Kaur*, Sergey Fomel, and Nam Pham, The University of Texas at Austin

SUMMARY egories of hyper-parameters such as discrete (e.g., whether to

use early stopping), continuous (e.g., learning rate), categori-
We propose a hyper-parameter optimization workflow for cal (e.g., type of optimizer), etc.
training the deep learning framework to simulate the effect of
boundary conditions for wave propagation. Hyper-parameter Several studies proposed different categories of HPO meth-
selection is a crucial step in model building and has a direct ods such as decision theoretic methods, Bayesian optimiza-
impact on the performance of machine learning models. We tion models, and multifidelity optimization techniques. These
implement three different hyper-parameter optimization tech- methods have their own advantages and limitations. Decision
niques, namely random search, Hyperband, and Bayesian opti- theoretic methods (Bergstra et al., 2011; Bergstra and Bengio,
mization, for the proposed network to simulate boundary con- 2012) define a hyper-parameter search space and determine the
ditions and compare the strengths and drawbacks of these tech- best performing hyper-parameter combination by performing
niques. The automated deep learning framework optimizes either an exhaustive search using grid search methods or a ran-
network training and significantly improves the efficiency of dom selection using random search methods. These methods
the workflow. The proposed method reduces human effort in treat each parameter configuration independently and may re-
the network tuning process, improves the performance of deep quire considerable time and space, especially the exhaustive
learning models by achieving the optimal minima, makes the grid search methods. Multifidelity optimization algorithms (Li
model more reproducible, and can be extended to different et al., 2017) like Hyperband address the limited resource is-
deep learning based applications. Tests on different models sue and in each iteration eliminates poorly performing hyper-
verify the effectiveness of the proposed approach. parameter combinations to save computation time. However,
similar to decision theoretic approaches, Hyperband also per-
forms each evaluation independently of the previous evalua-
tions and can thus perform unnecessary function evaluations in
INTRODUCTION poorly performing regions. This issue can be mitigated by us-
ing Bayesian optimization (Eggensperger et al., 2013), which
Deep learning algorithms have demonstrated high performance uses the history of previous evaluations to determine the next
in both research and commercial applications (Yu and Zhu, configuration. Bayesian models belong to the category of se-
2020). Different deep learning models are suitable for differ- quential models and are difficult to parallelize; however, they
ent types of problems and datasets. During the model tun- can usually converge to optimal hyper-parameter combinations
ing process, different model configuration arguments known within a few iterations.
as hyper-parameters are specified by the user to tune the net-
work for a given problem statement. Hyper-parameters must In this work, we propose an HPO workflow for a deep learning
be set prior to model training and can not be updated during the framework to simulate boundary conditions for wave propaga-
training process. Selecting the best hyper-parameters for mod- tion. We perform a comparative analysis between three differ-
els has a direct impact on the performance of the model (Yang ent HPO algorithms, namely random search methods, which
and Shami, 2020). Conventionally, hyper-parameters are man- are decision theoretic approaches, Hyperband optimization, wh-
ually tuned by using an iterative trial and error based approach ich is a multifidelity optimization algorithm, and Bayesian op-
(Zöller and Huber, 2021). However, manual tuning is inef- timization. We demonstrate that Bayesian optimization is more
ficient because of large number of hyper-parameters in com- efficient and leads to a better performance in the testing phase
plex models, nonlinear hyper-parameter interactions, and time- than random search or Hyperband optimization. Although in
consuming model evaluations (Yang and Shami, 2020). These this paper we demonstrate the HPO workflow on boundary
factors have inspired research on automated tuning of hyper- condition simulation problem, the proposed workflow can eas-
parameters known as hyper-parameter optimization (HPO). ily be integrated into a variety of deep learning frameworks
for different problem statements. Using numerical models of
HPO has multiple advantages over the manual hyper-parameter increasing complexity, we demonstrate that the proposed ap-
tuning process (Kaur et al., 2021). It reduces the considerable proach is efficient and converges to an optimal minima as op-
human effort in manual tuning of a large number of hyper- posed to the manual tuning of hyper-parameters.
parameters, especially for complex models and large datasets.
It improves the accuracy and efficiency of the models and makes
the models and research more reproducible (Yang and Shami, HYPER-PARAMETER OPTIMIZATION
2020; Yu and Zhu, 2020). HPO problems need specialized
optimization techniques because they can sometimes be non- We implement the HPO workflow for a deep learning frame-
convex and non-differentiable; therefore, traditional optimiza- work using UNet architecture (Ronneberger et al., 2015). We
tion techniques such as gradient descent may result in local optimize two different categories of hyper-parameters, i.e., de-
instead of global minima (Luo, 2016). In addition, HPO algo- sign hyper-parameters related to the construction of a deep
rithms should also be able to effectively handle different cat- learning model (number of filters, activation function, etc.) and

Second International Meeting for Applied Geoscience & Energy 10.1190/image2022-3743090.1

© 2022 Society of Exploration Geophysicists and the American Association of Petroleum Geologists Page 1935
optimizer hyper-parameters related to the training process of hyper-parameter configurations and their allocated budget by
the model (learning rate, dropout rate, mini-batch size, num- discarding the poorly performing configurations in each itera-
ber of epochs, etc.). For the design-based hyper-parameters, tion using a successive halving method and passing only well
we provide several filters and activation functions, and the op- performing configurations to the next iteration. We compare
timization process selects the best combination on the basis of test results in the next section.
the complexity of the problem. Next we specify the search 3) Bayesian optimization- Earlier mentioned algorithms in-
space for following optimizer hyper-parameters: dependently process each hyper-parameter configuration and
Learning rate - It is one of the most important hyper-parameters may thus end up wasting a lot of computational time in poorly
in a deep learning model, which defines step size for the con- performing regions of the search space. To circumvent this
vergence of the iterative process. A high learning rate can ac- issue, we implement Bayesian optimization. Unlike earlier
celerate the training process but may also oscillate the gradi- mentioned algorithms, Bayesian optimization determines the
ents and hinder convergence, whereas a low learning rate leads next hyper-parameter configuration on the basis of previously
to smooth convergence with a longer training time. We pro- tested configurations and, thus, given appropriate resources, it
vide a range of learning rates (Table 1) and the optimization assures convergence toward the optimal configuration (Yang
algorithm selects the appropriate learning rate to converge the and Shami, 2020). Bayesian optimization has two key com-
objective function to global minima. ponents: a surrogate model that fits observed points into the
Dropout rate - Dropout is a method of stochastic regulariza- objective function and an acquisition function that provides
tion, which prevents overfitting and enables the neural network a trade-off between exploration and exploitation regions. We
to learn robust features (Srivastava et al., 2014). It is a compu- first obtain the predictive distribution for the probabilistic sur-
tationally efficient method to simulate an ensemble of neural rogate model. Next we use the acquisition function for explo-
networks by randomly dropping neurons during the training ration where we sample instances in regions that have not been
process. We use a dropout range of 0.0 to 0.5 with a step size of previously sampled, as well as exploitation, where we sample
0.05, and the optimization algorithm finds the optimal dropout the most promising regions with higher likelihood of global
rate for the neural network training. optima. The advantage of using Bayesian optimization is that
Batch size - Batch size is the number of training samples used it can detect the optimal hyper-parameter configuration on the
to estimate the gradient. Selection of batch size impacts train- basis of previously tested values. However, dependence on the
ing stability and generalization performance (Bengio, 2012). previously tested values makes the model sequential and diffi-
Smaller batch size offers a regularization effect and lowers cult to parallelize.
generalization error. We incorporate different batch sizes in We train the network in two steps: hyper-parameter optimiza-
the search space for the optimization algorithm, and the work- tion and parameter tuning, where parameter tuning updates
flow automatically selects the optimal batch size for training. weights and biases. We create training labels using different
isotropic and anisotropic models with different source wavelet
Table 1: Hyper-parameters and their respective values. frequencies using 2,500 time slices from 5 shot locations at
the corner and in the center of the model. We train the net-
Filters Activation Batch Dropout Learning work to learn the mapping between the bounded model con-
functions size rate rate sisting of spurious reflections and the padded model used to
32-64 Sigmoid 8-16 0-0.5 0.001-0.01 simulate the unbounded domain (Figures1). We first define
ReLu, tanh hyper-parameters for the deep learning model, along with the
search space. Next, we implement hyper-parameter optimiza-
tion workflow using random search, Hyperband, and Bayesian
We initiate the optimization process using the earlier defined
optimization workflows with 20 trials. After optimizing the
search space, and optimize the hyper-parameters using three
hyper-parameters, we pick the best hyper-parameter configu-
optimization algorithms:
ration from the leader-board of models (green line in Figure 2)
1) Decision theoretic approach-The first algorithm that we im-
and use it to tune the parameter for network training. We in-
plement is a random search, which is a type of decision theo-
corporate K-fold cross-validation in the training process to an-
retic method. Random search algorithm randomly selects the
alyze the generalization ability of the proposed model. For this
hyper-parameter configurations in the search space defined by
work, we choose K = 5, which as shown by Kuhn et al. (2013),
the user and trains these candidate values until the defined tri-
has low bias and modest variance. We fit the model using K −1
als are exhausted. It can cover a large search space; however,
folds and validate the data with the remaining Kth fold. We re-
it treats each evaluation independently and does not keep track
peat this process until every Kth-fold serves as the test set, and
of the previously well-performing regions. We implement ran-
we note down the error for each fold. The mean square of
dom search with 20 trials, and discuss the test results of the
errors (MSE) from all iterations gives us theP cross-validation
best performing model in the next section.
(CV) performance metric given as CVk = 1k ki=1 MSEi . We
2) Multifidelity optimization-We further improve the efficiency
obtain values of 0.0027, 0.0023, 0.0035, 0.0027, and 0.0029
of the random search algorithm by using multifidelity opti-
for different folds with an average value of 0.0028, which in-
mization. One of the issues with random search method is that
dicates that we do not have a split bias and the network gen-
with a larger hyper-parameter configuration space, the longer
eralizes well on the unseen data. During training, the network
execution time may not let the algorithm converge to global
learns to attenuate spurious boundary reflections. Then we test
minima if the resources are limited. To overcome this limita-
the trained network using the best hyper-parameter configura-
tion, Hyperband achieves a trade-off between the number of

Second International Meeting for Applied Geoscience & Energy 10.1190/image2022-3743090.1

© 2022 Society of Exploration Geophysicists and the American Association of Petroleum Geologists Page 1936
tion by using the three optimization algorithms and perform a
comparative analysis during the testing phase in the following
examples.

Figure 2: HPO with Bayesian optimization workflow. Op-

(a) (b) timization workflow considers dynamic interaction between
hyper-parameters and converges to optimal minima. Green
Figure 1: One of the training slices for HPO and parameter tun- line shows best combination of hyper-parameters (with low-
ing. The network learns mapping from wavefield in bounded est loss) and red line shows worst performing hyper-parameter
domain (a) to the wavefield in unbounded domain simulated by combination (with highest loss).
padded model (b) using different HPO workflows. The trained
network is then tested for comparative analysis between differ-
ent HPO algorithms.
1994). The model is discretized on a 376 × 384 grid with
a spatial sampling of 25-m along horizontal and vertical di-
rections (Kaur et al., 2019, 2021). We simulate the wavefield
EXAMPLES using a low-rank extrapolation operator (Fomel et al., 2013)
with a 15-Hz Ricker wavelet and a time step of 2 ms. The test
Gradient velocity model wavefields with spurious reflections for two different shot lo-
We first use a synthetic model with large velocity variations cations are shown in Figures 4a and 4d. The output using the
(Figure 3a). The model is discretized on a 400 × 400 grid with Bayesian optimization workflow (Figures 4b and 4e) is free of
a 5-m spacing along vertical and horizontal directions (Sun the boundary reflections and wrap-arounds and is close to the
et al., 2016). We simulate the wavefield by using a low-rank output using the unbounded media simulated by padded model
extrapolation operator (Fomel et al., 2013) with a 20-Hz Ricker (Figures 4c and 4f).
wavelet and a time step of 2 ms. A blind test slices is shown
in Figure 3b, and the ground truth (not seen by the network) Table 2: SSIM and correlation coefficient for test case: Com-
is shown in Figure 3c. We test the trained network to ana- parative analysis between three HPO algorithms.
lyze output using the three HPO algorithms, namely, random
search, Hyperband, and Bayesian optimization (Figures 3d, 3e, Metrics Random Search Hyperband Bayesian
and 3f) with a constrained number of trials. The test output us- SSIM 0.87 0.95 0.97
ing the random search algorithm needs further improvement R2 0.87 0.96 0.98
(marked by the red arrow in Figure 3d). Next, we analyze the
test results using Hyperband algorithm (Figure 3e). Hyperband
eliminates poorly performing hyper-parameter configurations
in each run, and, therefore, in a given number of trials it can CONCLUSIONS
converge to near optimal minima, as compared to the random
search method. However, we still have some remnant spuri- We have developed an automated hyper-parameter optimiza-
ous reflections (red arrows in Figure 3e). Next, we test the tion workflow for a deep learning framework. We train the
Bayesian optimization model (Figure 3f). In the same number network using the best picked model from the hyper-parameter
of trials as the other two methods, the Bayesian optimization optimization workflow and perform a comparative analysis be-
model converges to an optimal minima, and the output is close tween three different optimization algorithms; random search,
to the ground truth (Figure 3c). To quantify the output using Hyperband, and Bayesian optimization. We demonstrate that
the three optimization methods, we compute the structural sim- for a given search space and computational resources, Bayesian
ilarity index (SSIM) (Wang et al., 2004) and correlation coeffi- optimization performs better than random search or Hyper-
cient (R2 ) (Table 2) for the test cases, which indicates that the band because for each hyper-parameter configuration, it takes
Bayesian optimization determines a more appropriate hyper- into account the previous evaluation, which helps convergence
parameter configuration because it takes into account previous within a few iterations. The proposed algorithm reduces man-
evaluations, and with each iteration it moves closer to the op- ual input into training process, optimizes values of hyper-para-
timal minima. meters for a deep learning framework, and thus maximizes the
Marmousi Model predictive accuracy of deep learning models. We show appli-
In the previous example, we observed that Bayesian optimiza- cation of the proposed workflow for simulating the boundary
tion produces the best hyper-parameter configuration, and its condition for wave propagation; however, the proposed hyper-
output is close to ground truth. We test the best picked model parameter optimization workflow can easily be integrated into
using Bayeisan optimization on the Marmousi model (Versteeg, a variety of deep neural networks for different problems.

Second International Meeting for Applied Geoscience & Energy 10.1190/image2022-3743090.1

© 2022 Society of Exploration Geophysicists and the American Association of Petroleum Geologists Page 1937
 (a) (b) (c)

(d) (e) (f)

Figure 3: Blind-test results for gradient velocity model at t=1.82 s (shot locations and time slices are not part of training); (a) gradient
velocity model, (b) wavefield time slice for bounded media with boundary reflections and wrap-arounds, (c) wavefield time slice
for unbounded media simulated using padded model. Output wavefields free from boundary reflections and wrap-arounds using
(d) random search optimization, (e) Hyperband optimization, and (f) Bayesian optimization. Red arrows indicate the regions where
random search and Hyperband optimization needs further improvement.

(a) (b) (c)

(d) (e) (f)

Figure 4: Marmousi model test data for network trained using Bayesian optimization. Wavefield snapshot at t=1.47 s: (a) wavefield
snapshot with boundary reflections, (b) wavefield snapshot using proposed method, and (c) wavefield snapshot without boundary
reflections in unbounded media simulated by padding the model. Wavefield snapshot at t=1.524 s for different blind-test shot loca-
tion: (d) wavefield snapshot with boundary reflections, (e) wavefield snapshot using proposed method, and (f) wavefield snapshot
without boundary reflections in unbounded media simulated by padding the model.

Second International Meeting for Applied Geoscience & Energy 10.1190/image2022-3743090.1

© 2022 Society of Exploration Geophysicists and the American Association of Petroleum Geologists Page 1938
References
Bengio, Y., 2012, Practical recommendations for gradient-based training of deep architectures: Springer.
Bergstra, J., R. Bardenet, Y. Bengio, and B. Kegl, 2011, Algorithms for hyper-parameter optimization: Advances in Neural Information Processing
Systems, 24, 10495258.
Bergstra, J., and Y. Bengio, 2012, Random search for hyper-parameter optimization: Journal of Machine Learning Research, 13, 281−305.
Eggensperger, K., M. Feurer, F. Hutter, J. Bergstra, J. Snoek, H. Hoos, and K. Leyton-Brown, 2013, Towards an empirical foundation for assessing
Bayesian optimization of hyperparameters: Presented at the NIPS workshop on Bayesian Optimization in Theory and Practice.
Fomel, S., L. Ying, and X. Song, 2013, Seismic wave extrapolation using lowrank symbol approximation: Geophysical Prospecting, 61, 526–536, doi:
[Link]
Kaur, H., S. Fomel, and N. Pham, 2019, Overcoming numerical dispersion of finite-difference wave extrapolation using deep learning: 89th Annual
International Meeting, SEG, Expanded Abstracts, 2318–2322, doi: [Link]
Kaur, H., S. Fomel, and N. Pham, 2021, Boundary conditions for acoustic and elastic wave propagation using deep learning: First International
Meeting for Applied Geoscience & Energy, SEG/AAPG, Expanded Abstracts, 1390–1394, doi: [Link]
Kuhn, M., and K. Johnson, 2013, Applied predictive modeling: Springer.
Li, L., K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar, 2017, Hyperband: A novel bandit-based approach to hyperparameter opti-
mization: The Journal of Machine Learning Research, 18, 6765–6816, doi: [Link]
Luo, G., 2016, A review of automatic selection methods for machine learning algorithms and hyper-parameter values: Network Modeling Analysis in
Health Informatics and Bioinformatics, 5, 1–16, doi: [Link]
Ronneberger, O., P. Fischer, and T. Brox, 2015, U-net: Convolutional networks for biomedical image segmentation: International Conference on
Medical Image Computing and Computer-assisted Intervention, 234–241, doi: [Link]
Srivastava, N., G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, 2014, Dropout: a simple way to prevent neural networks from over-
fitting: The Journal of Machine Learning Research, 15, 1929–1958.
Sun, J., S. Fomel, and L. Ying, 2016, Low-rank one-step wave extrapolation for reverse time migration: Geophysics, 81, S39–S54, doi: [Link]
10.1190/geo2015-0183.1.
Versteeg, R., 1994, The Marmousi experience: Velocity model determination on a synthetic complex data set: The Leading Edge, 13, 927–936, doi:
[Link]
Wang, Z., A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, 2004, Image quality assessment: from error visibility to structural similarity: IEEE
Transactions on Image Processing, 13, 600–612, doi: [Link]
Yang, L., and A. Shami, 2020, On hyperparameter optimization of machine learning algorithms: Theory and practice: Neurocomputing, 415, 295–
316, doi: [Link]
Yu, T., and H. Zhu, 2020, Hyper-parameter optimization: A review of algorithms and applications: arXiv preprint arXiv:2003.05689.
Zöller, M.-A., and M. F. Huber, 2021, Benchmark and survey of automated machine learning frameworks: Journal of Artificial Intelligence Research,
70, 409–472, doi: [Link]

Second International Meeting for Applied Geoscience & Energy 10.1190/image2022-3743090.1

© 2022 Society of Exploration Geophysicists and the American Association of Petroleum Geologists Page 1939