Knowledge and Information Systems (2025) 67:4035–4085

[Link]

REVIEW

Data augmentation with automated machine learning:

approaches and performance comparison with classical data
augmentation methods

Alhassan Mumuni1 · Fuseini Mumuni2

Received: 11 January 2024 / Revised: 9 January 2025 / Accepted: 15 January 2025 /

Published online: 22 February 2025
© The Author(s), under exclusive licence to Springer-Verlag London Ltd., part of Springer Nature 2025

Abstract
Data augmentation is arguably the most important regularization technique commonly used
to improve generalization performance of machine learning models. It primarily involves
the application of appropriate data transformation operations to create new data samples
with desired properties. Despite its effectiveness, the process is often challenging because
of the time-consuming trial and error procedures for creating and testing different candidate
augmentations and their hyperparameters manually. State-of-the-art approaches are increas-
ingly relying on automated machine learning (AutoML) principles. This work presents a
comprehensive survey of AutoML-based data augmentation techniques. We discuss various
approaches for accomplishing data augmentation with AutoML, including data manipula-
tion, data integration and data synthesis techniques. The focus of this work is on image
data augmentation methods. Nonetheless, we cover other data modalities, especially in cases
where the specific data augmentations techniques being discussed are more suitable for these
other modalities. For instance, since automated data integration methods are more suitable
for tabular data, we cover tabular data in the discussion of data integration methods. The
work also presents extensive discussion of techniques for accomplishing each of the major
subtasks of the image data augmentation process: search space design, hyperparameter opti-
mization and model evaluation. Finally, we carried out an extensive comparison and analysis
of the performance of automated data augmentation techniques and state-of-the-art methods
based on classical augmentation approaches. The results show that AutoML methods for
data augmentation currently outperform state-of-the-art techniques based on conventional
approaches.

Keywords Data augmentation · AutoML · Automated machine learning · Machine

learning · Data preparation · Image augmentation

B Alhassan Mumuni
[Link]@[Link]
Fuseini Mumuni
fmumuni@[Link]
1 Department of Electrical and Electronics Engineering, Cape Coast Technical University,
Cape Coast, Ghana
2 University of Mines and Technology (UMaT), Tarkwa, Ghana

123
4036 A. Mumuni, F. Mumuni

1 Introduction

1.1 Background

Practical implementations of machine learning systems require large data samples to produce
satisfactory results. Since data are often not available in sufficient quantities, regularization
techniques are critical for achieving good performance. These techniques commonly entail
tweaking the machine learning model configuration or applying data augmentation—a range
of methods for extending the available data by applying appropriate transformations. The
basic idea is to modify training datasets by applying suitable transformations in ways that
increase the quantity, representation quality and variability of the original data.
The most commonly used data augmentation techniques include geometric transforma-
tions—particularly, rotation, flipping, shearing and scaling—and photometric transforma-
tions such as color jittering, solarizaion, brightness, contrast adjustment, noise addition,
denoising, and color space conversion. Data augmentation can also involve creating com-
pletely new data from scratch [1, 2]. This approach can be useful when the training data
for the target application is inaccessible [3]. Methods for synthetic data generation include
explicitly creating samples with desired data distribution using computer graphics tools ([2])
or algorithmically generating artificial data with the aid of special deep learning models
(e.g., using techniques such as differential neural rendering [4, 5]. Neural style transfer [6],
generative modeling techniques such as variational autoencoders (VAEs) [7] and generative
adversarial networks (GANs) [8] have also been extensively used to generate synthetic data
for training deep learning models. Another way to augment training data is by integrating
existing data from several sources (e.g., see [9–11]). This is a useful and a more natural way
to augment training data since the approach can leverage the large quantities of data available
in various forms on the internet and other sources.

1.2 Limitations of classical data augmentation approaches

While classical data augmentation methods can significantly improve the predictive perfor-
mance of machine learning models, they are typically characterized by laborious manual
work. The process of generating and finding the best augmentations for a particular dataset
or task is a combinatorial problem, which requires infinitely large number of permutations of
different settings to be tested for a suitable method to be found. However, for a given dataset,
the number of unique augmentations that can be obtained with manual effort is generally
limited. Moreover, it is known that different types of augmentations work well for different
machine learning tasks, and determining a suitable augmentation methods for a particular
task is a nontrivial task.
Moreover, approaches that improve generalization on one dataset may fail to transfer
to other datasets. For instance, Lopes et al. [12] demonstrated that CutOut [13] improves
performance on CIFAR-10 but not on ImageNet dataset. Also, Raileanu et al. [14] argue that
classical data augmentation approaches do not readily work well for reinforcement learning
(RL) tasks. Generative modeling techniques such as VAEs and GANs have shown promise
in generating synthetic data to alleviate data shortage problems but they also suffer from
overfitting when trained on insufficient data.
Approaches based on GANs are also not guaranteed to produce good results even in
cases where sufficiently large and rich datasets are available [15]. Moreover, since generative
modeling-based approaches constrain the distribution of generated samples in some way

123
Data augmentation with automated… 4037

(e.g., by requiring some form of similarity of the target domain and generated samples),
many potentially useful augmentations that do not satisfy these constrains may be missed.
AutoML approaches avoid this limitation by solving the problem in a similarity-independent
manner.
The foregoing discussion shows that it is extremely challenging to achieve optimal aug-
mentation results using traditional data augmentation methods. Even in situations where
restrictions on the range of augmentations achievable do not exist, it is still difficult to obtain
good results without excessive trial and error work. Recently, automated algorithmic solutions
have been proposed as a means of simplifying the cumbersome data augmentation process.
For example, transferable AutoMl (Tr-AutoML) [16] is specifically designed to allow learned
features to be transferable to novel task domains. The approach combines meta-learning and
architecture search for feature extraction from multiple but related datasets. Cubuk et al.
[17] show that AutoML-based augmentations are also transferable to new datasets. This
could potentially provide a means for automatically learning useful augmentations that are
independent of datasets for different machine learning tasks.

1.3 Automated machine learning and data augmentation

Developing a machine learning model involves a series of tedious and repetitive tasks: data
preparation, hyperparameter selection, model selection, hyperparameter optimization, model
tuning and evaluation of the resulting outcome. For a single machine learning tasks, all these
steps are usually performed repeatedly until satisfactory results are achieved. This requires
human experts to manually perform each of the subtasks (Fig. 1A). The task is inherently
a combinatorial problem, as different types of hyperparameter settings would need to be
configured and tested. Consequently, with the manual method it is often impractical to find
an optimal solution due to the enormous variety of settings involved.
Automated machine learning (AutoML) [18] is an approach to automate all the processes
of designing, training, deploying and monitoring machine learning solutions. AutoML frame-
works can, for example, carry out data augmentation, perform additional processing and
feature engineering, and construct the network structure of the machine learning model. The
general idea of automated data augmentation is to create different types of basic transforma-
tions functions (e.g., rotations, flipping, color jittering, solarizaion, scaling, etc.) and then,
using AutoML techniques, algorithmically apply various combinations of operations on the
data and select the most effective set of data augmentation operations. Typically, black-
box optimization techniques are used to find the best augmentation strategies. The search
operation needs to find not only relevant transformations but also the optimal levels of trans-
formations. For image augmentation these levels may be rotation angles, translation offsets
and saturation values. Thus, automated data augmentation is mainly a task of combinatorial
optimization of primitive data transformation operations. All these processes are carried out
without the intervention of a human developer (see Fig. 1B). Given a task, dataset and perfor-
mance objective, automated data augmentation produces optimal data augmentation policies
that are applied on the dataset and trained in an end-to-end manner (i.e., from input data to
final stage).

1.4 Motivation for this survey

The concept of automated machine learning (AutoML) has received enormous attention in
recent years. One of its most important applications is in data preparation, which includes data

123
4038 A. Mumuni, F. Mumuni

Fig. 1 Classical deep learning versus AutoML: In classical deep learning A, all stages of the machine learning
task-data preparation, hyperparameter selection and tuning, model selection and tweaking as well as the
evaluation and validation of outcomes—are performed manually. In contrast, AutoML B incorporates an
automatic tuning mechanism to learn the best parameters and hyperparameters for all these tasks

pre-processing and data augmentation. Despite there being many surveys on data augmen-
tation methods, very few works have discussed automated data augmentation in sufficient
detail. Among the many surveys on AutoML techniques, only a few (e.g., [19–23]) have
discussed the application of AutoML to solve data augmentation problems. Unfortunately,
the coverage of automated data augmentation in theses works is very limited in scope and
depth. To the best of our knowledge, only Yang et al. [24] and Cheung [25] have presented
surveys that are specifically dedicated to automated data augmentation methods. However,
these works do not cover many important aspects of AutoML approaches in data augmen-
tation settings, including automated data integration and synthetic data generation methods.
Moreover, approaches for the composition of augmentation functions as well as hyperpa-
rameter optimization strategies are not given sufficient attention. In addition, to the best
of our knowledge, no work has conducted a detailed comparison of the predictive perfor-
mance of AutoML-based data augmentation approaches and classical methods. This work
has been motivated by the emerging importance of AutoML for data augmentation tasks and,
as discussed above, the acute lack of coverage of many important issues in the literature.

1.5 Main contributions

The main contributions of this survey are that:

• We discuss AutoML techniques for performing various data augmentation tasks, includ-
ing data manipulation, data integration and data synthesis. The last two classes of tasks
have not been covered in previous surveys.
• We present the main techniques for designing and composing transformation basic
operations for automating data augmentation. We exhaustively describe the main char-
acteristics, challenges and workarounds of the various approaches.
• We extensively discuss a wide variety of search methods for finding optimal augmenta-
tion policies. The important concepts, properties as well as strengths and limitations of
common black-box optimization methods are presented. In addition, we discuss alterna-
tive methods for obtaining effective augmentation policies without using these black-box
optimization techniques.
• In addition, we present quantitative performance results of automated data augmentation
approaches based on AutoML techniques. We additionally provide a comprehensive

123
Data augmentation with automated… 4039

comparison of the predictive performance of automated data augmentation methods and

classical approaches.
• Finally, we discuss pertinent issues pertaining to the automation of data augmentation,
and provide an overview of future research prospects.

1.6 Outline of survey

The rest of the survey is organized as follows. Section 2 presents the basic concepts of
AutoML and an overview of data augmentation in AutoML pipelines. We divide automated
data augmentation tasks into three main subtasks: search space construction, optimization
of augmentation policies and evaluation of learned strategies. Section 3 presents a detailed
discussion on three broad ways of achieving data augmentation, and discusses techniques for
their realization in AutoML frameworks. Various techniques for construction of the search
space are covered in Section 4. Here, we discuss common methods for creating the set of
data transformation operations or implicit parameters or neural models that can be used to
augment data. The hyperparameter optimization subtask is discussed in Section 5. In Sect. 6,
we present quantitative performance results that demonstrate the effectiveness of automated
data augmentation techniques. We also compare the performance of these techniques with
those of state-of-the-art classical data augmentation methods. Section 7 discusses pertinent
issues, unsolved problems and future research directions. Section 7 concludes the survey.

2 Overview and general principle of AutoML-based data augmentation

The task of AutoML-based data augmentation is to automatically generate the best augmen-
tation for the given dataset and task. This is typically achieved by composing and applying
a wide range of basic transformation operations on the given training data and then using
various optimization techniques and heuristic search algorithms to find the most useful com-
bination of augmentations.

2.1 General structure of AutoML-based data augmentation pipelines

Automated data augmentation schemes typically utilize a bi-level optimization scheme ( see
Fig. 2) in which an inner loop optimizes parameters of a deep neural network on training data
while an outer loop optimizes the augmentation parameters based on the set of transforma-
tion operations and their associated constraints defined in the search space. For instance, the
Automated augmentation scheme, proposed by Hataya et al. [26], simultaneously optimizes
augmentation and image classification model parameters. With the proposed architecture,
CNN parameters are optimized by minimizing the training loss while the tunable hyperpa-
rameters of the augmentation - transformation magnitude and the probability of application
of the augmentation operations - are optimized by minimizing the validation loss.

2.2 General procedure for AutoML-based data augmentation

The first step in the AutoML-based data augmentation process is to define the relevant
augmentations to incorporate. A good AutoML data augmentor should incorporate diverse
operations that can generate a rich set of data to account for diverse real-world situations. The

123
4040 A. Mumuni, F. Mumuni

Fig. 2 Bi-level optimization scheme and basic principle of operation of AutoML-based data augmentation
methods. The general approach is to jointly optimize two machine learning loops - an outer loop involving
augmentation hyperparameter search, and an inner loop that optimizes model parameters

specification of the types and nature of augmentations typically involves defining different
types of primitive data processing operations that transform data in desired ways (e.g., noise
perturbation, rotation, scaling, blurring, contrast adjustment, etc.). In addition to the specific
transformations, hyperparameters such as the degree or magnitudes of these transformations
(e.g., the range of scale factors, rotation angels, etc.) are also specified. Another aspect of the
augmentation problem is to specify how to combine these basic augmentations to compose
more complex augmentations, known as augmentation policies. An augmentation policy is
generally understood as a set of ordered transformation operations parameterized by selection
probabilities and intensity values (see Fig. 3). Optimization techniques and heuristic search
algorithms are used to find the best augmentation policy. The best augmentation strategy is
selected by evaluating the performance of candidate augmentations on the target or a proxy
task according to a prior defined performance criterion. This task is generally formulated as
a joint optimization problem, where performance on the end task is maximized by jointly
optimizing model parameters, augmentation operations and their associated hyperparameters.
The most important factors in the choice of search strategy are the accuracy of the resulting
policy, the search time and the cost in terms of computational resources.
Thus, the automated data augmentation tasks can generally be decomposed into three
sub-problems:

1. Search space composition: This defining basic transformation operations and hyperpa-
rameters to be used for data augmentation. The types of augmentation operations depend
on the data type and target task. For computer vision tasks, for example, these primitive
augmentation operations are typically image transformation operations (e.g., affine and
photometric transformations such as scaling, rotation, flipping, shearing, color jittering,
brightness adjustment and noise addition).
2. Augmentation policy search: This task, also known as augmentation policy optimization,
is the task of formulating and applying search strategies to find the best combination

123
Data augmentation with automated… 4041

of transformations and the associated hyperparameters in the search space that yield
effective augmentations.
3. Evaluation of performance of augmentation policies: The final step in the automated data
augmentation process is to evaluate the performance of all candidate policies and select
the best. This is typically achieved by validating the performance of the resulting model
on a proxy task or on the target task. It is common to solve tasks 2 and 3 as a single
optimization problem.
The different approaches for accomplishing each of these subtasks are thoroughly dis-
cussed in Sects. 4, 5 and 6. We present search space composition techniques-approaches for
designing the set of data transformation operations or implicit parameters capable of learning
transformations on data. Tunable hyperparameters that make it flexible to generate augmenta-
tions of varying magnitudes and with different attributes are also discussed. Heuristic search
algorithms for finding the most effective transformations and their associated hyperparam-
eters are covered in Sect. 5. Because in most implementations the evaluation strategies are
realized as part of the optimization step, we only briefly cover this subtask (in Sect. 6).

3 Approaches for data augmentation

3.1 Data manipulation approaches

The easiest and most common way to extend training data is to manipulate existing data by
applying appropriate transformations. With this method, typically, a large set of transforma-
tion operations, together with possible magnitudes and application probabilities are defined
(see Fig. 3). There are many ways to generate these basic transformation functions. They can
be explicitly specified (e.g., [17, 27, 28] or learned (e.g., [29, 30]). In essence, this approaches
consists in designing a set of data transformation operations or implicit parameters or neural
models capable of learning transformations on data.

3.1.1 Image data manipulation

For automated image augmentation (e.g., [31–33]) the search space typically consists of
a combination of spatial image transformation operations such as rotation, sheer, vertical
and horizontal flipping, cropping and scaling, as well as photometric transformations like
smoothing, solarization, sharpening, blurring brightness and contrast adjustment, noise addi-
tion, and color space conversion operations. Input data samples are augmented by different
policies selected according to specific search criteria. In most cases, a policy in turn consists
of sub-policies or proxy policies each of which applies a set of transformation operations on
the training data in a sequential fashion.
While this approach is widely successful, care must always be exercised in its application;
the technique is aimed at changing the very nature of the training data. In particular, the
performance of models trained on the resulting data is sensitive to not only the type of trans-
formation operations, but also to the parameters of transformation functions. Consequently,
incorrectly specified transformations or aggressive application of operations would often lead
to unintended distortions that harm performance. This problem has been reported in several
studies, including in the semial works AutoAugment [17] and Population-Based Augmenta-
tion [34]. Because AutoML-based data augmentation methods based on data manipulation
techniques have been extensively covered by other surveys (e.g., [24, 25, 35]), we rather

123
4042 A. Mumuni, F. Mumuni

Fig. 3 General process of data manipulation-based augmentation using AutoML pipelines. With this approach,
a subset of the input data is sampled by a learned sampler for transformation by different candidate aug-
mentation policies. Different augmentation outcomes are produced by varying the ordering and application
probabilities of the transformation functions OP1, OP2,.. OPn

focus on AutoML-based data augmentation strategies that have not received much attention.
The most prominent of these approaches are dataset integration and data synthesis methods.
These are covered in the next subsections.

3.1.2 Text data manipulation

Manipulation-based text augmentation creates variations in the original text while still main-
taining the meaning and grammatical structure. Approaches for manipulating textual data
include rephrasing, randomly deleting or inserting words, replacing words by their corre-
sponding synonyms, rearranging sentences in long texts, and introducing noise in the form
of typographical errors. Natural language processing techniques [36] are typically used to
apply these manipulation methods in AutoML pipelines. Instead of explicitly performing
transformations in the AutoML pipeline using basic NLP algorithms, approaches such as
Text AutoAugment [37] relies on learning text manipulation polices from input data. More
recently, large language models have been seen as a vital tool for automated text augmentation
[38, 39]. Besides manipulating text, they can also perform a number of high-level functions
such as model configuration or serving as an interface for interaction with the underlying
AutoML framework (see [40]).

123
Data augmentation with automated… 4043

Fig. 4 ARDA [44], an example of a data acquisition technique that relies on integrating data from multi-
ple sources. In addition to performing feature engineering operations, the technique searches for optimal
hyperparameters to integrate data from related tables

3.1.3 Tabular data manipulation

Similar to image and text modalities, data manipulation operations can be applied on tables
to directly transform existing elements. Common methods include feature jittering [41],
cell completion [42], table decomposition and reconstruction [43], and random row-wise
permutation. Feature jittering techniques generally aim to learn embeddings of the target
tabular data and perform suitable manipulations in the embedding space. Cell completion
involves populating empty cells, supplying additional entities or missing attribute in the target
tables. With decomposition and reconstruction techniques, entire tables are disentangled,
simplified and recomposed into new tables. The relative positions of rows can also be altered
in various ways to provide additional variation of the tabular data.

3.2 Data integration

Another common approach to data augmentation is to combine several complementary

datasets in order to obtain richer and expanded training data. The first step in the integration
process is data discovery, where relevant datasets per the given machine learning problem and
objective are identified and designated as candidate datasets. The second step of the process
consists in selecting suitable data points from the discovered datasets for use. This involves
collecting data or extracting relevant data elements from different sources, applying neces-
sary transformations and additional processing to make them homogeneous and compatible.
The third step is to then reconstitute them into a new target dataset. This process is shown in
Fig. 4.

3.2.1 Image data integration

Research (e.g., [45–47]) has shown the effectiveness of mixing different image datasets
to overcome the limitations of smaller datasets. Despite the attractiveness of this method,
it is particularly challenging to integrate image data from different sources as these data
often suffer from problems of inconsistent, incomplete, or noisy (i.e., inaccurate) annota-
tions, rendering the effectiveness of automated data integration methods for image modality
low. Moreover, data cleaning methods are ill-suited for image modalities. Because of these
difficulties, only a few works have attempted using automated machine learning methods
to integrate image data from varied sources. Some researchers (e.g., Kim et al. [48, 49])
have proposed to address this challenge by employing dedicated label correction or filtering
algorithms in the integration framework.

123
4044 A. Mumuni, F. Mumuni

Instead of attempting to clean noisy image data or to correct erroneous labels, an alternative
class of approaches [46, 47, 50–52] aims to modify the training process in such a way that
makes the resulting model more robust to the impact of noisy image labels. For instance, Gao
et al. [46] propose a so-called Automated Robust Loss (ARL) framework, an AutoML-based
meta-learning method, that learns loss functions that are robust to noisy labels. Yao et al. [49]
employ a function approximation method based on a modified Newton algorithm within an
AutoML framework to filter out inconsistent and noisy labels by selectively sampling only
images with clean labels.

3.2.2 Tabular data integration

Techniques for the integration of tabular data [9–11, 44, 53–56] generally aim to automatically
discover, select and aggregate related data in order to extend a given dataset. Tabular data
is typically characterized by problems such as missing values, large discrepancy in data
representation, inconsistency of keys, and the presence of a wide variety of variables. This
makes the problem of integrating this type of data an extremely challenging but useful
task. Chepurko et al. [44] presented an AutoML-based data acquisition strategy aimed at
integrating different but related tabular data into a single dataset to provide effective training
of machine learning models. Given a set of database tables and corresponding keys, the
proposed model automatically searches for, and joins the most related tables in an optimal
way. Bai et al. [10] propose neural architecture search (NAS) [57] method to find the best
network topology and connections that provide the most effective integration. Kumar et al.
[58] demonstrate that integrating relational data from multiple sources does not always lead
to performance improvements; it requires a rigorous and careful implementation to achieve
improvements. It is the task of the AutoML framework to find integration strategies that
provide optimal performance.

3.2.3 Integrating text and multimodal data

Approaches have been devised to allow automated machine learning models to seamlessly
integrate text [59, 60] as well as heterogeneous and multimodal data [61, 62]. Owing to the
differences in construction of the different datasets, it often requires enormous preprocessing
to achieve good integration. Automated machine learning techniques are able to handle all
the needed processing tasks for integrating multi-modal data. For instance, Shi et al. [61]
propose to employ specialized transformer models as multimodal data processing units to
allow automated learning on multimodal data—specifically, text and tabular information
in different formats. The Transformer units rely on natural language processing (NLP) to
obtain useful features from text datasets. In the processing stage, these feature sets are then
aggregated, transformed and combined with tabular data, where AutoGluon-Tabular [63], an
AutoML framework for learning on tabular data, is then used to seamlessly processed the
aggregated information from multiple data sources and formats.
Another challenge is that data from different sources may exhibit different statistical
distributions [64, 65]. Moreover, in situations where the collected data is from sources created
by diverse players with different levels of expertise or conflicting interests and goals, data
quality and bias become important issues and need to be handled in the integration process
[64]. This is currently achieved using automated cleaning methods [66].

123
Data augmentation with automated… 4045

Fig. 5 A comparison of classical and automated data synthesis approaches. Classical methods of data synthesis
involve three distinct steps A: (1) creation of primitive data elements (e.g., geometric models), (2) model
training, and (3) evaluation of results and fine-tuning. With AutoML approaches, however, the entire data
synthesis process is carried out end-to-end in a single process. Like in many AutoML procedures, the process
typically utilizes a bi-lel optimization scheme to optimize hyperparameters for both synthesis primitives (outer
loop) and model hyperparameters (inner loop)

3.3 Data synthesis with AutoML

Data synthesis methods effectively generate new data samples from “scratch”. The use of
synthetic data [67–70] for training machine learning models has emerged as an important
approach to address data scarcity issues in many domains. The generation of synthetic data
is currently a tedious and time-consuming process. Data synthesis for medical image anal-
ysis [67, 71] and other computer vision tasks [72], for example, typically involve mundane
processes of creating and manually manipulating representation primitives such as basic
geometric shapes and simulation parameters to produce realistic data.
Unlike classical data acquisition methods that perform synthesis separately as a distinct
subproblem, approaches based on automated machine learning principles typically treat the
synthesis task and the target application as a unified problem, where optimal simulation
parameters can be jointly learned for the downstream task. This distinction is shown in
Fig. 5. The workflow of a typical data synthesis method, Task2Sim [73], is shown in Fig. 6.
Figure 7 shows samples of image data generated by the technique. The approach contrasts
sharply with traditional approaches such as A3D [74] that do not make use of feedback
signals from the performance on the downstream task to optimize simulation parameters.
The synthesis process in the latter case, thus, involves extensive manual tuning.

3.3.1 Image data synthesis

AutoML approaches aimed at automating synthetic image data generation pipelines include
[48, 75–77]. For example, Kim et al. [48] propose a deep neural network that aims to collect
real-world data by means of adaptive sampling. Their model, LADA, is both a data integra-
tion and synthesis method as it learns to acquire informative samples and at the same time

123
4046 A. Mumuni, F. Mumuni

Fig. 6 Task2Sim [73] generates synthetic data from primitive elements and utilizes reinforcement learning-
based gradient approximation method known as REINFORCE algorithm [78] to estimate the gradients of the
performance on the downstream task with respect to the data simulation and model parameters. The model
first learns to map input images to the best set of simulation primitives. It is then trained to generate synthetic
image samples for specified tasks in a pre-training phase. The aim is to transfer this ability to unseen tasks

Fig. 7 Sample synthetic image data generated by Task2Sim model proposed by Mishra et al. [73]-top images
in each column. After pre-training, the model is able to generate new samples for unseen tasks. Here, given an
exemplar image, different views are generated by pre-trained Task2Sim model and two other methods, domain
randomization and ImageNet pre-training

generates synthetic instances from the acquired samples using AutoML-based data synthesis
principles. In order to generate new samples, they train a policy to maximize the acquisition
score based on feedback of the training loss from a classification network. To guarantee the
informativeness of the synthetic samples, the authors propose a so-called look-ahead data
acquisition technique that speculates about (i.e., predicts) the quality of plausible candi-
date data in advance. Also, they employ an oracle to annotate the unlabeled real-world data
acquired.
In computer vision domains (e.g., [73, 79–83]), the task is usually to synthesize visual
scenes or images. The search space in this case consists of mathematical operations describ-
ing geometry primitives and other scene parameters such as textures, color, lighting, pose
and camera properties. The synthesis process involves the conversion of these elementary
information into realistic 2D or 3D data. The role of automation in the synthesis process is to
jointly optimize the rendering parameters and the machine learning model conditioned on the
performance of the synthetic data on the target task. Ruiz et al. [80] utilize AutoML technique
with reinforcement learning-based optimization to control the quality of synthetic data gen-
erated using computer graphics modeling method. Their approach aims to automatically find
scene parameters for the simulation process that maximize the accuracy of trained model.
More modern techniques such as NeuralSim [84] (Fig. 8) employ neural radiance fields as
implicit representations of simulation primitives instead of relying on explicitly modeled

123
Data augmentation with automated… 4047

Fig. 8 Basic architecture of NeuralSim model [84]. The approach is a typical AutoML-based data synthesis
model that employs a bi-level optimization scheme to finetune model hyperparameters with the help of multiple
feedback signals. The outer loop optimizes hyperparameters for rendering primitives while the inner loop the
machine learning parameters

geometric primitives. The method has been designed to synthesize data for object detection
tasks. It employs a bi-level optimization strategy to jointly optimize model parameters of the
object detector and rendering parameters of the synthetic data (e.g., illumination, texture,
object pose, material, color and appearance).
Another area where automated data synthesis has been particularly successful is in tasks
such as depth estimation, optical flow and stereo vision. Since these tasks do not require
photorealistic data, the synthesis process is less resource-intensive from the computational
perspective. Consequently, models pre-trained on synthetic datasets consistently outper-
formed those trained on real data alone. Because of this success, techniques to automate
the synthesis of data [82, 83, 85] in these domains are currently receiving serious attention.
Sun et al. [82], for example, propose a technique, called AutoFlow, to automatically ren-
der synthetic 2D data for optical flow tasks by learning model hyperparameters to optimize
the appearance, shape and motion of the generated data. The synthetic data generated by
AutoFlow is designed to be used to pre-train optical flow models before fine-tuning on real
data. Unlike classical approaches (e.g., [73, 86, 87]) that tackle the pre-training task as an
independent process, Sun et al. [82] framed the data synthesis and pre-training performance
(i.e., performance on target data) as a joint optimization problem. They demonstrate that the
approach achieves better performance than conventionally rendered synthetic 3D datasets
such as FlyingThings3D [88] and Flying Chairs [89]. Moreover, their approach has proven
to be significantly more data-efficient in pre-training optical flow models on MPI-Sintel [90]
and KITTI [91] datasets than traditional approaches.

3.3.2 Tabular data synthesis

Some of the most popular traditional methods for generating synthetic tabular data involve
random oversampling. This approach is particularly useful for balancing imbalanced data,
where minority classes are typically sampled and interpolated to create new datapoints.
SMOTE [92], a seminal work among this class of approaches, is commonly used to generate
synthetic data to populate tables by means of k-Nearest Neighbor algorithm which inter-
polates between existing data points to create new samples. These oversampling methods
can be incorporated within larger neural frameworks to automate the data synthesis process.
For instance, Wang and Pai [93] propose an approach that employs SMOTE together with a
GAN model to generate novel tabular clinical data. Similarly, DeepSMOTE [94] combines
SMOTE with an encoder-decoder sub-model for synthetic data generation. Aragao et al. [95]
propose a self-balancing, synthetic tabular data generation pipeline that utilizes both under-
sampling and oversampling techniques within an AutoML framework to generate new data.
Some new approaches bypass the step of modeling data analytically and instead learn the
generation process end-to-end from data. Rashidi et al. [96], for example, devise a so-called

123
4048 A. Mumuni, F. Mumuni

Synthetic Tabular Neural Generator (STNG) that trains a dedicated neural network within a
larger AutoML framework to specifically generate rich tabular data.

3.3.3 Text data synthesis

In the past, models struggled to generate meaningful text automatically. Approaches often
utilize semantic parsers together with dedicated templates to simplify the process. AutoQA
[97], for example, employs an end-to-end neural paraphrasing model to generate answers (i.e.,
text) to different questions using template-based parsing technique. Large language models
have recently revolutionized text generation in NLP domains [98, 99]. Consequently, they
have been increasingly used to generate text for AutoML pipelines [100]. Large language
models are extremely powerful and flexible for this use case. They are adept at producing
semantically meaningful text on many different contexts and tasks. Generally, in this process,
the LLM framework serves as an interface for not only text generation but also for performing
additional functions, including model configuration (see [40, 100]) imposed in many different
ways. We discuss the most common classes of techniques next. These are grouped into three
main approaches: (1) methods based on fixed or pre-defined data augmentation operations, (2)
deeply learned data transformation operations, and (3) methods based on neural architecture
search [101–103] - i.e., learning the best neural architectures for the given data.

4 Composition of search space

The search space can be composed in many different ways. We discuss the most common
classes of techniques next. These are grouped into three main approaches: (1) methods based
on manual or pre-defined data augmentation operations, (2) deeply learned data transforma-
tion operations, and (3) those based on neural architecture search (NAS) - i.e., learning the
best neural architectures for the given data.

4.1 Fixed augmentation operations

Many AutoML techniques (e.g., [17, 27, 28, 31, 33, 104–106]) employ a fixed set of
pre-defined transformation operations to perform data augmentation. In these works, the
transformation operations themselves are fixed and can only be varied by learning tunable
parameters that control hyperparameters such as augmentation intensities and application
probabilities. Thus, the task of the AutoML process in this case is reduced to simply learning
the most useful combination of these fixed augmentations along with the optimal settings for
these hyperparameters. In the procedure, the choice of specific transformation operations and
their corresponding magnitudes to include in the search space is based on domain knowledge
and intuition. Figure 9 shows the common methods typically employed by approaches that
utilize pre-defined transformation operations to perform data augmentation.
The process of composing pre-defined augmentation operations is typically sub-divided
into three main sub-problems: (1) composition of transformation operations to be used for
augmentation, (2) a specification of (mostly) discrete probabilities with which the transfor-
mation operations would be applied on the input data, and (3) a specification of the applicable
transformation magnitudes (i.e., augmentation intensities). Again, for simplicity, transforma-
tion magnitudes are often discrete values. The original work, AutoAugment (AA) [17], for
example, uses a set of 16 common image transformation operations to augment image data

123
Data augmentation with automated… 4049

Fig. 9 Composition of fixed augmentation operations. Many works-including AA [17], FastAA [105],
FasterAA [27], PAA [33], DADA [28] and BDA [106]-employ this or similar search space construction

for classification tasks. It trains a deep CNN model on each set of transformation operations
to learn optimal parameters for effective image augmentations using validation data. Many
subsequent works (e.g., [27, 28, 32, 105, 107]) adopt a similar formulation of augmentation
operations in the search space as in AA [17] but utilize more efficient optimization meth-
ods and additional mechanisms to reduce computational requirements. For instance, AdvAA
[107] maintained the search space specification in AA [17] but, instead of optimizing the
augmentation policy hyperparameters and network parameters separately, jointly optimizes
both tasks through adversarial training. Also, instead of RL, a gradient approximation method
is used for the policy search. The visual results of applying fixed transformation operations
with different magnitudes on image data are shown in Fig. 10. The sample results indicated
are specifically for augmentation policies of FasterAA [27].
While approaches such as AutoAugment [17], FastAA [105], FasterAA [27] and [110]
associate magnitudes to each operation as transformation-dependent attributes, Direct Dif-
ferentiable Augmentation Search (DDAS) [104] treats different transformation magnitudes
as unique augmentations. This representation simplifies the search space and greatly reduces
the computational demand for performing subsequent search operations to optimize model
parameters. Many recent approaches [31, 111] follow this philosophy of designing more
simplified search space that facilitate efficient optimization.
The task of creating effective augmentation policies based on manually-designed, fixed
transformation operations has a number of significant drawbacks. First, the process is tedious,
time-consuming and is highly dependent on domain expertise. Second, the approach con-
strains the degree of automation that can be achieved. More significantly, given the large

123
4050 A. Mumuni, F. Mumuni

Fig. 10 Augmentation policies of FasterAA [27] and the result of applying these policies on sample images
of the CIFAR-10 [108] and SVHN [109] datasets

space of possible augmentations, it is not practical to manually incorporate all potentially

useful augmentation operations, thereby further limiting the benefits that can be derived from
automation. In view of the limitations of composing a search space consisting solely of fixed,
manually-designed transformation operations, some new approaches have proposed learning
more useful augmentations from data.

4.2 Learned augmentation operations

As already noted, transformation operations used in most current automated data augmenta-
tion pipelines are designed manually. This constrains their scope to the range of intuitive and
discrete transformation routines that can be constructed analytically. To further extend the
capabilities of automated machine learning-based data augmentation methods, more recent
works [29, 30, 112, 113] attempt to automatically learn useful augmentation operations
directly from the training data. The design of these types of operations requires less domain
expertise to compose task-specific augmentation policies. These approaches have already
demonstrated very competitive results. In many cases, the performance of models designed
using learned augmentations are superior to those employing fixed augmentation operations.

4.2.1 Fully learned augmentations

An alternative to learning to dynamically modify primitive transformation operations is to

deeply learn the transformation functions from data. Techniques such as spatial transformer
networks (STN) [114], Deformable Convolutional Networks (DCN) [115] and [113] have
demonstrated the effectiveness of learning effective data transformation operations using deep
learning models. Zhao et al. [113] formulate the data augmentation problem as a composite
transformation function consisting of spatial transformation defined by voxel displacement

123
Data augmentation with automated… 4051

and an appearance transformation defined by pixel intensity transformations. Chu et al.

[116] propose a natural language processing (NLP) model consisting of three sub-models-an
encoder, a custom augmentation network and a document classification model-and then find
the best augmentation strategies by employing a reinforcement learning technique to jointly
optimize all sub-models.
The STN, in particular, has recently been widely used within automated machine learning
models (e.g., [29, 30, 48, 117]) as a means of generating the basic transformation operations
that can be used to form data augmentation policies. For example, Mounsaveng et al. [29]
employ STN-based augmenter networks to learn effective transformations from data and use
a bi-level optimization scheme to search for model and augmentation parameters. Similarly,
Miao and Rahman [30] incorporate a Spatial Transformer in a network structure based on
AutoAugment [118] to learn generic affine transformations in a traffic sign classification task.
Kim et al. [48] employ an STN-based model known as InfoSTN together with InfoMixup,
an adaptive form of MixUp [119], for automated data acquisition and further augmentation.
A large number of learned augmentation methods (e.g., [107, 112, 113, 117, 123, 124])
utilize generative modeling techniques to automatically generate augmentations instead of
relying on manual image transformation operations. Techniques based on generative adver-
sarial networks and variational autoencoders are particularly useful in this regard. They are
able to synthesize realistic data with rich appearance variations. Augmentation policies are
then learned to select the most effective augmentations based on training and evaluating on
the large synthetic set generated. For instance, Gao et al. [112] propose a fully automated
data augmentation pipeline that employs three separate GAN sub-models to perform different
image transformations in order to generate augmented data- an appearance perturbation sub-
model, as well as global and local affine transformation units. They then use a game theoretic
formulation based on two min-max [125] optimization scheme to learn effective augmen-
tations. Similarly, Chinbat and Bae [123] constituted a learned search space of synthetic
data using a GAN model as a policy network. Another interesting implementation of learned
augmentations based on generative modeling can be found in works such as [117]. Instead
of directly specifying the augmentation operations, a deep learning model was used to learn
appropriate transformations from data. Here, the authors employed a so-called Augmentation
STN (A-STN) that was a modification of the original STN model, proposed by Jaderberg
et al. [114], to learn affine transformations. They also incorporated a complementary model
based on variational Autoencoder architecture [20]—known as Deformation VAE (or D-
VAE)–that performs local deformations on input samples or feature maps. A third model,
known as perturbation VAE (or P-VAE) was used to learn photometric augmentations such
as brightness and contrast adjustment, color jittering and noise addition. These three aug-
mentation networks were jointly trained with the target task to learn effective augmentation
strategies.
One of the most important advantages of methods based on deeply learned transformation
operations is their universality—ability to generalize well in a large variety of scenarios
without prior knowledge about the target tasks. Yet in some practical situations, it is more
effective and beneficial to leverage domain knowledge to encode important attributes for
specific tasks [126, 127].

4.2.2 Learning dynamic augmentations from basic transformation operations

An alternative to automatically learning data transformation operations directly from data

(e.g., see [113, 117, 123]) is to compose simple and flexible transformation functions whose
properties can be modified in the process of training. In contrast to approaches such as

123
4052 A. Mumuni, F. Mumuni

Fig. 11 Sample-Adaptive Policy for Augmentation (SapAugment) [120] uses basic transformation operations
and simple data augmentation methods such as SamplePairing [121] and CutMix [122] to transform input
data (A). The approach additionally implements a mechanism to adapt the learnable parameters of these basic
operations to the specific input dataset according to the training loss (B)

AA [17], FastAA [105] and FasterAA [27] that compose a search space consisting of fixed
augmentations and evaluate different combinations of these to obtain effective policies, many
recent approaches (e.g., [34, 120, 128, 129]) propose to learn augmentation policies in a more
flexible and dynamic way. They design the search space as a composition of loosely-specified
primitive data transformation operations that can be modified in the course of training to
generate better augmentations. This flexibility allows the model to adapt the augmentation
policies to the specific task, dataset and, in some cases (e.g., see [34]), specific input instances.
Population-Based Augmentation (PBA) [34], for example, utilizes an evolutionary algorithm
technique that allows augmentation policies to generate child policies which evolve over

123
Data augmentation with automated… 4053

time in the course of training. Also, in SaPAugment [120], Hu et al. propose a so-called
sample-adaptive policy that dynamically adapts transformation parameters according to the
training loss (see Fig. 11). Even though the search space (i.e., the set of primitive augmentation
operations) is fixed like in AutoAugment [17], FastAA [105] and FasterAA [27], SapAugment
(SAPA) [120] is able to adaptively learn and apply different transformation parameters to
different datasets in a context dependent manner. Chen et al. [130] proposed an automated
augmentation method for object detection tasks based on learning dynamic augmentations
that are robust to scale variations. Deep AutoAugment (DeepAA) [128] eliminates the need
for manually constructed default augmentation policies in the search space by replacing them
with sequentially stacked augmentation layers which can be trained to generate more nuanced
augmentation policies. Adversarial AutoAugment (AdvAA) [107] is aimed at generating
dynamic augmentation policies through adversarial training.

4.2.3 Instance-adaptive dynamic search space

Instead of employing a single augmentation strategy, many recent works such as MetaAug-
ment [131], AdaAug [132], InstaAug [133] and [116] have proposed learning adaptive,
instant-dependent data augmentation policies from data. These methods are based on the
observation that certain transformations may only produce useful augmentations for specific
input types but can be harmful when applied globally. For example, whereas the interpreta-
tion of the digit "8" is unaffected by flipping, a vertical flipping of the latin symbol “R” may
change it to the Cyrillic letter , thereby corrupting its semantic interpretation. Similarly, a
180o rotation of the Arabic numeral “6” may change its semantic label to “9” whereas the
letter "O" is generally agnostic to such a transformation. Therefore, instead of applying the
same set of transformation operations on all input samples, instance-adaptive methods [116,
131–133] learn input specific transformations according to the dataset and task.
These approaches focus on realizing more fine-grained transformation parameterizations
that provide flexible augmentations which can be applied to individual instances instead of
using coarse transformations that satisfy all samples of the entire dataset. In this formula-
tion, the instance-level augmentations correspond to specific configurations of transformation
hyperparameters. MetaAugment [131] utilizes an auxiliary model within an AutoML frame-
work which re-weights input samples in order to learn the application probabilities of different
transformation operations for specific input instances. Thus, by learning the probability of
applying transformations for different instances, the sub-model essentially predicts useful
augmentations that may be specific to the underlying instance. Based on a similar philos-
ophy, the developers of AdaAug [132] propose an adaptive data augmentation technique
that, like [134], learns effective augmentation policies in a category (e.g., [134])- and sample
(e.g., [132])-dependent way. Their model re-uses extracted features from the input data to
map samples to useful transformations, thus adapting the augmentation operations for each
instance. The technique is specifically aimed at enhancing the generalization ability of mod-
els trained with this augmentation method. The authors claim that data augmentation policies
learned by AdaAug can perform well on new datasets without further fine-tuning. Using a
slightly different approach, InstaAug [133] learns instance-adaptive augmentation strategies
by mapping instances to distributions of transformation operations instead of single trans-
formations, which are relatively coarse. The authors [133] propose to map input samples to
transformation distribution parameters in order to learn instance-specific data augmentations
policies from training data. Similar to MetaAugment, InstaAug [133] incorporates a dedi-
cated neural network sub-model, so-called augmentation module, to perform the mapping of
input instances to desired transformations. In the training process, the augmentation module

123
4054 A. Mumuni, F. Mumuni

samples transformations from the set of transformation distributions and applies them to
individual input instances in order to generate augmented samples. The resulting augmented
samples are fed into a classification network which is then trained to make predictions and,
hence, provides information about the quality of the augmentations. A comparison of the
main search space approaches is presented in Table 1.

4.3 NAS -based search space

While most AutoML-based data augmentation methods typically learn effective data aug-
mentation operations for fixed model architectures, an alternative approach to automated data
augmentation is to explore the search space for possible neural network architectures that
perform well on the given data. These approaches (e.g., ARDA [44], Ref. [136], T-AutoML
[135], Tr-AutoML [16]), instead of learning transformation functions to perform explicit data
manipulation, are concerned with automatically generating the network itself. The goal of
neural architecture search [57] is to automate the process for creating ML models. In essence,
it aims to select the best topology (i.e., network structure, including details of synaptic con-
nections) for a given dataset and task. The approach is to take an input dataset and problem
specification, specifically, in this case transformation of input images, and generate a model
architecture that solves the given problem better than all other architectures for the given
dataset. This involves finding hyperparameters that define the model structure (e.g., filter
size and configuration, synaptic connection schemes, pooling and sampling details, model
width and depth). The procedure is to iterate through model selection, where in each iteration
the algorithm generates a different network structure, sets model parameters and validate its
effectiveness based on performance on the target task. The process is completed when the best
performing model is found. A large number of studies are devoted to this subject. Some NAS-
based automated data augmentation approaches have proposed to simultaneously find the best
neural network architectures and effective augmentation policies to apply in order to achieve
optimal results. The search space in this scenario consists of possible neural architectures,
augmentation primitives and tunable hyperparameters. For instance, Kashima et al. in [32]
proposed a NAS-based approach that aims to jointly find best NN architectures and effective
augmentation policies that optimizes the performance of these architectures. Their approach
combined FasterAA [27] and DARTS [138] to jointly learn good neural architectures and the
corresponding optimal data augmentation parameters. In this formulation, DARTS provides
the set of possible model architectures while FasterAA implements the basic transformation
operations that form the basis for generating data augmentation policies. Yang et al. [135]
argue that automating only some components of the ML task while hand-crafting others can
result in sub-optimal performance. They introduce a transformer-based AutoML framework
that aims to automate the entire deep learning pipeline—model architecture, data augmen-
tation, as well as hyperparameter optimization. Thus, their approach jointly formulates (the
problems of) neural architecture search (NAS), data augmentation, model training and eval-
uation. To achieve this, first, candidate architectures and augmentation operations, as well as
the associated hyperparameters are represented as a one-dimensional vector. A predictor is
then trained to compare different combinations and configurations neural architectures and
augmentations while finding the best hyperparameter values.
One advantage of the NAS-based auto-augmentation approaches is that, compared to
methods that perform explicit transformations (e.g., [17, 27, 31, 33, 105]), they are able to
generate far superior models that perform well on unseen datasets. However, the downside of
most artificially generated models is that they tend to be more generic in nature and require

123
 Table 1 A comparison of the main search space approaches for automated data augmentation
Main direction Example works Main strengths Weaknesses

Fixed augmentations AA [17], DDAS [104], Easy to design and implement Limited scope of data augmentation
[105],FasterAA [27], PAA [33] Intuitive and transparent Decreased operations Requires domain
search space expertise
Data augmentation with automated…

Learned augmentations GA3N [123], AdvAA [107], Saves labor (on constructing Only applicable in domains where
OnlineAugment [117], LADA [48], transformation operations) Less large data samples are accessible
Ref. [113], Ref. [113] dependence on domain expertise Time-consuming training process
Can model subtle instances that May fail to encode rare examples
may not be observable to human
developers
Dynamic augmentations AdaAug [132], InstaAug [133], More flexible and adaptive Less transparent Computationally
DeepAA [128], DivAug [129], augmentations Relatively wide expensive Design process not
SAPA [120], MetaAugment [131] range of augmentations possible straightforward
More generic and easily
transferable to different tasks and
datasets
Automatically generated neural T-AutoML [135], ARDA [44],Ref. May complement the traditional Generally unintuitive Requires more
architectures (i.e. NAS-based [136], Tr-AutoML [16], Ref. [32], AutoML data augmentation computational resources
techniques) MedPipe [137] methods Can result in more simple
models with powerful
augmentation capabilities Can be
used in conjunction with other
approaches

123
4055
4056 A. Mumuni, F. Mumuni

relatively larger sizes, even for tasks where smaller models might suffice. Also, because of
the need to train and validate all the possible neural architectures, the optimization process
incurs very high computational costs and is unduly slow. Moreover, it requires huge amounts
of storage resources since all candidate models need “live” for a little while in order to be
trained and tested.

5 Optimization of augmentation policies

A naïve solution to the search problem is to test all possible augmentation operations and
their possible combinations with different model configurations. Obviously, such a strategy
is computationally costly and would be excessively time-consuming to practically realize.
Therefore, various optimization techniques are used to heuristically find the most effective
combination of augmentations and their corresponding hyperparameters without carrying out
exhaustive search (i.e., testing all possible augmentations). Many approaches utilize well-
known optimization techniques - so-called black-box optimization methods-such as Bayesian
optimization [139], evolutionary computation algorithms [140] and reinforcement learning
[141]. These techniques are well suited for finding effective transformation operations and
their parameters from a discrete space of primitive operations that cannot be described by
analytical relationships. In this section, we present common search strategies, highlighting
the main principles of operation, as well as the key strengths and weaknesses of each method.
The optimization process is an iterative procedure whereby the optimization algorithm
is executed repeatedly while comparing the results of each iteration until an optimal or a
satisfactory result is achieved. Although gradient-based methods have become the de facto
approach for optimizing deep neural networks, the discontinuous nature of augmentation
operations and their associated hyperparameters make it difficult to apply these techniques
out-of-the-box. Heuristic search strategies are therefore the most viable approach for find-
ing effective augmentation policies. classical optimization methods include random search,
particle swarm optimization (PSO), ant colony optimization (ACO), simulated annealing,
greedy algorithm, genetic algorithms and particle swarm optimization. The techniques have
been developed for solving large-scale optimization problems in computer science. The pop-
ularity of these techniques stem from their ability to solve complex and ill-formalized AI
problems for which analytical search or gradient-based methods are unsuited or do not provide
the required level of performance. While a large diversity of heuristic search optimization
techniques exist, only a small subset of the methods has been applied for the purpose of
automating data augmentation. They include reinforcement learning, Bayesian Optimization
(BO) and evolutionary computational algorithms. New search space design strategies have
also been proposed that allow much simpler search techniques to produce good performance.
We discuss these approaches in this section.

5.1 Reinforcement learning

Reinforcement learning [141] is one of the first optimization techniques to be used in auto-
mated data augmentation [17]. It is a classical optimization technique for solving ill-formed
problems and has wide applications in machine learning. Indeed, a very large number of
automated data augmentation approaches (e.g., AA [17], FastAA [105], AWS [142], PAA
[33] Learn2Augment [143], Ref. [116] and RTS [80]) utilize RL for the optimization task.
The reinforcement learning approach is based on the principle of an artificial agent inter-

123
Data augmentation with automated… 4057

acting with the environment through specific permissible actions and making observations
about the outcomes of different actions through rewards and penalties. In the context of data
augmentation, the actions are specified as sequences of basic data transformation operations.
Observations are the performance results based on a pre-defined evaluation criterion. Through
repeated actions (i.e., application of the specified augmentation operations), the agent finds
optimal augmentation strategies that maximize performance over time. In one of the pioneer-
ing works on AutoML-based data augmentation, Cubuk et al. [17] utilize a reinforcement
learning technique to simultaneously find good augmentations policies and optimal model
hyperparameters that produced state-of-the-art performance on several benchmark datasets
(see Figs. 16 in Sect. 7.2). The authors [17] utilized a recurrent neural network (RNN) as a
controller in the RL formulation to dynamically predict the most effective data augmentation
policy for a given dataset in the course of training. Their model, AutoAugment (AA) [17],
searches for the best transformation operations, their probabilities, as well as the respective
intensities with the help of the recurrent neural network (RNN)-based controller.
The approach is exceedingly expensive - even on such a relatively small dataset as
CIFAR-10, it take about 5,000 GPU hours to train. Consequently, most subsequent works
in this direction aim to achieve comparable performance while reducing the computational
overheads. FasterAA [27] and AWS [142], for example, propose to improve computational
efficiency by employing weight-sharing mechanism within a reinforcement learning frame-
work and achieve comparable results as AA on image classification tasks but with significantly
lower computational budget. Patch AutoAugment (PAA) [33] formulates the data augmenta-
tion optimization problem as cooperative multi-agent RL problem in which each agent learns
an optimal augmentation policy for a small image patch. A multi-agent reinforcement learn-
ing algorithm based on decentralized partially observable Markov decision process [144,
145] is then used to learn a joint optimal augmentation policy over the entire image.
Gowda et al. [143] propose a reinforcement learning approach to automatically learn to
select the most useful video frames for augmentations on video datasets without actually exe-
cuting and evaluating these augmentations. The technique incorporates a so-called Semantic
Matching sub-module to exploit the natural association of activities (i.e., foregrounds) and
backgrounds. Augmentations are created by combining a pair of useful video segments,
where one segment’s background is mixed with another’s foreground consisting of actor and
objects. The approach achieves a performance gain of up to 4.4% over classical augmentation
methods.
Although many of the recent RL-based methods have significantly reduced computational
cost of the optimization tasks, relative to other search methods, RL techniques are still
more expensive for many practical purposes. Consequently, many alternative approaches are
actively being explored.

5.2 Bayesian optimization

Because of the enormous computational requirements of RL methods such as AutoAug-

ment and many follow-up works, recent approaches have focused on reducing computational
requirements without incurring significant generalization performance penalty. To this end, a
large number of techniques [27, 120, 146] have been devised based on Bayesian optimization
(BO) [147]. BO methods, as opposed to reinforcement learning approaches discussed in the
previous subsection, allow to encode domain knowledge by keeping track of, and using past
evaluation results to generate better policies in subsequent searches. For these approaches,
the search space is formulated as a probability distribution in which subsequent searches

123
4058 A. Mumuni, F. Mumuni

Fig. 12 Simplified flowchart of Bayesian optimization. The approach relies on probabilistic modeling to
predict the best augmentation hyperparameters for a given dataset and task. Based on these hyperparameters,
a deep neural network is then trained and evaluated to establish the quality of the predicted hyperparameters

are more focused on areas where the likelihood of the best hyperparameters lie. Thus, the
BO approach implements a highly efficiency sampling strategy by selecting only the most
promising set of hyperparameters to evaluate based on previous calls to the “evaluator”. A
simplified workflow of the process is shown in Fig. 12.
To further improve model efficiency, instead of training on the entire dataset, Fast
AutoAugment (FastAA) [105] divides the training dataset into smaller subsets which are
then trained simultaneously on different sub-models using BO strategy to find optimal aug-
mentation policies. BO approaches have shown competitive results with relatively lower
computational overheads compared to RL-based approaches due to the ability to estimate
the quality of augmentations and skip bad augmentations policies before they are evaluated.

5.3 Evolutionary computation algorithms

As an alternative to the reinforcement learning and Bayesian optimization approaches, many

works [148] have proposed evolutionary computational techniques [140] for automating
the search for effective data augmentation policies. Algorithms based on such techniques
utilize specialized operators inspired by biological evolution processes-specifically, natural
selection, crossover and mutation-for searching for, and optimizing the performance of useful
augmentations in an iterative process during training (see Fig. 13). These techniques use
the concept of natural selection to ensure that only high performing augmentation policies
are maintained from one iteration to the next. The basic idea is to search from an initial
pool of possible augmentations and select good transformation operations, and during each
iteration of the search process to continually eliminate non-optimal transformations. The
augmentations obtained may be combined with other augmentations in a process called
crossover. Further, the population-based technique periodically induces random changes in
one or more augmentation operations of the current set by replacing some of the parameters
with random parameters, thereby mutating the affected augmentations. Consequently, new
and better set of augmentations are obtained by the process of crossover and mutation. Thus,
in contrast to the other black-box optimization approaches discussed earlier that search over
static transformation operations, approaches employing evolutionary computation algorithms
allow the underlying augmentation operations to be varied and improved in the course of
training. The enhancement is achieved by periodically making random changes to the current
best policies, and inter-mixing to produce better characteristics. These operation are known
in scientific literature as mutation and crossover, respectively. One of the first works to use

123
Data augmentation with automated… 4059

Fig. 13 Basic principle of evolutionary computation algorithms. This is conceptually similar to the process of
natural selection in living organisms

evolutionary algorithm method for data augmentation was PBA [34]. Subsequently, Cheng et
al. proposed a modified version of PBA, Progressive Population Based Augmentation (PPBA)
[148], by gradually reducing the search space in the course of training. This augmentation
strategy was applied on multimodal data and showed good results.
When composing a search space for this approach, the main focus is to construct an initial
set of useful transformations and a corresponding set of policies which can evolve with time.
In this case, the augmentations can be composed using a variety of methods. For example,
they can be sample space transformation operations (e.g., [34]) or feature space (e.g., [149]).
Cheung et al. [149] propose an evolutionary computation approach to find good augmentation
policies composed in the latent space. Poor performing augmentations in the search space
are gradually eliminated in the course of training while good ones are retained and enhance
over time.

5.4 Gradient-based methods

Gradient-based automated data augmentation approaches are designed to approximate the

gradients of augmentation hyperparameters rather than to iteratively search a discrete space

123
4060 A. Mumuni, F. Mumuni

for good augmentation policies. They are mainly aimed at achieving higher efficiency com-
pared with computational methods like RL and evolutionary algorithms
While gradient-based methods have proven to be universal and well adapted for training
deep neural networks to find optimal parameters in modern machine learning algorithms,
they are ill-suited for out-of-the-box AutoML-based data augmentation formulations. The
application of gradient descent methods in solving automated data augmentation problems is
constrained by the inherently non-differentiable nature of the search space. This is as a result
of problems associated with discrete nature of transformation hyperparameters (e.g., discrete
scale and rotation angle specifications), making it difficult to apply simple and effective
gradient-based optimization techniques.
To enable the use of gradient descent methods for automated data augmentation, it is
required that the search space be necessarily both continuous and monotonous, and that
at each iteration in the search process, the direction of the greatest increase (or decrease)
in the objective function can be determined until an optimal augmentation parameters are
found. Several workarounds exist for ensuring that these conditions are met. Many of these
approaches are based on previously developed techniques for solving general black-box
optimization problems using the concept of approximate gradient estimation [78, 150, 151].
The approximate gradient estimator is essentially a differential operator that takes discrete
parameter values or random, continuous variables and returns the corresponding gradients;
it is a generalization of the concept of a gradient descent for non-differentiable and discrete
functions. The use of this concept makes it possible to apply GD methods for finding optimal
augmentation policies in a discrete search space.
Many approaches (e.g., DHA [152], DAAS [153], AdvAA [107]) invariably reformulate
the search space in such a way that approximate numerical methods can be used for the compu-
tation of derivatives of the objective function. Faster AutoAugment [27] uses straight-through
estimator to simplify the representation of the search problem and allow the application of
backpropagation to find optimal transformation operations. Similarly, instead of directly
performing differentiation to estimate the gradient, MADAO [26] data augmentation as a
differentiable task using Neumann series approximation to compute implicit gradients for
the objective function. Xu et al. [154] employ stochastic relaxation method [155] to approxi-
mate the representation of non-differentiable augmentation hyperparameters in differentiable
form to allow gradient-based policy optimization.
Some new methods such as [107, 111] are based on applying reinforcement learning
concepts within a connectionist (i.e., network) framework to estimate the gradients of aug-
mentation parameters. Zhang et al. [107] proposed Adversarial AutoAugment (AdvAA)
which computes approximate gradients with the aid of REINFORCE algorithm-a general
statistical method based on reinforcement learning-proposed by Williams [78]. A similar
approach is used in Online Hyper-parameter Learning for Auto-Augmentation (OHLAA)
[111] to estimate the gradient of the loss function with respect to the augmentation policy
hyperparameters.
Different from the above methods, some approaches (e.g., [81, 156]) rely on proxy models
where augmentation variables can be represented as continuous and differentiable param-
eters, and used to indirectly optimize the discrete augmentation hyperparameters (through
joint training). DADA [156], for example, proposes a continuous formulation of the discrete
search space by introducing a continuous reparameterization of the discrete augmentation
policy variables using a differentiable neural network as a surrogate model. The method is
based on RELAX, one of several techniques proposed by Grathwohl et al. [150] to extend gra-
dient descent to non-differentiable black-box optimization problems. It eliminates the need
for continuous relaxation of the discrete variables by utilizing a surrogate neural network,

123
Data augmentation with automated… 4061

whose parameters can be used to control the augmentation variables through joint optimiza-
tion with the augmentation policy hyperparameters. Likewise, Shirobokov et al. [81] utilize
differentiable surrogate neural networks that allow the optimization of simulator parameters
for the synthesis of artificial data that are representative of real physical processes. We sum-
marize the main strengths and weaknesses of the various optimization techniques covered in
this work in Table 2.

5.5 Ensemble optimization methods

A recent hyperparameter search approach is to implement multiple optimization algorithms

in a single AutoML model and then algorithmically determine the most effective ones for
a given dataset and task. The ensemble search approach aims to leverage the strengths of
different optimization methods to achieve higher performance by combining multiple search
algorithms in a single framework. Since different optimization techniques are effective for
different datasets and tasks [160], these approaches are particularly useful for AutoML models
designed for generic applications that need to be able to handle multimodal data or process
a broad range of data types for different machine learning problems.
Predictably, modern large-scale AutoML frameworks (e.g., TPOT [161], AutoDES [162],
Hyperopt-Sklearn [163] and AutoWeka [164]) typically rely on ensemble optimization tech-
niques that can incorporate search strategies. In these frameworks, multiple optimization
algorithms can be applied simultaneously. For instance, Auto-Tuned Models (ATM) [165]
defines Bayesian optimization and reinforcement learning (multi-armed bandit) algorithms
in its pipeline which can be used simultaneously to find optimal hyperparameters for the input
data. Some models such as H2O AutoML[166] and AutoDES [162] incorporate a flexible
representation of plausible pipeline structures that can be applied depending on the target
task or input data type. Despite the increased model complexity and computational cost, the
approach is often justified when solving broad machine learning problems involving very
large and complex data.

5.6 Searchless optimization of data augmentation policies

Although the optimization techniques discussed in Sects. 5.1, 5.2, 5.3 and 5.4 have demon-
strated impressive results, they are still too expensive for most practical purposes. For
instance, the optimization method employed in AutoAugment takes about 15,000 GPU
hours to train on ImageNet [128]. To address the challenge of computational complexity,
some recent works (e.g., [31, 159, 167, 168]) propose simplified search space design where
good augmentation policies can be found using much simpler search strategies like random
search [169], grid search [170] or greedy search [171]. Greedy AutoAugment [167] proposes
to reduce the search space by eliminating the need to search for different combinations of
augmentation operations in a combinatorial manner. The basic idea is to dynamically expand
the space of augmentation operations in the direction of effective augmentation policies by
creating and modifying sub-policies with good performance. In the search process, only
transformation type and intensity are the only hyperparameters considered. The probabil-
ity of applying the operations are determined later when the search is complete and is only
applied on the effective policies found. This significantly reduces the amount of computations
needed to find good augmentation strategies. Because of its simplicity and effectiveness, the
greedy search policy optimization procedure has been used in several works [172].

123
 4062

123
Table 2 A comparison of the main strengths and weaknesses of the common optimization techniques used for automated data augmentation
Optimization method Example works Main strengths Weaknesses

Reinforcement learning [17, 33, 105, 143] Effective when cannot be intuitively determined Does Expensive to train Generally complex architecture
not require prior knowledge about augmentation
operations
Bayesian optimization [27, 120, 146] More flexible augmentation process Computationally Less transparent Computationally expensive Design
efficient process not straightforward
Evolutionary Algorithms [34, 148, 149] Explicit mechanism to adapt augmentation policies Relatively expensive Relatively complex schemes
Evolution of policies can lead to better than can be
intuitively designed No rigid upper ceiling for
performance
Gradient methods [26, 27, 81, 111, 156] Fast training Simpler models Readily integrates with Often requires approximations; can compromise the
DNN training process fidelity of representation Not always possible
Ensemble approaches [27, 107, 157, 158] Allows to exploit the complementary benefits of Can result in very complex models Unintended
different methods Can generate more diverse interactions may harm performance
augmentations than single methods Potentially more
robust
Searchless methods [31, 159] Fast training Relatively simpler models Less flexible augmentations Performance may have a
“hard” upper ceiling
A. Mumuni, F. Mumuni
Data augmentation with automated… 4063

Approaches for further simplifying the search process have also been considered. In some
cases, for example, UniformAugment (UA) [159], these simplifications allow the search
phase in the optimization process to be eliminated altogether without incurring significant
performance penalty. RandomAugment (RA) [31] employs just two tunable parameters—
augmentation intensity (i.e., transformation magnitudes) and selection probability-as vari-
ables to characterize the search space. Given the simplicity of this formulation, a simple
grid search is sufficient to effectively search for optimal values of augmentation parameters.
Instead of modeling selection probabilities as independent hyperparameters, in RA [31] the
probability of applying any given augmentation operation is a constant value that is solely
dependent on the number of applicable augmentation operations in the search space; it is
computed as the inverse of the total number of operations. In the same way, the augmentation
intensities are assumed to be uniform for all transformation operations. Based on this sim-
plified reformulation of the search space, it is then possible to obtain optimal augmentation
policies using only a single variable parameter, the augmentation intensity, and a constant-the
total number of transformation operations which defines the application probability. Rando-
mAugment RA [31], is thus able to reduce the search space from 1032 possible operations to
just 100. For this simple scenario, a naïve grid search provides very good results comparable
to state-of-the-art performance.
Unlike in RA [31] and all previous optimization strategy formulations, Uniform Aug-
ment (UA) [159] completely bypasses the search stage and instead propose to uniformly
sample augmentation policy hyperparameters from the search space. To achieve this, the
authors formulate the search space as distribution invariant by ensuring that the transforma-
tion operations and associated intensities result in label-preserving augmentations, and the
transformed data remains within the distribution of the original input data. A machine learn-
ing model is then able to learn optimal augmentations by sampling randomly from the search
space and optimizing model parameters using gradient descent. Uniform Augment, thus far,
is the cheapest in terms of the number of computations needed to find optimal augmentation
policies. Indeed, the number of search operations is theoretically zero.

6 Evaluation methods

An important step after selecting a subset of operations for the search space and an optimiza-
tion method is to evaluate the performance of the resulting model. This process is repeated for
different combinations of model configuration settings (i.e., all plausible sets of augmentation
operations, hyperparameters and search methods). At the end of the process, performance
results are compared and an optimal pipeline and hyperparameters chosen according to a
specified performance criterion. The costly nature of the evaluation stage significantly lim-
its the range of plausible configurations that can be explored to achieve optimal predictive
performance. Instead of training the network to converge before evaluating its performance,
approaches have been devised to accelerate the process by enabling near-optimal strategies
to be found without exhaustive evaluations.
One of the most popular approaches is the so-called early stopping strategy [20], a tech-
nique based on terminating the evaluation process for settings that are predicted to perform
relatively poor on a validation set. Some works, for instance, RandAugment (RA) [31], Triv-
ialAugment (TA) [168] and OHLAA [111], reduce the range of augmentation operations and
hyperparameters in the search space in the evaluation process. A popular method to speed
up model evaluation is to reduce the fidelity of the input data [20]. In image augmentation,

123
4064 A. Mumuni, F. Mumuni

this is achieved by reducing the sizes or resolutions of the images used. Another common
approach is to train on proxy tasks–i.e., employ reduced model size or subsets of train-
ing data. Examples of AutoML-based data augmentation models that employ this strategy
include AutoAugment (AA) [17], FastAA [105], [105], FasterAA [27], DADA [156], AWS
[142], and PBA [34]. Another method for reducing the enormous computaional overhead at
the evaluation stage is to employ surrogate models as evaluators [173, 174] to predict the
performance of the target models. This approach circumvents the need to perform costly
evaluation on different configurations of the real model.

7 Quantitative performance of data augmentation approaches

We present results on the performance of AutoML-based data augmentation methods and

compare these with the state-of-the-art data augmentation methods based on classical
approaches. For fairness we compare methods that use the same datasets and similar train-
ing settings (i.e., similar model configuration and number of training epochs). The results
presented here are curated from the original works. We highlight instances where results or
other performance information from secondary sources is used.
We first describe the datasets and settings used in most of the works. We also introduce the
common performance benchmarks and evaluation metrics commonly used to compare the
data augmentation methods investigated in this work. Finally, we present quantitative perfor-
mance results and compare AutoML-based methods against several state-of-the-art classical
methods. In addition to performance comparisons, we also show the effect of combining clas-
sical and AutoML-based augmentations. These results show that appreciable improvements
can be gained by combining the two classes of methods.

7.1 Datasets and model settings

7.1.1 Datasets

To assess the predictive performance of automated data augmentation methods, test results on
the following datasets were considered: CIFAR-10 and CIFAR-100, SHVN [109], ImageNet
[175], and MS COCO [176]. We briefly each of these datasets in the next paragraphs.
The CIFAR-10 [108] and CIFAR-100 [108] are highly popular image classification
datasets. Each of the datasets contains 60,000 labeled 32x32 RGB images that are divided
into 50,000 training and 10,000 test images. The images in CIFAR-10 are divided into 10
classes of 6,000 images per class, while CIFAR-100 has 100 classes, each containing 600
(500 training and 10 test) images.
The Street View House Numbers (SVHN) [109] is a digit (i.e., decimal numbers 0
to 9) recognition dataset containing 600,000 color (RGB) images. The dataset is made up
of cropped, 32×32 images derived from Google Street View [177]. Thus, the images are
pictures of real-world house number plates in challenging settings. SVHN include bounding
box information in a separate file and can therefore be used for digit detection in addition to
image classification.
ImageNet [175] is a large-scale visual recognition dataset for generic computer vision
tasks. A subset of the dataset used by most works for image classification -ImageNet Large
Scale Visual Recognition Challenge (ILSVRC) [178]-has a total of 1,431,167 images divided

123
Data augmentation with automated… 4065

into 1000 different object categories. The entire set is made up of 1,281,167 training, 100,000
test and 50,000 validation images.
The Microsoft Common Objects in Context (MS COCO) dataset [176] is a large-scale
dataset which can be used for many computer vision tasks. It is primarily designed for object
detection, pose estimation, keypoint detection, image captioning, and segmentation tasks. It
has a total of 330,000 images split into 80 classes.

7.1.2 Performance metrics and model settings

For the classification tasks, the performance metric commonly used is the generalization
accuracy. Results for ImageNet include top-1% and top-5% accuracy measures. Mean average
precision (mAP) is used to characterize performance of the object detection tasks, i.e., on the
MS COCO dataset (Table 5).
Performances on CIFAR-10 and CIFAR-100 have been tested using three different back-
bone models: Wide Residual Networks [179], specifically, Wide-ResNet-28-10 (see the
original work for a detailed description of this setting); Shake-Shake (26 2x32d) [180]; and
PyramidNet [181]. Results for ImageNet dataset have are based on ResNet-50 and ResNet-
200 backbone models (see He et al. [182] for details). RetinaNet [183] is used with various
ResNet models [184] on the MS COCO dataset.

7.2 Performance of AutoML-based data augmentation methods

Most of the existing works report results for image classification using large-scale CNN
models. The most popular datasets used for these purposes are the ImageNet and CIFAR
(CIFAR-10 and CIFAR-100) family of datasets described in Sect. 7.1. Detailed performance
results for automated data augmentation methods based on these datasets are summarized in
Tables 3, 4 and 7. Table 3 shows image classification results for CIFAR-10 and CIFAR-10 for
three different backbone models- WideResNet 28-10 (WRN28-10) [179], Shake-Shake 26-
32 (S-S 26-32) [180] and PyramidNet (PNet) [181]. For each data augmentation method, we
indicate the average classification accuracy for the three backbone models. From the Table,
the mean gain in accuracy range between 0.6 and 1.5 on CIFAR-10 and between 0.8% and
3.1% on CIFAR-100. This demonstrates consistently strong performance for all methods.
Indeed, the average performance gain for all the methods is 1.02% and 2.17% on CIFAR-10
and CIFAR-100, respectively. Note that for CIFAR-100, many of the methods do not have
results for PyramidNet backbone on CIFAR-100. We exclude these methods in our average
performance computations. Similar to Tables 3, Table 4 shows classification results for the
ImageNet dataset based on ResNet-50 and ResNet-200 backbone models. Results are shown
for top-1 and top-5% accuracies. The table also indicates percentage performance gain for
each of these accuracy measures. The gains are consistent for all methods, with the lowest
top-1% accuracy gain on RestNet-50 being +0.9% (for PBA method) and +1.5% on ResNet-
200. The performance gains for various AutoML-based data augmentation methods on the
SVHN dataset are summarized in Fig. 14. Again, the results show impressive performance
on challenging digit classification tasks. In Table 7 we compute the average performance
of different categories of automated data augmentation methods based on the optimization
strategies they employ. The results show that performance difference across different search
methods, on average, is largely insignificant.
While the majority of approaches have reported results for image classification tasks,
a few works have evaluated the performance of automated data augmentation techniques

123
4066 A. Mumuni, F. Mumuni

Fig. 14 Digit recognition

accuracy on the SVH dataset with
Wide-ResNet-28-10 model. The
results show performance gain, in
percentage, of different
AutoML-based data
augmentation methods

Fig. 15 A comparison of gain in predictive performance (percentage accuracy over baseline) of automated data
augmentation methods and augmentation techniques based on conventional paradigms. The baseline accuracy
is 96.1%

on machine learning tasks other than image classification. For instance, Liu et al. [104],
Cubuk et al. [31] and Chen et al. [185] investigate the performance of common automated
data augmentation methods-AA [17], DADA [28], DDAS [104]-on object detection tasks.
The tests were conducted using large-scale detection models, specifically, ResNet-101 and
RetinaNet. The results of these experiments are summarized in Table 5. These studies show
that AutoML-based data augmentation methods are effective for object detection. Indeed,
per the results in Table 5, the data augmentation techniques leads to an average perfor-
mance improvement of 1.54% (mean average precision or mAP) for object detection models
(Table 3).

123
 Table 3 Performance of automated data augmentation methods on CIFAR-10 and CIFAR-100 datasets
Method CIFAR-10 CIFAR-100
WRN28-10 S-S 26-32 PNet +SD Mean WRN 28-10 S-S 26-32 PNet Mean

Baseline 96.1 97.1 97.3 96.83 81.2 82.9 86.0 83.37

AutoAugment [17] 97.4 98.1 98.5 98.0 82.9 85.7 89.3 85.97
AdaAug 97.4
AdvAA [107] 98.1 98.2 98.6 98.3 84.5 85.9 89.6 86.67
AWS [142] 98.0 98.3 98.7 98.3 84.7 85.9 89.6 86.73
DADA [28] 97.3 98.0 98.3 97.9 82.5 84.7 88.8 85.33
Data augmentation with automated…

DDAS [104] 97.3 97.9 – 97.6 83.4 84.9 – 84.15

DeepAA [128] 97.4 98.1 – 97.8 83.7 85.2 – 84.45
DivAug [129] 98.1 98.1 98.5 98.2 84.2 85.3 – 84.75
FastAA [105] 97.3 98.0 98.3 97.87 82.8 85.4 88.3 85.50
FasterAA [27] 97.4 98.0 – 97.70 82.2 84.4 – 83.30
KeepAA 97.8 97.8 97.8 97.8 – – – –
MADAO 97.3 – – 97.3 – – – –
MetaAugment [186] 97.7 98.3 98.6 98.2 83.8 86.0 89.5 86.43
OLHA [111] 97.4 – – 97.4 – – – –
OnlineAugment [117] 97.6 – – 97.6
PAA [33] 97.8 98.2 – 98.0 83.3 – – 83.3
PBA [34] 97.4 98.0 98.5 97.97 83.3 84.7 89.1 85.70
RA [31] 97.3 98.0 98.5 98.0 83.3 – – 83.30

123
4067
 Table 3 continued
4068

Method CIFAR-10 CIFAR-100

WRN28-10 S-S 26-32 PNet +SD Mean WRN 28-10 S-S 26-32 PNet Mean

123
RUA [187] 97.4 – 98.5 97.4 83.6 – – 83.6
TA [168] 97.5 98.2 98.6 98.1 84.3 86.2 – 85.25
TeachAugment [188] 97.5 98.0 98.5 98.0 – – – –
TDGA AA [189] 97.14 – – 97.14 – – – –
UA [159] 97.3 98.1 – 97.7 82.8 85.0 – 83.90
OnlineAugment [117] 97.6 – – 97.6 83.4 – – 83.40
The backbone models used are Wide-ResNet-28-10 (WRN 28-10), Shake-Shake-26 2x32d (S-S 26-32) and PyramidNet (PNet). The average recognition accuracy values over
all three backbone models are indicated as mean
A. Mumuni, F. Mumuni
 Table 4 Performance of automated data augmentation methods on ImageNet dataset with ResNet-50 and ResNet-200 as backbone models
Method ResNet-50 ResNet-200
Top-1 Top-5 Acc. improvement Top-1 Top-5 Acc. improvement
Top-1 Top-5 Top-1 Top-5

Baseline 76.3 93.1 – – 78.5 94.2 – –

AA [17] 77.6 93.8 +1.3 +0.7 80.0 95.0 +1.5 +0.8
Data augmentation with automated…

AdvAA [107] 79.9 94.5 +3.6 +1.4 81.3 95.3 +2.8 +1.1
AWS [142] 79.4 94.5 +3.1 +1.4 81.4 95.3 +2.9 +1.1
DADA [28] 77.5 93.5 +1.2 +0.4 – – – –
FastAA [105] 77.6 93.7 +1.3 +0.6 80.6 95.3 +2.1 +1.1
OLHA [111] 78.9 94.3 +2.6 +1.1 – – – –
OnlineAugment [OAug] 77.6 –
MetaAugment [186] 79.7 94.6 +3.4 +1.5 81.4 95.5 +2.9 +1.3
PAA [33] 77.5 – +1.2 – – – – –
PBA [34] 77.2 93.4 +0.9 +0.3 – – – –
TeachAugment 77.8 93.7
RA [31] 77.6 93.8 +1.3 +0.7 – – – –
RUA [187] 77.7 – +1.4 – – – – –
OnlineAugment [117] 77.5 – +1.2 – – – – –
TA [168] 78.1 93.9 +1.8 +0.8 – – – –
The gain in recognition accuracy is indicated as "Acc. improvement"

123
4069
4070 A. Mumuni, F. Mumuni

Table 5 Object detection performance (in mean average precision) of different automated augmentation
methods on MS COCO dataset
Backbone model Aug. Method mAP Gain in mAP Results from

ResNet-50 + RetinaNet Baseline 36.9 0 Ref. [104]

DADA 38.4 1.5
DDAS 38.1 1.2
AA – –
ResNet-101 + RetinaNet Baseline 38.6 0 Ref. [185]
DADA 40.0 1.4
DDAS 40.1 1.5
AA 39.8 1.2
ResNet-101 + RetinaNet Baseline 38.8 0 Ref. [31]
AA 40.4 1.6
RA 40.1 1.3
ResNet-200 + RetinaNet Baseline 39.9 0 Ref. [31]
AA 42.1 2.2
RA 41.9 2.0
Results have been compiled from performance reports of various works. These are: [31, 104, 185]

Fig. 16 Performance of automated data augmentation techniques versus manual methods on CIFAR-100 using
Wide-ResNet-28-10 CNN backbone. The results depict improvement gain (in % accuracy) over the baseline.
Baseline performance is 81.2%

123
Data augmentation with automated… 4071

Fig. 17 Comparison of gain in performance for automated data augmentation and conventional methods on
ImageNet dataset using ResNet-50. The baseline performance is 76.3% and 93.1% for top-1 and top-5%
accuracy, respectively

7.3 Comparison of performance of classical and AutoML-based data augmentation

methods

7.3.1 Classical data augmentation methods

The performance of many state-of-the-art classical data augmentation methods have also been
evaluated for the considered benchmark datasets (i.e., CIFAR-10, CIFAR-100 and ImageNet)
and model settings. The traditional augmentation methods compared on ImageNet include
CutMix [122], SuperMix [190], MixUp [119], StochasticDepth [191], ISDA [192], Manifold
Mixup [193], PuzzleMix [194], DropBlock, IHDA [195] and SaliencyMix [196]. For CIFAR-
10 and CIFAR-100, the following methods are covered: MixUp [119], CutOut [13], CutMix
[122], ISDA [192], AgMax [197] and Random Erase (RE) [198]. The performance of these
works are reported in their original sources and in various surveys. Using those results,
we compute the performance gains for various datasets and models, and compare with the
performance of automated data augmentation methods. These results are discussed in Sect.
7.3.2.

7.3.2 Comparison of automated data augmentation methods and state-of-the art

classical approaches

In this section, we compare the performance of state-of-the-art data agumentation meth-

ods based on classical approaches with automated data augmentation methods. The results
are summarized in Figs. 15, 16 and 17. Again, to ensure a uniform point of reference for
the assessment, we selected methods that have been evaluated on the same datasets and
CNN backbone models for comparison. Specifically, the results provided for CIFAR-10 and
CIFAR-100 have been tested with Wide-ResNet-28-10, Shake-Shake 26 2x32d and Pyramid-

123
4072 A. Mumuni, F. Mumuni

Net backbones; SVHN with Wide-ResNet-28-10 backbone; and ImageNet with ResNet-50
and ResNet-200 backbones.
For the sake of clarity, we computed and compared the gain in accuracy for all the meth-
ods based on quantitative results reported in the various original works. Overall, the results
show consistently high performance for automated data augmentation methods over classical
methods. In Fig. 15, we compare the performance of the best automated data augmentation
methods with state-of-the-art approaches based on classical techniques on CIFAR-10. In the
case of classical methods, IHDA [195] shows the highest performance (+1.7%), followed by
MixUP (+1.2%) and GridMask (+1.1%). These are all well-below the best automated aug-
mentation methods, which achieve an average of +2.17% gain in classification accuracy. In
Fig. 15, it can be observed that only one classical data augmentation method, IHDA [195] at
6th, ranks among the top ten data augmentation methods. Similarly, for CIFAR-100 (Fig. 16),
the automated methods have convincingly outperformed the classical augmentation meth-
ods, averaging +1.47% gain in performance compared with +2.28% for traditional methods.
Only two classical methods-SuperMix(+2.4%) and GridMask (+2.2%)-are part of the ten
best performing augmentation methods. Figure 17 shows the relative performance of auto-
mated and traditional methods on ImageNet dataset with ResNet-50 backbone. Results are
shown for top-1% and top-5% accuracies. Here, too, AutoMl-based augmentation methods
significantly outperform their classical counterparts. For instance, for the top-5% classifica-
tion accuracies, four out of the top five and eight out of the top ten methods are automated
augmentation methods. These results convincingly demonstrate the superior performance of
data augmentation techniques based AutoML pipelines.

7.3.3 Improving performance by combining classical and AutoML methods

Combining automated data augmentation strategies with multiple classical data augmenta-
tion methods has been shown by several authors (e.g., Atienza [197] and Tang et al. [117]) to
be effective in image classification tasks. Atienza [197] compared the gain in classification
accuracy from AutoAugment (AA) and several other state-of-the-art classical data augmen-
tation techniques (specifically, CutMix [199], CutOut [13], MixUp [119] and AgMax [197])
with a deep-learned augmentation strategy (AA [17]).
They first assessed the performance of each technique separately before pairing AutoAug-
ment with each of the classical augmentation methods. The results obtained by their
experiments showed marked improvement by the paired augmentation strategies over
AutoAugment. The combined augmentations also outperformed each of the classical meth-
ods when applied independently. All cases of combined augmentations resulted in significant
performance improvements over corresponding single augmentations. More importantly,
approaches that utilize a combination of three augmentations also showed a marked improve-
ment over those employing only two augmentation strategies. The results of the study are
summarized in Table 6. The author used CIFAR10 dataset with WideResNet-40-2 backbone
in their experimental setting. Note that the original works AA, MixUp and CutMix do not
report results for the WideResNet-28-10 backbone. Because of the difference in training
settings, the accuracies of individual methods reported (and shown in Fig. 15) are slightly
different from those obtained by the original sources. Nonetheless, they reflect the general
performance of the models based on the specific techniques and dataset configurations. The
experimental results show that while automated data augmentation methods hold significant
promise, there is still scope for the application of classical methods to complement augmen-
tations generated automatically. However, applying multiple augmentations in this manner

123
Data augmentation with automated… 4073

Table 6 Performance gain by Data Augmentation method mAP Acc. Gain

combining classical and
AutoML-based data AA and classical methods applied separately
augmentation methods
Standard (no aug) 95.1 o
AA [17] 95.9 0.8
CutMix [122] 96.2 1.1
CutOut [13] 96.2 1.1
MixUp [119] 95.8 0.7
AgMax [197] 95.6 0.5
Pairing of AA with one classical method
AA + CutOut 96.4 1.3
AA + MixUp 96.0 0.9
AA + CutMix 96.4 1.3
AA[5] + AgMax 96.4(0.5) 1.3
Pairing of classical methods
CutMix + AgMax 96.7(0.5) 1.6
CutOut + AgMax 96.6(0.4) 1.5
MixUp + AgMax 96.3(0.5) 1.2
combination of two classical methods and AA
CutOut+AA + AgMax 97.1(0.7) 2.0
MixUp+AA + AgMax 96.6(0.6) 0.5
CutMix+AA + AgMax 96.8(0.4) 1.7
The models were tested according to the specific settings described by
Atienza [197]

ought to be carefully considered as combining augmentations have been shown to harm

performance in some settings (e.g., [200]) (Table 7).

8 Discussions

8.1 The importance of computational efficiency

Because of the enormous compute resource demand of earlier approaches, most works (e.g.,
FasterAA, PAA AdvAA, OHLAA and AWS) have focused more on balancing generalization
accuracy and computational efficiency. These works have achieved comparable performance
to AutoAugment, the original approach proposed by Cubuk et al. [17] while reducing the
computational overhead. Many studies investigate more efficient optimization techniques
in order to overcome the inherent computational complexity of RL methods. For instance,
results of DDAS, FastAA and RA all demonstrate equivalent or slightly lower generalization
accuracy than AA [17]-one of the original works that introduced the concept of data augmen-
tation based AutoML-but significantly reduced the computation time from 5000 GPU hours
in AA to just 10 (e.g., 5 h in PBA, 3.5 in FastAA, and 0.23 h in FasterAA). Some other works
[e.g., UA and RA) focus on simplifying the search space so that good augmentation policies
can be found using much simpler search strategies or without explicitly conducting search.
These approaches, despite the significant reduction in compute time, have shown competi-

123
4074 A. Mumuni, F. Mumuni

Table 7 Average performance of different search strategies for automated data augmentation
Search strategy Example works Acc. (Avg) Time (Avg)

Reinforcement learning (RL) AA[17], AWS [142], 78.2 210

PAA [33], Ref. [116],
FasterAA [27], BDA
[106]
Evolutionary algorithm (EA) PBA [34]), MODALS 77.2 42
[149], PPBA [148],
TDGA [189]
Gradient descent (GD) DADA [156], AutoDO 78.8 3.5
[201], OHLAA [111],
AdvAA [107]
Bayesian optimization SAPA [120], BO-Aug 77.6 6.3
[146], FastAA [105]
Greedy search and random search GreedyAA [167] Ref. 77.5 8.8
[167, 172]
Searchless methods RA [31], UA [159] 77.6 0.0

tive performance. However, there seems to be little room to further extend their predictive
performance.

8.2 Predictive performance versus computational requirements

As stated earlier, a major problem with AutoML-based data augmentation approaches is the
tendency to excessively expand the training set in a bid to increase predictive performance.
This can lead to situations where small performance gains are achieved at the expense of a dis-
proportionately high increase in model complexity and computational resource requirements.
Consequently, in practical settings, there is often a trade-off between model predictive accu-
racy on one hand, and computational and space complexity and interpretability on the other.
Some state-of-the-art methods such as [202] and [203] provide mechanisms to set priority
levels to achieve the right balance of predictive performance, computational budget and model
interpretability. For instance, Tsamardinos et al. [202] design different model configuration
options, with each configuration prioritizing a specific objective: interpretability, predictive
performance, and minimization of model size through more aggressive feature reduction.
These configuration settings allow users to customize the AutoML pipeline according to
their particular needs and constraints. This capability is widely used in popular AutoML
tools that are designed for generic applications.

8.3 Open problems and research directions

Automated data augmentation methods have in recent years shown a lot of promise and,
as confirmed by the comparative quantitative results in Sect. 7, have outperformed state-of-
the-art manual data augmentation techniques on various benchmarks. Despite the impressive
feat, there are a number of important challenges and unsolved problems. We summarize some
of the key ones here.
• One of the most challenging tasks in the implementation of automated data augmentation,
and indeed in the development of AutoML models in general, is the task of composing

123
Data augmentation with automated… 4075

a good search space that can provide effective solution for the target application. Even
though approaches that learn to automatically generate effective search spaces from data
have been devised, they generally require substantial domain knowledge to design, mak-
ing it challenging for nonexperts to accomplish. Recently, large language models (LMMs)
have been leveraged to automate complex machine learning tasks. Preliminary work by
Yang et al. [204] has shown that with LLMs it will be possible to fully automate the devel-
opment of AutoML and operation of frameworks in a way that allows lay users to build
and use these models. For instance, leveraging the rich knowledge of LLMs will allow
nonexpert developers to easily generate effective, context-relevant search spaces and
associated hyperparameters for data augmentation. In addition, intuitive user interfaces
based on LLMs will enable lay users to simply specify natural language instructions to
configure various settings of the resulting model. These frameworks, through the LMM
interfaces, can also provide useful feedback and suggestions, as well as explanations
about the internal mechanisms and operation of the model.
• The automation of data augmentation is an extremely computationally intensive process.
Given that for a dataset several augmentations can be combined in different ways (e.g.,
varying numbers of transformation operations and different ordering of their application)
and each transformation operation can be applied with infinitely wide range of intensities,
the number of possible augmentations is effectively unlimited. In order to establish the
effectiveness of any new augmentation using appropriate search strategies, it is usually
necessary to conduct exhaustive training and subsequently validate the performance on
a proxy task or the target task. This procedure is a combinatorial problem and, using
current approaches, in some cases it is impractical to arrive at an optimal solution within
a practical time frame. A potentially effective workaround is to leverage large language
model prompts to guide the search process toward better solutions. Such a mechanism
could also provide a means of high-level interaction that allows developers’ priorities
and constraints (e.g., infrastructure limitations, economic constraints, time laxity, etc.)
to be factored in the search process. The performance evaluation step could also benefit
from such new techniques by leveraging real-time feedback from LLMs.
• One of the major problems of machine learning tasks that require data augmentation
is imbalanced data-a situation where some classes are underrepresented while others
are overrepresented. This results in a situation where predictive performance on minor-
ity classes is severely compromised. Even though there are data augmentation-based
workarounds specifically designed to overcome this problem (i.e., to balance imbalanced
data), automated data augmentation methods have not yet been extended to this domain.
Future research is expected to produce dedicated AutoML frameworks specifically aimed
at balancing imbalance training data.
• Currently, the automated data augmentation process is based on creating different vari-
ations of the input data and combing them in a random manner. However, it is known
that the effectiveness of the data augmentation greatly depends on the order of augmen-
tation. We expect future research to provide better theoretical grounding relating to the
(general) ordering of augmentation operations for specific data types and tasks. Such an
approach may rely on knowledge of the representative quality of the data with respect to
the given task as various transformations are applied in sequence. Alternatively, princi-
pled formulations may be devised to quantify in advanced the effect of applying given
transformations in specific sequences.
• In manual data augmentation processes, the human expert relies on intuition and domain
expertise to determine the most suitable augmentations and their strengths. This greatly
reduces the number of artificial samples that need to be created. It also increases robust-

123
4076 A. Mumuni, F. Mumuni

ness and reliability since the most important missing samples will logically be included
in the augmented data. Automated augmentation methods, on the other hand, typically
create a large number of redundant samples. In some situations, this can adversely harm
performance and reliability. Overcoming this challenge requires the incorporation of
context knowledge in automated data augmentation schemes. Context-aware automated
data augmentation strategies would become vital in the near future as the requirements
for machine learning systems are continuously been pushed to the limits. This objective
may be achieved by introducing additional, domain-specific optimizations to force the
generated data to align with real-world constraints. To accomplish this, logical rules or
domain knowledge represented in any appropriate form may be utilized.
• While conventional data augmentation strategies rely on constructing dataset- and task-
specific solutions, automatically learned policies have been shown (e.g., [17]) to be
generally more transferable to new datasets. This property could potentially be exploited
to provide further insights toward developing generic augmentations strategies for a wider
range of deep learning tasks that are independent of datasets. Some interesting results
[205] in this direction have already been reported. Future line of research will involve
devising mechanisms to optimize models based on their transferability. This will enable
the underlying models to encode more transferable architectural features and hyperpa-
rameter settings, thereby generating task- and dataset-agnostic AutoML frameworks.

9 Conclusion

In this work, we present a comprehensive survey of data augmentation methods based on

AutoML techniques. We discuss different ways of realizing data augmentation using AutoML
approaches. In particular, we cover data manipulation, data integration and data synthesis
techniques. We also consider approaches for accomplishing all the subtasks of the data
augmentation process: search space construction, model and hyperparameter optimization,
and evaluation of intermediate solutions. Importantly, we provide a thorough discussion of
the performance of automated data augmentation methods. In this regard, the performance
of automated data augmentation methods are compared with results based on state-of-the-
art classical approaches. The results show that automated data augmentation methods are
currently superior to classical approaches in terms of predictive performance. Their main
drawback, however, is their relatively high complexity of models and the enormous com-
putational requirements. Because of their enormous performance advantage, the automated
methods are expected to improve further and assume an even more dominant role in large-
scale data augmentation tasks.
Author contributions Initial study conception is by AM. AM and FM contributed equally to the study design.
Material preparation, data collection and analysis were performed by AM and FM. The initial draft of the
paper was prepared by FM. AM and FM did all subsequent revisions of the manuscript.

Data availability No datasets were generated or analyzed during the current study.

Declarations

Conflict of interest The authors declare no conflict of interest.

123
Data augmentation with automated… 4077

References
1. Kollias D (2023) Abaw: learning from synthetic data & multi-task learning challenges. In: European
conference on computer vision. Springer, pp 157–172
2. Tabak J, Polić M, Orsag M (2023) Towards synthetic data: dealing with the texture-bias in sim2real
learning. In: intelligent autonomous systems 17: proceedings of the 17th international conference IAS-
17. Springer, pp 630–642
3. Murtaza H, Ahmed M, Khan NF, Murtaza G, Zafar S, Bano A (2023) Synthetic data generation: state
of the art in health care domain. Comput Sci Rev 48:100546
4. Kwon O, Park J, Oh S (2023) Renderable neural radiance map for visual navigation. arXiv
preprintarXiv:2303.00304
5. Zhang M, Zheng S, Bao Z, Hebert M, Wang Y-X (2023) Beyond rgb: scene-property synthesis with
neural radiance fields. In: Proceedings of the IEEE/CVF winter conference on applications of computer
vision, pp 795–805
6. da Silva MV, Silva LH, Junior JDD, Escarpinati MC, Backes AR, Mari JF (2023) Generating synthetic
multispectral images using neural style transfer: a study with application in channel alignment. Comput
Electron Agric 206:107668
7. Feng Y, Chandio BQ, Thomopoulos SI, Thompson M (2023) Variational autoencoders for generating
synthetic tractography-based bundle templates in a low-data setting. bioRxiv, pp 2023–02
8. Wang R, Bashyam V, Yang Z, Yu F, Tassopoulou V, Chintapalli SS, Skampardoni I, Sreepada L, Sahoo
D, Nikita K et al. (2023) Applications of generative adversarial networks in neuroimaging and clinical
neuroscience. In: NeuroImage, p 119898
9. Suri S, Ilyas IF, Ré C, Rekatsinas T (2021) Ember: no-code context enrichment via similarity-based
keyless joins. arXiv preprint arXiv:2106.01501
10. Bai J, Wang J, Li Z, Ding D, Zhang J, Gao J (2021) Atj-net: auto-table-join network for automatic
learning on relational databases. Proc Web Conf 2021:1540–1551
11. Li Y, Yu X, Koudas N (2021) Data acquisition for improving machine learning models. arXiv preprint
arXiv:2105.14107
12. Lopes RG, Yin D, Poole B, Gilmer J, Cubuk ED (2019) Improving robustness without sacrificing
accuracy with patch gaussian augmentation. arXiv preprint arXiv:1906.02611
13. DeVries T, Taylor GW (2017) Improved regularization of convolutional neural networks with cutout.
arXiv preprint arXiv:1708.04552
14. Raileanu R, Goldstein M, Yarats D, Kostrikov I, Fergus R (2021) Automatic data augmentation for
generalization in reinforcement learning. Adv Neural Inf Process Syst 34:5402–5415
15. Ravuri S, Vinyals O (2019) Seeing is not necessarily believing: Limitations of biggans for data augmen-
tation
16. Xue C, Yan J, Yan R, Chu SM, Hu Y, Lin Y (2019) Transferable automl by model sharing over grouped
datasets. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp
9002–9011
17. Cubuk ED, Zoph B, Mane D, Vasudevan V, Le QV (2019) Autoaugment: learning augmentation strategies
from data. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp
113–123
18. Kim D, Koo J, Kim U-M (2022) A survey on automated machine learning: problems, methods and
frameworks. In: International conference on human-computer interaction. Springer, pp 57–70
19. Waring J, Lindvall C, Umeton R (2020) Automated machine learning: review of the state-of-the-art and
opportunities for healthcare. Artif Intell Med 104:101822
20. He X, Zhao K, Chu X (2021) Automl: a survey of the state-of-the-art. Knowl-Based Syst 212:106622
21. Nagarajah T, Poravi G (2019) A review on automated machine learning (automl) systems. In: 2019 IEEE
5th international conference for convergence in technology (I2CT). IEEE, pp 1–6
22. Tuggener L, Amirian M, Rombach K, Lörwald S, Varlet A, Westermann C, Stadelmann T (2019)
Automated machine learning in practice: state of the art and recent results. In: 2019 6th Swiss conference
on data science (SDS). IEEE, pp 31–36
23. Karmaker SK, Hassan MM, Smith MJ, Xu L, Zhai C, Veeramachaneni K (2021) Automl to date and
beyond: challenges and opportunities. ACM Comput Surv (CSUR) 54(8):1–36
24. Yang Z, Sinnott RO, Bailey J, Ke Q (2023) A survey of automated data augmentation algorithms for
deep learning-based image classification tasks. Knowl Inf Syst 65(7):2805–2861
25. Cheung T-H, Yeung D-Y (2023) A survey of automated data augmentation for image classification:
learning to compose, mix, and generate. In: IEEE transactions on neural networks and learning systems
26. Hataya R, Zdenek J, Yoshizoe K, Nakayama H (2022) Meta approach to data augmentation optimization.
In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp 2574–2583

123
4078 A. Mumuni, F. Mumuni

27. Hataya R, Zdenek J, Yoshizoe K, Nakayama H (2022) Faster autoaugment: learning augmentation
strategies using backpropagation. In: European conference on computer vision. Springer, pp 1–16
28. Li Y, Hu G, Wang Y, Hospedales T, Robertson NM, Yang Y (2020) Differentiable automatic data
augmentation. In: European conference on computer vision. Springer, pp 580–595
29. Mounsaveng S, Laradji I, Ben Ayed I, Vazquez D, Pedersoli M (2021) Learning data augmentation with
online bilevel optimization for image classification. In: Proceedings of the IEEE/CVF winter conference
on applications of computer vision, pp 1691–1700
30. Miao H, Rahman LT (2020) Multi-class traffic sign classification using autoaugment and spatial trans-
former. CS230. Retrieved January 27, 2025, from [Link]
[Link]
31. Cubuk ED, Zoph B, Shlens J, Le QV (2020) Randaugment: practical automated data augmentation with
a reduced search space. In: Proceedings of the IEEE/CVF conference on computer vision and pattern
recognition workshops, pp 702–703
32. Kashima T, Yamada Y, Saito S (2020) Joint search of data augmentation policies and network architec-
tures. arXiv preprint arXiv:2012.09407
33. Lin S, Yu T, Feng R, Li X, Jin X, Chen Z (2021) Local patch autoaugment with multi-agent collaboration.
arXiv e-prints, arXiv–2103
34. Ho D, Liang E, Chen X, Stoica I, Abbeel (2019) Population based augmentation: efficient learning of
augmentation policy schedules. In: International conference on machine learning. PMLR, pp 2731–2741
35. Xu M, Yoon S, Fuentes A, Park DS (2023) A comprehensive survey of image augmentation techniques
for deep learning. Pattern Recognit 137:109347
36. Niu T, Bansal M (2019) Automatically learning data augmentation policies for dialogue tasks. arXiv
preprint arXiv:1909.12868
37. Ren S, Zhang J, Li L, Sun X, Zhou J (2021) Text autoaugment: learning compositional augmentation
policy for text classification. arXiv preprint arXiv:2109.00523
38. Dai H, Liu Z, Liao W, Huang X, Cao Y, Wu Z, Zhao L, Xu S, Liu W, Liu N, et al. (2023) Auggpt:
Leveraging chatgpt for text data augmentation. arXiv preprint arXiv:2302.13007
39. Zhao H, Chen H, Ruggles TA, Feng Y, Singh D, Yoon H-J (2024) Improving text classification with
large language model-based data augmentation. Electronics 13(13):2535
40. Tornede A, Deng D, Eimer T, Giovanelli J, Mohan A, Ruhkopf T, Segel S, Theodorakopoulos D, Tornede
T, Wachsmuth H et al., (2023) Automl in the age of large language models: current challenges, future
opportunities and risks. arXiv preprint arXiv:2306.08107
41. Margeloiu A, Bazaga A, Simidjievski N, Liò, Jamnik M (2024) Tabmda: tabular manifold data augmen-
tation for any classifier using transformers with in-context subsetting. arXiv preprintarXiv:2406.01805
42. Zhang S, Balog K (2019) Auto-completion for data cells in relational tables. In: Proceedings of the 28th
ACM international conference on information and knowledge management, pp 761–770
43. Fang J, Tang C, Cui Q, Zhu F, Li L, Zhou J, Zhu W (2022) Semi-supervised learning with data aug-
mentation for tabular data. In: Proceedings of the 31st ACM international conference on information &
knowledge management, pp 3928–3932
44. Chepurko N, Marcus R, Zgraggen E, Fernandez RC, Kraska T, Karger D (2020) Arda: automatic rela-
tional data augmentation for machine learning. arXiv preprint arXiv:2003.09758
45. Bazrafkan S, Nedelcu T, Filipczuk, Corcoran, (2017) Deep learning for facial expression recognition: A
step closer to a smartphone that knows your moods. In: 2017 IEEE international conference on consumer
electronics (ICCE). IEEE, 217–220
46. Gao B, Gouk H, Hospedales TM (2021) Searching for robustness: loss learning for noisy classification
tasks. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 6670–6679
47. Gao B (2023) Meta-learning to optimise: loss functions and update rules. [Doctoral dissertation, The Uni-
versity of Edinburgh]. Edinburgh Research Archive. [Link]
48. Kim Y-Y, Song K, Jang J, Moon I-C (2021) Lada: look-ahead data acquisition via augmentation for deep
active learning. Adv Neural Inf Process Syst 34:22919–22930
49. Yao Q, Yang H, Han B, Niu G, Kwok JT-Y (2020) Searching to exploit memorization effect in learning
with noisy labels. In: International conference on machine learning. PMLR, pp 10789–10798
50. Shu J, Yuan X, Meng D, Xu Z (2023) Dac-mr: data augmentation consistency based meta-regularization
for meta-learning. arXiv preprint arXiv:2305.07892
51. Gao C, Liu C, Shu J, Liu F, Liu J, Yang L, Gao X, Meng D (2024) “Are dense labels always necessary
for 3d object detection from point cloud?”. arXiv preprint arXiv:2403.02818
52. Gao B, Gouk H, Yang Y, Hospedales T (2022) Loss function learning for domain generalization by
implicit gradient. In: International conference on machine learning. PMLR, pp 7002–7016
53. Kumar A, Naughton J, Patel JM (2015) Learning generalized linear models over normalized data. In:
Proceedings of the 2015 ACM SIGMOD international conference on management of data, pp 1969–1984

123
Data augmentation with automated… 4079

54. Esmailoghli M, Quiané-Ruiz J-A, Abedjan Z (2021) Cocoa: correlation coefficient-aware data augmen-
tation. In: EDBT, pp 331–336
55. Koutras C, Siachamis G, Ionescu A, Psarakis K, Brons J, Fragkoulis M, Lofi C, Bonifati A, Katsi-
fodimos A (2021) Valentine: evaluating matching techniques for dataset discovery. In: 2021 IEEE 37th
international conference on data engineering (ICDE). IEEE, pp 468–479
56. Li G, Zhou X, Cao L (2021) Ai meets database: Ai4db and db4ai. In: Proceedings of the 2021 international
conference on management of data, pp 2859–2866
57. Elsken T, Metzen JH, Hutter F (2019) Neural architecture search: a survey. J Mach Learn Res 20(1):1997–
2017
58. Kumar A, Naughton J, Patel JM, Zhu X (2016) To join or not to join? Thinking twice about joins before
feature selection. In: Proceedings of the 2016 international conference on management of data, pp 19–34
59. Mustafa A, Rahimi Azghadi M (2021) Automated machine learning for healthcare and clinical notes
analysis. Computers 10(2):24
60. Nikitin NO, Vychuzhanin Sarafanov M, Polonskaia IS, Revin I, Barabanova IV, Maximov G, Kalyuzh-
naya AV, Boukhanovsky A (2022) Automated evolutionary approach for the design of composite machine
learning pipelines. Futur Gener Comput Syst 127:109–125
61. Shi X, Mueller J, Erickson N, Li M, Smola A (2021) Multimodal automl on structured tables with text
fields. In: 8th ICML workshop on automated machine learning (AutoML)
62. Erickson N, Shi X, Sharpnack J, Smola A (2022) Multimodal automl for image, text and tabular data.
In: Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pp
4786–4787
63. Erickson N, Mueller J, Shirkov A, Zhang H, Larroy, Li., Smola A (2020) Autogluon-tabular: Robust
and accurate automl for structured data. In: arXiv preprint arXiv:2003.06505
64. Nargesian F, Asudeh A, Jagadish H (2022) Responsible data integration: next-generation challenges. In:
Proceedings of the 2022 international conference on management of data, pp 2458–2464
65. Chai C, Liu J, Tang N, Li G, Luo Y (2022) Selective data acquisition in the wild for model charging.
PVLDB 15(7):1466–1478
66. Shende MK, Feijoo-Lorenzo AE, Bokde ND (2022) cleanTS: Automated (AutoML) tool to clean uni-
variate time series at microscales. Neurocomputing 500:155–176
67. Mangrulkar A, Rane S, Sunnapwar V (2020) Image-based bio-cad modeling: overview, scope, and
challenges. J Phys Conf ser 1706:012189. [Link]
68. Nikolenko SI (2021) Synthetic data for deep learning. Springer, Germany, p 174
69. de Melo CM, Torralba A, Guibas L, DiCarlo J, Chellappa R, Hodgins J (2021) Next-generation deep
learning based on simulators and synthetic data. Trends Cognit Sci
70. Tremblay J, Prakash A, Acuna D, Brophy M, Jampani V, Anil C, To T, Cameracci E, Boochoon S,
Birchfield S (2018) Training deep networks with synthetic data: Bridging the reality gap by domain
randomization. In: Proceedings of the IEEE conference on computer vision and pattern recognition
workshops, pp 969–977
71. Starly B, Fang Z, Sun W, Shokoufandeh A, Regli W (2005) Three-dimensional reconstruction for
medical-cad modeling. Comput-Aided Des Appl 2(1–4):431–438
72. Chang AX, Funkhouser T, Guibas L, Hanrahan, Huang Q, Li Z, Savarese S, Savva M, Song S, Su H
et al. (2015) Shapenet: an information-rich 3d model repository. arXiv preprint arXiv:1512.03012
73. Mishra S, Panda R, Phoo C, Chen C-FR, Karlinsky L, Saenko K, Saligrama V, Feris RS (2022) Task2sim:
towards effective pre-training and transfer from synthetic data. In: Proceedings of the IEEE/CVF con-
ference on computer vision and pattern recognition, pp 9194–9204
74. Wang Y, Mu N, Grandi D, Savva N, Steinhardt J (2022) A3d: studying pretrained representations with
programmable datasets. In: Proceedings of the IEEE/CVF conference on computer vision and pattern
recognition, pp 4878–4889
75. Du Y, Watkins O, Darrell T, Abbeel, Pathak (2021) Auto-tuned sim-to-real transfer. In: 2021 IEEE
international conference on robotics and automation (ICRA). IEEE, pp 1290–1296
76. Thompson JL (2022) Augmenting biological pathway extraction with synthetic data and active learning,
Ph.D. dissertation, University of Missouri–Columbia
77. Owoyele O, Pal P, Vidal Torreira A, Probst D, Shaxted M, Wilde M, Senecal PK (2022) Application of an
automated machine learning-genetic algorithm (automl-ga) coupled with computational fluid dynamics
simulations for rapid engine design optimization. Int J Engine Res 23(9):1586–1601
78. Williams RJ (1992) Simple statistical gradient-following algorithms for connectionist reinforcement
learning. Mach Learn 8(3):229–256
79. Behl HS, Baydin AG, Gal R, Torr H, Vineet V (2020) Autosimulate:(quickly) learning synthetic data
generation. In: European conference on computer vision. Springer, pp 255–271
80. Ruiz N, Schulter S, Chandraker M (2018) Learning to simulate. In: arXiv preprint arXiv:1810.02513

123
4080 A. Mumuni, F. Mumuni

81. Shirobokov S, Belavin V, Kagan M, Ustyuzhanin A, Baydin AG (2020) Black-box optimization with
local generative surrogates. Adv Neural Inf Process Syst 33:14650–14662
82. Sun D, Vlasic D, Herrmann C, Jampani V, Krainin M, Chang H, Zabih R, Freeman WT, Liu C (2021)
Autoflow: learning a better training set for optical flow. In: Proceedings of the IEEE/CVF conference
on computer vision and pattern recognition, pp 10093–10102
83. Kar A, Prakash A, Liu M-Y, Cameracci E, Yuan J, Rusiniak M, Acuna D, Torralba A, Fidler S (2019)
Meta-sim: learning to generate synthetic datasets. In: Proceedings of the IEEE/CVF international con-
ference on computer vision, pp 4551–4560
84. Ge Y, Behl H, Xu J, Gunasekar S, Joshi, Song Y, Wang X, Itti L, Vineet V (2022) Neural-sim: learning
to generate training data with nerf. In: European conference on computer vision. Springer, pp 477–493
85. Han Y, Luo K, Luo A, Liu J, Fan H, Luo G, Liu S (2022) Realflow: Em-based realistic optical flow
dataset generation from videos. In: European conference on computer vision. Springer, pp 288–305
86. Kortylewski A, Egger B, Schneider A, Gerig T, Morel-Forster A, Vetter T (2019) Analyzing and reducing
the damage of dataset bias to face recognition with synthetic data. In: Proceedings of the IEEE/CVF
conference on computer vision and pattern recognition workshops
87. Ebadi SE, Dhakad S, Vishwakarma S, Wang C, Jhang Y-C, Chociej M, Crespi A, Thaman A, Ganguly
S (2022) Psp-hdri +: a synthetic dataset generator for pre-training of human-centric computer vision
models. arXiv preprint arXiv:2207.05025
88. Mayer N, Ilg E, Hausser, Fischer, Cremers D, Dosovitskiy A, Brox T (2016) A large dataset to train
convolutional networks for disparity, optical flow, and scene flow estimation. In: Proceedings of the
IEEE conference on computer vision and pattern recognition, pp 4040–4048
89. Dosovitskiy A, Fischer, Ilg E, Hausser, Hazirbas C, Golkov V, Van Der Smagt, Cremers D, Brox T (2015)
Flownet: learning optical flow with convolutional networks. In: Proceedings of the IEEE international
conference on computer vision, pp 2758–2766
90. Butler DJ, Wulff J, Stanley GB, Black MJ (2012) A naturalistic open source movie for optical flow
evaluation. In: European conference on computer vision. Springer, pp 611–625
91. Geiger A, Lenz, Urtasun R (2012) Are we ready for autonomous driving? The kitti vision benchmark
suite. In: 2012 IEEE conference on computer vision and pattern recognition. IEEE, pp 3354–3361
92. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer W (2002) Smote: synthetic minority over-sampling
technique. J Artif Intell Res 16:321–357
93. Wang W, Pai T-W (2023) enhancing small tabular clinical trial dataset through hybrid data augmentation:
combining smote and wcgan-gp. Data 8(9):135
94. Dablain D, Krawczyk B, Chawla NV (2022) Deepsmote: fusing deep learning and smote for imbalanced
data. IEEE Trans Neural Netw Learn Syst 34(9):6390–6404
95. Aragão MVC, de Freitas Carvalho M, de Morais Pereira T, de Figueiredo FA, Mafra SB (2024) Enhancing
AutoML performance for imbalanced tabular data classification: a self-balancing pipeline. Res Sq.
[Link] (preprint)
96. Rashidi H, Albahra S, Rubin B, Hu B (2023) STNG (Synthetic Tabular Neural Generator): a novel and
fully automated platform for synthetic tabular data generation and validation. Res Sq. [Link]
10.21203/[Link]-3716775/v1 (preprint)
97. Xu S, Semnani SJ, Campagna G, Lam MS (2020) Autoqa: from databases to qa semantic parsers with
only synthetic training data. arXiv preprint arXiv:2010.04806
98. Li Z, Si L, Guo C, Yang Y, Cao Q (2024) Data augmentation for text-based person retrieval using large
language models. arXiv preprint arXiv:2405.11971
99. Glazkova A, Zakharova O (2024) Evaluating llm prompts for data augmentation in multi-label classifi-
cation of ecological texts. arXiv preprint arXiv:2411.14896
100. Xu J, Li J, Liu Z, Suryanarayanan NAV, Zhou G, Guo J, Iba H, Tei K (2024) Large language models
synergize with automated machine learning. arXiv preprint arXiv:2405.03727
101. Ma L, Li N, Yu G, Geng X, Cheng S, Wang X, Huang M, Jin Y (2023) Pareto-wise ranking classifier
for multiobjective evolutionary neural architecture search. IEEE Trans Evolut Comput 28(3):570–581.
[Link]
102. Ma L, Kang H, Yu G, Li Q, He Q (2024) Single-domain generalized predictor for neural architecture
search system. IEEE Trans Comput 73(5):1400–1413. [Link]
103. Sun Y, Xue B, Zhang M, Yen GG, Lv J (2020) Automatically designing cnn architectures using the
genetic algorithm for image classification. IEEE Trans Cybernet 50(9):3840–3854
104. Liu A, Huang Z, Huang Z, Wang N (2021) Direct differentiable augmentation search. In: Proceedings
of the IEEE/CVF international conference on computer vision, pp 12219–12228
105. Lim S, Kim I, Kim T,Kim C, Kim S (2019) Fast autoaugment. In: Advances in neural information
processing systems, vol 32

123
Data augmentation with automated… 4081

106. Lu S, Zhao M, Yuan S, Wang X, Yang L, Niu D (2023) Bda: bandit-based transferable autoaugment. In:
Proceedings of the 2023 SIAM international conference on data mining (SDM). SIAM, pp 550–558
107. Zhang X, Wang Q, Zhang J, Zhong Z (2019) Adversarial autoaugment. arXiv preprint arXiv:1912.11188
108. Krizhevsky A (2009) Imagenet classification with deep convolutional neural networks. Master’s thesis,
University of Toronto. [Link]
109. Netzer Y, Wang T, Coates A, Bissacco A, Wu B, Ng AY (2011) Reading digits in natural images with
unsupervised feature learning
110. Wei L, Xiao A, Xie L, Zhang X, Chen X, Tian Q (2020) Circumventing outliers of autoaugment with
knowledge distillation. In: European conference on computer vision. Springer, pp 608–625
111. Lin C, Guo M, Li C, Yuan X, Wu W, Yan J, Lin D, Ouyang W (2019) Online hyper-parameter learning
for auto-augmentation strategy. In: Proceedings of the IEEE/CVF international conference on computer
vision, pp 6579–6588
112. Gao Y, Tang Z, Zhou M, Metaxas D (2021) Enabling data diversity: efficient automatic augmentation
via regularized adversarial training. In: International conference on information processing in medical
imaging. Springer, pp 85–97
113. Zhao A, Balakrishnan G, Durand F, Guttag JV, Dalca AV (2019) Data augmentation using learned
transformations for one-shot medical image segmentation. In: Proceedings of the IEEE/CVF conference
on computer vision and pattern recognition, pp 8543–8553
114. Jaderberg M, Simonyan K, Zisserman A (2015) Spatial transformer networks. In: Advances in neural
information processing systems, vol 28
115. Dai J, Qi H, Xiong Y, Li Y, Zhang G, Hu H, Wei Y (2017) Deformable convolutional networks. In:
Proceedings of the IEEE international conference on computer vision, pp 764–773
116. Chu C-T, Rohmatillah M, Lee C-H, Chien J-T (2022) Augmentation strategy optimization for language
understanding. In: ICASSP 2022-2022 IEEE international conference on acoustics, speech and signal
processing (ICASSP). IEEE, pp 7952–7956
117. Tang Z, Gao Y, Karlinsky L, Sattigeri, Feris R, Metaxas D (2020) Onlineaugment: online data augmen-
tation with less domain knowledge. In: European conference on computer vision. Springer, pp 313–329
118. Cubuk ED, Zoph B, Mane D, Vasudevan V, Le QV (2018) Autoaugment: learning augmentation policies
from data. arXiv preprintarXiv:1805.09501
119. Zhang H, Cisse M, Dauphin YN, Lopez-Paz D (2017) mixup: beyond empirical risk minimization. arXiv
preprint arXiv:1710.09412
120. Hu T-Y, Shrivastava A, Chang J-HR, Koppula H, Braun S, Hwang K, Kalinli O, Tuzel O (2021)
Sapaugment: learning a sample adaptive policy for data augmentation. In: ICASSP 2021-2021 IEEE
international conference on acoustics, speech and signal processing (ICASSP). IEEE, pp 4040–4044
121. Inoue H (2018) Data augmentation by pairing samples for images classification. arXiv preprint
arXiv:1801.02929
122. Yun S, Han D, Oh SJ, Chun S, Choe J, Yoo Y (2019) Cutmix: regularization strategy to train strong clas-
sifiers with localizable features. In: Proceedings of the IEEE/CVF international conference on computer
vision, pp 6023–6032
123. Chinbat V, Bae S-H (2022) Ga3n: generative adversarial autoaugment network. Pattern Recogn
127:108637
124. Peng X, Tang Z, Yang F, Feris RS, Metaxas D (2018) Jointly optimize data augmentation and network
training: Adversarial data augmentation in human pose estimation. In: Proceedings of the IEEE onference
on Computer Vision and Pattern Recognition, pp 2226–2234
125. Liu S, Lu S, Chen X, Feng Y, Xu K, Al-Dujaili, Hong M, O’Reilly (2019) Min-max optimization without
gradients: Convergence and applications to adversarial ml. arXiv preprint arXiv:1909.13806
126. Lee DJ-L, Macke S (2020) A human-in-the-loop perspective on automl: milestones and the road ahead.
IEEE Data Eng Bull
127. Li Y, Wang Z, Xie Y, Ding B, Zeng K, Zhang C (2021) Automl: from methodology to application. In:
Proceedings of the 30th ACM international conference on information & knowledge management, pp
4853–4856
128. Zheng Yu, Zhang Zhi, Yan Shen, Zhang Mi (2022) Deep Autoaugment. In: International conference on
learning representations. [Link]
129. Liu Z, Jin H, Wang T-H, Zhou K, Hu X (2021) Divaug: plug-in automated data augmentation with
explicit diversity maximization. In: Proceedings of the IEEE/CVF international conference on computer
vision, pp 4762–4770
130. Chen Y, Zhang, Kong T, Li Y, Zhang X, Qi L, Sun J, Jia J (2022) Scale-aware automatic augmentations
for object detection with dynamic training. IEEE Trans Pattern Anal Mach Intell
131. Zhou F, Li J, Xie C, Chen F, Hong L, Sun R, Li Z (2021) Metaaugment: sample-aware data augmentation
policy learning. Proc AAAI Conf Artif Intell 35(12):11097–11105

123
4082 A. Mumuni, F. Mumuni

132. Cheung T-H, Yeung D-Y (2021) Adaaug: learning class-and instance-adaptive data augmentation poli-
cies. In: International conference on learning representations
133. Miao N, Rainforth T, Mathieu E, Dubois Y, Teh YW, Foster A, Kim H (2023) Learning instance-specific
augmentations by capturing local invariances
134. Yoo J, Kang S (2023) Class-adaptive data augmentation for image classification. IEEE Access 11:26393–
26402
135. Yang D, Myronenko A, Wang X, Xu Z, Roth HR, Xu D (2021) T-automl: automated machine learning
for lesion segmentation using transformers in 3d medical imaging. In: Proceedings of the IEEE/CVF
international conference on computer vision, pp 3962–3974
136. Lopes V, Gaspar A, Alexandre LA, Cordeiro J (2021) An automl-based approach to multimodal image
sentiment analysis. In: 2021 international joint conference on neural networks (IJCNN). IEEE, pp 1–9
137. Chu X, He X (2022) Medpipe: end-to-end joint search of data augmentation policy and neural architecture
for 3d medical image classification
138. Liu H, Simonyan K, Yang Y (2018) Darts: differentiable architecture search. arXiv preprint
arXiv:1806.09055
139. Snoek J, Larochelle H, Adams R (2012) Practical bayesian optimization of machine learning algorithms.
In: Advances in neural information processing systems, vol 25
140. Bäck T, Fogel DB, Michalewicz Z (1997) Handbook of evolutionary computation. Release 97(1):B1
141. Kaelbling L, Littman ML, Moore AW (1996) Reinforcement learning: a survey. J Artif Intell Res 4:237–
285
142. Tian K, Lin C, Sun Zhou L, Yan J, Ouyang W (2020) Improving auto-augment via augmentation-wise
weight sharing. Adv Neural Inf Process Syst 33:19088–19098
143. Gowda SN, Rohrbach M, Keller F, Sevilla-Lara L (2022) Learn2augment: learning to composite videos
for data augmentation in action recognition. In:European conference on computer vision. Springer, pp
242–259
144. White DJ (1993) A survey of applications of Markov decision processes. J Oper Res Soc 44(11):1073–
1096
145. Spaan MT (2012) Partially observable markov decision processes. In: Reinforcement learning. Springer,
pp 387–414
146. Zhang C, Li X, Zhang Z, Cui J, Yang B (2022) Bo-aug: learning data augmentation policies via bayesian
optimization. Appl Intell 53:1–16
147. Wang X, Jin Y, Schmitt S, Olhofer M (2023) Recent advances in bayesian optimization. ACM Comput
Surv 55(13s):1–36
148. Cheng S, Leng Z, Cubuk ED, Zoph B, Bai C, Ngiam J, Song Y, Caine B, Vasudevan V, Li C et al.
(2020) Improving 3d object detection through progressive population based augmentation. In: European
conference on computer vision. Springer, pp 279–294
149. Cheung T-H, Yeung D-Y (2020) Modals: modality-agnostic automated data augmentation in the latent
space. In: International conference on learning representations
150. Grathwohl W, Choi D, Wu Y, Roeder G, Duvenaud D (2017) Backpropagation through the void: opti-
mizing control variates for black-box gradient estimation. arXiv preprint arXiv:1711.00123
151. Tucker G, Mnih A, Maddison CJ, Lawson J, Sohl-Dickstein J (2017) Rebar: low-variance, unbiased
gradient estimates for discrete latent variable models. In: Advances in Neural Information Processing
Systems, vol 30
152. Zhou K, Hong L, Hu S, Zhou F, Ru B, Feng J, Li Z (2021) Dha: end-to-end joint optimization of data
augmentation policy, hyper-parameter and architecture. arXiv preprint arXiv:2109.05765
153. Wang X, Chu X, Yan J, Yang X (2021) Daas: differentiable architecture and augmentation policy search.
arXiv preprint arXiv:2109.15273
154. Xu J, Li M, Zhu Z (2020) Automatic data augmentation for 3d medical image segmentation. In: Medical
image computing and computer assisted intervention–MICCAI 2020: 23rd international conference,
Lima, Peru, October 4–8, 2020, Proceedings, Part I 23. Springer, pp 378–387
155. Akimoto Y, Shirakawa S, Yoshinari N, Uchida K, Saito S, Nishida K (2019) Adaptive stochastic nat-
ural gradient method for one-shot neural architecture search. In: International conference on machine
learning. PMLR, pp 171–180
156. Li Y, Hu G, Wang Y, Hospedales T, Robertson NM, Yang Y (2020) Dada: differentiable automatic data
augmentation. arXiv preprint arXiv:2003.03780
157. Luo Z, He Z, Wang J, Dong M, Huang J, Chen M, Zheng B (2021) Autosmart: an efficient and automatic
machine learning framework for temporal relational data. In: Proceedings of the 27th ACM SIGKDD
conference on knowledge discovery & data mining, pp 3976–3984

123
Data augmentation with automated… 4083

158. Alaa A, Schaar M (2018) Autoprognosis: automated clinical prognostic modeling via bayesian opti-
mization with structured kernel learning. In: International conference on machine learning. PMLR, pp
139–148
159. LingChen TC, Khonsari A, Lashkari A, Nazari MR, Sambee JS, Nascimento MA (2020) Uniformaug-
ment: asearch-free probabilistic data augmentation approach. arXiv preprint arXiv:2003.14348
160. Feurer M, Hutter F (2018) Towards further automation in automl. In: ICML AutoML workshop, vol 13
161. Olson RS, Moore JH (2016) Tpot: a tree-based pipeline optimization tool for automating machine
learning. In: Workshop on automatic machine learning. PMLR, pp 66–74
162. Zhao Y (2022) Autodes: automl pipeline generation of classification with dynamic ensemble strategy
selection. arXiv preprint arXiv:2201.00207
163. Komer B, Bergstra J, Eliasmith C (2014) “Hyperopt-sklearn: automatic hyperparameter configuration
for scikit-learn,” in ICML workshop on AutoML, 9. Citeseer, 50
164. Thornton C, Hutter F, Hoos HH, Leyton-Brown K (2013) Auto-weka: Combined selection and hyper-
parameter optimization of classification algorithms. In: Proceedings of the 19th ACM SIGKDD
international conference on Knowledge discovery and data mining, pp 847–855
165. Swearingen T, Drevo W, Cyphers B, Cuesta-Infante A, Ross A, Veeramachaneni K (2017) Atm: a
distributed, collaborative, scalable system for automated machine learning. In: 2017 IEEE international
conference on big data (big data). IEEE, pp 151–162
166. LeDell E, Poirier S (2020) H2o automl: scalable automatic machine learning. In: Proceedings of the
AutoML Workshop at ICML, p 2020
167. Naghizadeh A, Abavisani M, Metaxas DN (2020) Greedy autoaugment. Pattern Recogn Lett 138:624–
630
168. Müller SG, Hutter SG (2021) Trivialaugment: tuning-free yet state-of-the-art data augmentation. In:
Proceedings of the IEEE/CVF international conference on computer vision, pp 774–782
169. Karnopp DC (1963) Random search techniques for optimization problems. Automatica 1(2–3):111–121
170. LaValle SM, Branicky MS, Lindemann SR (2004) On the relationship between classical grid search and
probabilistic roadmaps. Int J Robot Res 23(7–8):673–692
171. Wilt C, Thayer J, Ruml W (2010) A comparison of y search algorithms. In: Proceedings of the interna-
tional symposium on combinatorial search 1(1):129–136
172. Momeny M, Neshat AA, Gholizadeh A, Jafarnezhad A, Rahmanzadeh E, Marhamati M, Moradi B,
Ghafoorifar A, Zhang Y-D (2022) Greedy autoaugment for classification of mycobacterium tuberculosis
image via generalized deep CNN using mixed pooling based on minimum square rough entropy. Comput
Biol Med 141:105175
173. Yao Q, Wang M, Chen Y, Dai W, Li Y-F, Tu W-W, Yang Q, Yu Y (2018) Taking human out of learning
applications: a survey on automated machine learning. arXiv preprint arXiv:1810.13306
174. Zöller M-A, Huber MF (2021) Benchmark and survey of automated machine learning frameworks. J
Artif Intell Res 70:409–472
175. Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image
database, In: 2009 IEEE conference on computer vision and pattern recognition. IEEE, pp 248–255
176. Lin T-Y, Maire M, Belongie S, Hays J, Perona, Ramanan D, Dollár, Zitnick CL (2014) Microsoft coco:
common objects in context, In: Computer Vision–ECCV 2014: 13th European Conference, Zurich,
Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer, pp 740–755
177. Anguelov D, Dulong C, Filip D, Frueh C, Lafon S, Lyon R, Ogale A, Vincent L, Weaver J (2010) Google
street view: capturing the world at street level. Computer 43(6):32–38
178. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein
M et al (2015) Imagenet large scale visual recognition challenge. Int J Comput Vision 115:211–252
179. Zagoruyko S, Komodakis N (2016) Wide residual networks. arXiv preprint arXiv:1605.07146
180. Gastaldi X (2017) Shake-shake regularization. arXiv preprint arXiv:1705.07485
181. Lin T-Y, Dollár, Girshick R, He K, Hariharan B, Belongie S (2017) Feature pyramid networks for
object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition,
pp 2117–2125
182. He K, Zhang X, Ren S, Sun J (2016) Identity mappings in deep residual networks, In: Computer
vision–ECCV 2016: 14th European conference, Amsterdam, The Netherlands, October 11–14, 2016,
Proceedings, Part IV 14
183. Lin T-Y, Girshick Goyal R, He K, Dollár (2017) Focal loss for dense object detection, In: Proceedings
of the IEEE international conference on computer vision, pp 2980–2988
184. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of
the IEEE conference on computer vision and pattern recognition, pp 770–778
185. Chen, Liu S, Zhao H, Jia J (2020) Gridmask data augmentation. arXiv preprint arXiv:2001.04086

123
4084 A. Mumuni, F. Mumuni

186. Zhou F, Li J,Xie C, Chen F, Hong L, Sun R, Li Z (2020) Metaaugment: sample-aware data augmentation
policy learning. arXiv preprint arXiv:2012.12076
187. Dong X, Potter M, Kumar G, Tsai Y-C, Saripalli VR (2021) Automating augmentation through random
unidimensional search. arXiv preprint arXiv:2106.08756
188. Suzuki T (2022) Teachaugment: data augmentation optimization using teacher knowledge. In: Proceed-
ings of the IEEE/CVF conference on computer vision and pattern recognition, pp 10904–10914
189. Terauchi A, Mori N (2021) Evolutionary approach for autoaugment using the thermodynamical genetic
algorithm. Proc AAAI Conf Artif Intell 35(11):9851–9858
190. Dabouei A, Soleymani S, Taherkhani F, Nasrabadi NM (2021) Supermix: supervising the mixing data
augmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition,
pp 13794–13803
191. Huang G, Sun Y, Liu Z, Sedra D, Weinberger KQ (2016) Deep networks with stochastic depth. In:
European conference on computer vision. Springer, pp 646–661
192. Wang Y, Pan X, Song S, Zhang H, Huang G, Wu C (2019) Implicit semantic data augmentation for deep
networks. In: Advances in neural information processing systems, vol 32
193. Verma V, Lamb A, Beckham C, Najafi A, Mitliagkas I, Lopez-Paz D, Bengio Y (2019) Manifold mixup:
Better representations by interpolating hidden states. In: International conference on machine learning.
PMLR, 2019, pp 6438–6447
194. Kim J-H, Choo W, Song HO (2020) Puzzle mix: exploiting saliency and local statistics for optimal
mixup. In: International conference on machine learning. PMLR, pp 5275–5285
195. Khan A, Fraz K (2020) Post-training iterative hierarchical data augmentation for deep networks. Adv
Neural Inf Process Syst 33:689–699
196. Uddin A, Monira M, Shin W, Chung T, Bae S-H et al. (2020) Saliencymix: a saliency guided data
augmentation strategy for better regularization. arXiv preprint arXiv:2006.01791
197. Atienza R (2022) Improving model generalization by agreement of learned representations from data
augmentation. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision,
pp 372–381
198. Zhong Z, Zheng L, Kang G, Li S, Yang Y (2020) Random erasing data augmentation. Proc AAAI Conf
Artif Intell 34(07):13001–13008
199. Walawalkar D, Shen Z, Liu Z, Savvides M (2020) Attentive cutmix: an enhanced data augmentation
approach for deep learning based image classification. arXiv preprint arXiv:2003.13048
200. Wen Y, Jerfel G, Muller R, Dusenberry MW, Snoek J, Lakshminarayanan B, Tran D (2020) Combining
ensembles and data augmentation can harm your calibration. arXiv preprint arXiv:2010.09875
201. Gudovskiy D, Rigazio L, Ishizaka S, Kozuka K, Tsukizawa S (2021) Autodo: robust autoaugment for
biased data with label noise via scalable probabilistic implicit differentiation. In: Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 16601–16610
202. Tsamardinos I, Charonyktakis Papoutsoglou G, Borboudakis G, Lakiotaki K, Zenklusen JC, Juhl H,
Chatzaki E, Lagani V (2022) Just add data: automated predictive modeling for knowledge discovery and
feature selection. NPJ Precis Oncol 6(1):1–17
203. Xanthopoulos I, Tsamardinos I, Christophides V, Simon E, Salinger A (2020) Putting the human back
in the automl loop. In: EDBT/ICDT Workshops
204. Yang Z, Zeng W, Jin S, Qian C, Luo, Liu W (2024) Autommlab: automatically generating deployable
models from language instructions for computer vision tasks. arXiv preprint arXiv:2402.15351
205. Cao K, You J, Liu J, Leskovec J (2023) Autotransfer: Automl with knowledge transfer–an application
to graph neural networks. arXiv preprint arXiv:2303.07669

Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and
institutional affiliations.

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under
a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted
manuscript version of this article is solely governed by the terms of such publishing agreement and applicable
law.

123
Data augmentation with automated… 4085

Alhassan Mumuni is currently a researcher at the Department of Elec-

trical and Electronics Engineering, Cape Coast Technical University
(CCTU), Ghana. His areas of interest are knowledge and data engineer-
ing, machine learning, robotics, virtual and augmented reality, auto-
matic control, and embedded systems.

Fuseini Mumuni specialized in Robotics and Automated Systems at the

Department of Robotics Technology and Automation of Technological
Processes, Saint Petersburg State Electrotechnical University, Russia.
He is currently at the University of Mines and Technology (UMaT),
Tarkwa, Ghana. His research interests include control systems, robotics
and artificial intelligence, cognitive systems and machine vision.

123