Mel spectrogram based audio forgery detection

using CNN
Arda USTUBIOGLU
Trabzon University
Beste USTUBIOGLU (  bustubioglu@[Link] )
Karadeniz Technical University
Guzin ULUTAS
Karadeniz Technical University

Research Article

Keywords: Copy-move forgery detection ,Audio forgery ,Audio forensic ,Spectrogram-CNN

Posted Date: July 19th, 2022

DOI: [Link]

License:   This work is licensed under a Creative Commons Attribution 4.0 International License.
Read Full License

Page 1/24
Abstract
In this time of technology, digital speech can be created and falsified by a very diverse of hardware and
software technologies. Audio Copy-move forgery is an audio forgery technique that goals to create forged
audio by hiding undesirable words or repeating wanted words in the identical speech. Therefore, audio
authentication has been a necessary requisition. In this study, an effective approach to spectral images
based on audio copy-move forgery detection using Convolutional Neural Networks (CNN) with data
augmentation is proposed. There are only a few hand-crafted methods conducted for the detection of
audio copy-move forgery. None of the existing works on audio copy-move forgery detection has proposed
deep feature learning from speech recording with Mel spectrogram. This is the first method to employ
deep learning with Mel spectrogram of audio for the detection of audio copy-move forgery. The proposed
CNN architecture classifies the suspicious Mel spectrogram images into two classes: original and forged.
The proposed CNN system is successfully trained on these Mel spectrogram images feature extraction.
The proposed algorithm has been tested on our datasets generated from Arabic Speech Corpus and
TIMIT speech database. The results show the effectiveness, robustness to post-processing operations,
and high accuracy of the proposed approach compared to other studies.

1. Introduction
The use of speech recordings in daily life has increased with the improving of recent free developed audio
editing software like Adobe, Audition CC. Thanks to the easy use of such software, even an ordinary user
can easily create forged audio. Copy-move forgery process is one of the frequently used forged audio
generation methods. The attacker copies one or more segments of the speech and pastes this segment or
segments into different positions within the identical audio to create forged speech with copy-move
forgery. As a result of this operation, the semantic meaning of the speech changes. Moreover, this speech
recordings can be submitted to the court as digital proof. For example, the sentence “I will not say any
more, I'm offender.” can be modified to “I will not say any more, I'm not offender.” by copying and pasting
the word “not” said by the identical talker in the identical audio recording. Since the speech records are
also used as digital evidence, determination of the authentication of these records has made this area an
active research. Considering the fact that the speech records are digital evidence, it is very important to
authenticate these records.

Most of the existing audio authentication techniques can be classified into active and passive
techniques. Active techniques generate a watermark that contains information about the original audio
data using some method and inserts it into the audio without any clues. Other party must regenerate
watermark during authentication and must check the reconstructed one with received one. If they are
inconsistent, the audio is forged and original otherwise. The most important disadvantage of this method
is that it requires a special device or software to achieve watermark insertion. Unlike active methods,
passive methods are more preferred because they do not need any preceding knowledge (watermark).
Passive methods given below used in audio authentication utilized from traces left during the creation of
forged speech.
Page 2/24
In the literature, there are a lot of methods to detect audio forgery types such as splicing, resampling,
deletion, re-quantizing, insertion, copy-move, re-compression. However, among these methods, the number
of methods detecting audio copy-move forgery is quite limited. This is mainly because it is usually
unnoticeable because of the forged segments are reproduced from the identical audio file. A malicious
user can easily create a forged speech recording with copy move forgery by copying some segments in
the speech recording and pasting these segments into different parts of the same speech to change the
original content of the speech recording. At the same time, attacker applies post-processing operations
such as adding noise, filtering and compression to the forged speech recording to hide the traces of copy-
move forgery. Figure 1 shows an example of a copy-move forgery operation. Original audio file received
from the TIMIT database (“[Link]”) are shown in Fig. 1(a). The first red part of the audio in Fig. 1(b)
was copied and pasted at the end of the identical audio to generate the copy-move forged audio from
Fig. 1 (a). As another example, spectrogram image of the forged speech obtained from the original
speech (ARANORM 0006. wav) taken from the Arabic Speech Corpus database is given Fig. 2 (b) and
Fig. 2(a) shows the original speech spectrogram. The red arrows on the spectrogram indicate the start
and end of the copy-pasted segments.

In the last few years, some researchers have shown an interest in the detection of audio copy-move
forgery and developed some methods to discriminate forged speech from original speech. The proposed
methods use a general approach to solving the forgery detection problem. First, the suspicious speech is
divided into voiced and unvoiced segments by using voice activity detection algorithms. Afterwards,
features are extracted from these segments with a certain method that is robust to post-processing
operations. By performing a similarity analysis between the extracted features, the segments on the
speech that are forged with copy-paste are detected. Nevertheless, most of the methods to detect audio
copy-move forgery were relying on hand-crafted features extracted from speech only.

In this paper, we present a novel ACMFD methodology based on deep neural learning using the CNN with
data augmentation to detect audio copy-move forgery in a fast and robust manner. The proposed method
generates Mel spectrogram color image from input audio signal as a feature. Afterwards, these
spectrogram images are classified into two class: forged and original with the developed CNN
classification system. The development CNN system extracts spectrogram features and generates
feature maps. The system finds the feature correspondences and dependencies using the average of the
generated feature maps. After the developed CNN system is trained, suspicious audio files given as input
are classified and labeled as fake or original. In addition, since there is no general database in the
literature, in this study, an audio copy-move database was created using TIMIT and Arabic Speech Corpus
databases to test the performance of the proposed methodology.

The major contributions of this article include:

1. This method uses Mel spectrogram features. The above methodologies are all to use speech directly
to detect audio copy-move forgery. Our approach extracts feature from the speech in the form of
spectrogram images.

Page 3/24
 2. In the literature, there are only a few hand-crafted methods conducted for the detection of audio
copy-move forgery. To the best of our knowledge, none of the existing works on audio copy-move
forgery detection has proposed deep feature learning from speech recording. This is the first study to
employ deep learning with Mel spectrogram of audio for the detection of audio copy-move forgery.
3. A copy-moved forged audio database is also created from two different speech databases in this
study to evaluate the performance of the proposed method.
4. It was seen from the experimental results that the proposed method achieved superior accuracy rates
for original and post-processed forged audios than other detection methods[16][19][17][20][21].

The rest of the paper is arranged as follows: Section 2 presents related works. The proposed approach is
given in Section 3. Experiments and results are reported in Section 4. Finally, in Section 5, the study is
concluded.

2. Related Work
In the literature, many studies have been carried out in the field of detecting various audio forgeries such
as resampling, splicing, deletion, insertion, re-quantizing,, copy-move, re-compression and the field of
audio authentication has attracted great attention. Pan et al. [1] detected audio splicing forgery utilizing
extraordinary differences in the local noise levels. Chen et al. [2] used DWT and examined singularity
points of speech recording to detect audio forged operations in the time domain like insertion, deletion,
substitution, and splicing. Gupta et al. [3] suggested a content-based method to detect audio copy move
forgery. This method compares query and the test fingerprints and calculates number of matching
fingerprints. In the literature, studies in the field of splicing audio forgery detection are more than the
detection of other audio forgery types, because it is relatively easier to detect recordings of different
environments [4, 5], microphones classification [6, 7] and speakers identification [8–10].

Other studies are also available in the literature [11–14] Studies in the field of audio copy move forgery
are relatively few compared to other studies about audio forgeries. Xiao [15] split the speech into
segments according to the constant time. This method calculated the similarity between all segments in
a speech recording to detect duplicate segments. Huang’s method [16] extracted voiced parts of audio
utilizing a voice activity detection method. They obtained features from voiced parts with discrete Fourier
transform (DFT) and Mel-Frequency Cepstral Coefficients (MFCC) methods. After their method sorted all
voiced part according to the features, they compared one voice part with some adjacent parts instead of
all parts in the sorted list to reduce the time complexity of algorithm. Wang et al. [17] suggested a method
using singular value decomposition (SVD) transform to detect audio copy move forgery. They obtained
voiced parts from speech recording using a voice activity detection (VAD) method. Their method
extracted features from these voiced parts with Discrete cosine transform (DCT). Xie et al. [18] proposed
a method to detect audio copy-move forgery with a multi-feature decision. They obtain four features:
gammatone, Mel-frequency cepstral coefficients (MFCCs), pitch, and discrete Fourier transform (DFT)
coefficients. Their method fused detection result of these four features with the C4.5 decision tree. Imran
et al. [19] extracted features from the voiced parts after extracting the voiced parts with their proposed
Page 4/24
VAD algorithm. They utilized 1-D local binary patterns (LBP) to obtain features. Afterwards, their method
compared all LBP histograms to detect duplicated parts that have similar histograms. Yan [20] utilized a
pitch tracking method YAAPT [22] to obtain voiced parts and computed the similarities of pitch
sequences to detect audio copy move forgery. In another study by the same authors, [21] is utilized pitch
and formant sequences. They split the speech into voiced and unvoiced parts, then obtained the pitch
and formant sequences of voiced parts as features. Finally, it computed the similarities of each feature
set with Dynamic time warping (DTW) to detect duplicated parts.

3. Proposed Approach
The aim of this study is to build up an effective deep ACMFD (Audio Copy Move Forgery Detection)
method that can obtain high performance with noticeably low cost to detect audio copy move forgery.
Proposed algorithm which uses CNN model given in Fig. 4 consists of two stages: Creation of the forged
database and Detection of audio copy-move forgery.

CNN, which was developed based on multi-layer perceptron network structure, has been used effectively
in many fields in the literature and has become quite popular. A CNN model contains of several layers.
These layers are convolution, pooling, normalization, and fully connected layers. The structure of the CNN
model is given in Fig. 3.

The CNN model starts with the input layer that takes 2D images as input. The input layer is followed by a
certain number of Convolutional (CNV) layers. The convolution layer takes the input feature maps from
the local regions in the input image and creates the output feature maps by applying these regions to the
convolutional filter. 2D filter is applied to images to produce features in different layers in image
classification. After each CNV layer comes a pooling layer to generate the outputs. Pool layer implements
down-sampling by calculating the maximum or average value of each local region int he image. As a
result of this pooling operation, size of the feature map is reduced which which provides to a efficient
calculation performance. At the final stage, Fully Connected (FC) layer combines the outputs of
convolution layer with different weights. Thus, it implements a linear transformation on the input feature
vector. Soft-max algorithm that computes the probability for each possible class is used to classify the
output of FC layer.

Proposed method consists of two stages as stated before. While one of them is used to create forged
database, the other one detects copied and pasted parts on the audio file. In the first stage, speech taken
from the Arabic speech corpus database are segmented into voiced and unvoiced parts with pitch
extraction method. Afterwards, a forged speech is created by choosing one of the speech segments at
random and pasting it on another randomly chosen segment. In the copy-move forgery detection stage of
the proposed method, whether the speech is forged or not is determined by using proposed CNN
architecture. Spectrogram is used by the method to transform the audio file into an image with its spatial
frequency representation. Specially designed CNN architecture gets the spectrogram image as input and
decides about the originality of the audio file. Details of these stages are given below.

Page 5/24
3.1. Creation of the Forged Database
In this study, an audio forgery database is created to measure the performance of our study and to
compare the study with another methods about audio forgeries in the literature. The proposed audio
forgery database creation operation is given in Fig. 5.

As can be seen in the figure, speeches that are taken from the Arabic Speech Corpus [25] and TIMIT [23]
database are segmented into voiced and unvoiced parts at first. Details of original speech recordings in
the Arabic Speech Corpus and TIMIT database will be presented in section 4. Proposed method extracts a
pitch sequence from the audio for segmentation of voiced and unvoiced parts. Pitch is a measure that
refers to the fundamental frequency and represents the vibration frequency of the vocal utterance. Even if
the same word is said twice in a speech, the pitch sequences of these repeated words will not be the
same [21]. Proposed approach utilized from YAAPT algorithm for segmentation of the audio file and the
details of it are given below.

YAAPT is utilized pitch tracking algorithm. Primary stages of this algorithm are given as follows:

Preprocessing

Multiple versions of the signal are generated utilizing nonlinear processing. Non-linear processing also
can be utilized to partially restore a missing pitch.

Pitch track prediction utilizing spectral knowledge

An approximate pitch track is predicted by utilizing a spectral harmonic correlation SHC method. SHC is
described by.

WL / 2 N +1
SHC(t, f) = ∑ f ' = − WL / 2 ∏ r =H1 S(t, rf + f ' ) (1)

where S(t, f) is the magnitude spectrum for frame t at frequency f, N H is the number of harmonics, WL
shows spectral window length SHC(t, f) is amplitude normalized to [0,1]. The normalized low frequency
energy ratio NLFER is also calculated as a help for pitch track prediction. NLFER is utilized to separate
voiced frames and unvoiced frames. NLFER is described by;

F
∑ f =0_2max
xF S( t,f) (2)
0_min
NLFER(t) = 1 T F
T ∑ t = 1 ∑ f =0_2max
xF S( t,f)
0_min

where. T is the total number of frames, and the frequency range, based on F 0_ min and F 0_ max. S(t, f) is
the spectrum of the signal for frame t and frequency f.

Pitch candidate prediction

Page 6/24
Nominees are created from both the original and nonlinearly processed signals in the time domain
utilizing normalized cross correlation function (NCCF) [24]. NCCF is described by; given a frame of
sampled speech

1 N − K _ max (3)
NCCF(k) = e 0e k
∑n = 0 s(n)s(n + k),
√

where,

n = N − K _ max 2
e0 = ∑ n = 0 s (n), ek = ∑ n = k
n = k + N − K _ max 2
s (n) (4)

K_min ≤ k ≤ K_max (5)

In the equation, N is the frame length in samples. K_min and K_max are the lag values. The NCCF is
self-normalized for a range of [− 1, 1].

Final pitch determination by Dynamic Programming

Dynamic programming is performed with the information from stages 2 and 3 to create outcome pitch.
[22].

After extracting pitch sequence from the speech with the YAAPT method, places with frequency values
greater than zero in the pitch sequence are marked as the voiced part and the places with zero frequency
are marked as the unvoiced part in the speech. Figure 6(a) and 6(b) shows the waveform of audio shown
in Fig. 2(a) and the pitch sequence respectively. The dashed lines show the voiced parts boundaries in the
audio.

One of the voiced segments in the speech is copied and pasted in place of another voiced segment to
create a forged speech. For this, as can be seen from the Fig. 5. Two numbers (i, j) are randomly selected
between 1 and N. N represents total number of vocal segments in the speech. The first of these numbers,
S irepresents the voiced segment to be copied, and the second one denoted by S j corresponds to the
voiced segment to which the copied segment will be pasted. The S j is deleted before the S i is copied and
pasted. Applying this process for all original speech recordings creates forged speeches and finally
forged database are created by using forged speeches. Forged speech generated from audio given in
Fig. 6(a) is given in Fig. 7(a) and the pitch sequence of this forged speech is given in Fig. 7(b). The
dashed lines show the duplicated regions in the forged audio.

3.2. Detection of Audio Copy-Move Forgery

Proposed algorithm uses the CNN model given in Fig. 4 to detect audio copy-move forgery. It is a non-
block-based algorithm, which works on the spectrogram image. Extracting features and classifying
various speech records through short audio clips are not easy. Many speech recordings have background
Page 7/24
noises, very short intervals, and fast changes in the recordings. Those noises and attributes make it very
difficult for the CNN model to classify them. The proposed method works on the spectrogram image
instead of the audio. For these reasons, It is a very rare approach to convert audio files into spectrogram
images for the detection of audio copy-move forgery. The spectrogram, which is the visual representation
of audio, has two dimensions representing frequency and time on the vertical and horizontal axes,
respectively. Amplitude of the signal at a given frequency at a specific time is represented by the color
intensity in the spectrogram. For example, while the light blue in the spectrogram shows the lowest
amplitude, the highest amplitude is displayed by dark red.

In the study, the Short-Time Fast Fourier transform (STFT) was utilized to create the spectrogram of the
speech signal. Firstly, the audio signal is first splited into 30 ms frames and each frame is multiplied by
the Hamming window. Afterwards, FFT is applied to all subframes according to (1).

N−1
S(f, t) = ∑ n = 0 w nx t(n)exp( − j2π(f / f s)n) (6)

f = kf s / N

where w n is the Hamming window function, x n is the original speech signal, f is the frequency range for
k = 1, 2, … N/2 + 1 and N refers to the number of samples in the frame.

There are many different methods used in spectogram representation such as amplitude spectogram,
power spectogram, phase spectogram, and log amplitude spectogram. The frequencies for all these
representations are represented in equal intervals. However, some methods also use the features of the
human sensory system and are more sensitive to low frequency components while less sensitive to high
frequency components. Mel spectogram is one of the methods based on the human sensory system and
used in the time-frequency representation of sound. Mel spectogram is STFT filtered with Mel frequency
filter bank. Mel spectogram coefficientsS mel(k, t) are calculated according to (2).

(7)
L−1
|
S mel(k, t) = ∑ l = 0 m k(l) S(l, t) 2 |
Where L is the frequency component number and m k(l) refers the k filter of the mel filter bank.

Figure 8 shows mel spectograms created by this method for forged and original speech signals.

When looking at the Mel spectrogram images in the figure, the duplicated columns on the spectrogram
image do not change even if post-processing operations are applied to the images. This shows that Mel
spectrogram images are an effective method for feature extraction from speech.

After creating the forged and original spectograms obtained from the speech signals received from the
input, a training and test set was created from these spectograms for the proposed CNN architecture.
However, due to the large number of parameter adjustments in CNN, a large amount of data is needed
Page 8/24
during the training stage. If insufficient amount of data is used in training, there will be inconsistency
between training accuracy and test accuracywhich results in overfitting problem. For this reason, data
augmentation technique was also used during creation of training set in the proposed method. With the
data augmentation technique, which has become a very common approach in deep learning systems,
more training data is generated and the overfitting problem in the training phase can be prevented.

In this study, frequently used image data augmentation techniques were used on spectrogram images
using Kera's package (Referans). Augmentation operations applied to the spectrogram images are Width
shift: 0.5, Height shift: 0.5, Fill mode: nearest, Shear range: 0.30. By applying these augmentation
operations to spectogram images, desired number of images can be produced. In Fig. 9, block diagram of
the data augmentation method applied to the spectogram images in the study is shown.

As can be seen in Fig. 9, after the number of spectrogram images is increased by data augmentation
operations, spectrogram images are given to the proposed CNN architecture. For this, firstly, spectrogram
images are divided into two classes as fake and original, and a training set is created. As a result of the
CNN training, the label of the suspicious test speech is predicted. Training process stops after 140
epochs, the Adam optimizer and the categorical cross-entropy loss function were preferred, and the batch
size, for the dataset was selected 64.

The description of layers with filters, max-pooling function, dropout, and activation function is given in
below.

Convolutional neural network architecture

Layer1

The first layer contains of 32 kernels with a 3*3 respective field. Stride value for this layer is 2*2. The
Rectified Linear Unit (ReLU) is used as an activation function.

Layer2

The second layer includes again 32 kernels with a 3*3 respective field and 2*2 stride value. It is followed
by 2*2 strided max-pooling function. In order to avoid overfitting 0.5 dropout used in this layer. The
activation function utilized is the Rectified Linear Unit (ReLU).

Layer3

The third layer comprises of 64 kernels with 3*3 respective field and 2*2 stride value. The max-pooling
function is not involved in this layer. The Rectified Linear Unit (ReLU) is utilized as the activation function.

Layer4

The fourth layer comprises of 64 kernels with 3*3 respective field and 2*2 stride value. The max-pooling
function is not involved in this layer. In order to prevent overfitting 0.5 dropout is selected. The Rectified
Page 9/24
Linear Unit (ReLU) is utilized as the activation function

Layer5

The fifth layer flatten the output of the convolutional layers to generate a single long feature vector and it
is connected to the fully-connected layer.

Layer6

The sixth layer is the dense layer that contains of 64 hidden units. Rectified Linear Unit (ReLU) is utilized
as an activation function. In order to prevent overfitting 0.5 dropout rate selected in this layer.

Layer7

The last layer is the output layer. It contains of the output units, equal in numbers to the total number of
classes (forged - original) in the dataset. Softmax is utilized as the activation function.

4. Experimental Results
This part presents an involved analyze of the results obtained with our method. Moreover, the proposed
method was compared with other studies in this field in the literature to show the efficiency of the
proposed method. The experiments have been performed using Python 3.5 and Keras with TensorFlow
backend toolkits on a machine with Intel Core i5, 64 bits processor, 8 GB RAM, operating by Windows 10.

4.1 Dataset
In this study, the Arabic Speech Corpus and TIMIT database is utilized to produce the audio copy-move
forgery databases. The TIMIT database contains of English-language speeches with sampling rate of
8kHz about from two seconds to six seconds. The Arabic Speech Corpus with Modern Standard Arabic
(MSA) speech corpus for speech synthesis is compound of phonetic and orthographic transcriptions of
more than 3.7 hours of MSA speech. This database contains1813 .wav files. While creating the forged
audio file, a random segment was taken in the speech and this segment was pasted on the segment at a
random position in the same speech. The duration of each repeated segment thus formed is
approximately between 0.2 and 0.6 seconds. To show the efficiency of our method; 368 forged audio files
from the TIMIT database and 1329 forged audio files from the Arabic Speech Corpus are generated. As a
result, forged audio files are produced. (created database is available at
[Link]

4.2 Evaluation Metrics

The performance of the proposed method has been evaluated with five different metrics: accuracy,
precision, recall, F1-score, and area under the ROC curve. At the same time, a confusion matrix was
utilized to visualize the classification performance of the method. A confusion matrix is a table that
allows visualization of the performance of an algorithm, shows the predicted and actual classification.
Page 10/24
The size of the confusion matrix is nxn where n is the number of different classes. For n × n matrices,
True Positive (TP), True Negative (TN), False Negative (FN), and False Positive (FP) are computed. In the
study, speech recordings are labeled as forged or original with the proposed CNN architecture. In the 2-
class classification; TP shows positive samples correctly identified as positive samples. FP shows
negative samples incorrectly identified as positive samples. TN represents negative samples correctly
identified as negative samples and FN shows positive samples incorrectly identified as negative samples.
Accuracy, precision, recall and F1-score are computed according to (8), (9), (10) and (11).

TP + TN (8)
Accuracy = TP + TN + FP + FN

TP (9)
Precision = TP + FP

TP (10)
Recall = TP + FN

Precision (11)
F1 − score = 2 Precision + Recall

where, Accuracy indicates the number of correct predictions made, Precision and Recall indicate how well
the model predicts positive values and true positives, respectively.

AUC-ROC curve is utilized to evaluate the algorithm’s classification performance. ROC is the probability
curve, and AUC shows the degree of separability. It indicates how well the model can distinguish between
classes. In general, the AUC shows how well the model estimates forged and original classes correctly.
For example, the greater the AUC is, the more accurate the model differentiates between forged and
original audio recordings.
4.3 Data Augmentation
Training the CNN with an insufficient dataset causes overfitting. As a result of overfitting, the algorithm
cannot extract features during the training process. For this reason, data augmentation techniques were
used in the proposed study. In addition to increasing the training set by using data augmentation
techniques, the evaluation process of the proposed algorithm has been generalized. In this work, we
applied to the spectrogram images are Width shift: 0.5, Height shift: 0.5, Fill mode: nearest, Shear range:
0.30.

The training dataset before data augmentation has 2898 spectrogram images: 930 forged and 1968
original taken from the Arabic Speech Corpus dataset. The sizes of the images 434x343. After the data
augmentation, the size of the training set was increased by 4 times and a total of 11592 spectrogram
images were obtained.

The test dataset has 1243 spectrogram images: 399 forged and 844 originals taken from the Arabic
Speech Corpus dataset.

4.4 Results with data augmentation

Page 11/24
In this part is illustrated obtained evaluation results for the proposed deep ACMFD algorithm. The
proposed algorithm has been applied on the Arabic Speech Corpus database. Several experiments have
been carried out using data augmentation to increase the testing accuracy.

In the proposed method, firstly, the k-fold cross-validation method was applied to the training set without
data augmentation. This technique was used to fully test the dataset and make an effective evaluation.
When applying k-fold cross validation to the dataset, the dataset is randomly divided into k groups of
approximately equal size. Afterwards, the proposed model is trained each time with the (k-1) group and
tested with the remaining group as well. This process is repeated k times. We firstly investigated the
effect of data augmentation. For this purpose, the proposed algorithm was trained for 10-fold cross
validation before applying data augmentation. Training accuracy and validation accuracy graphics for
10-fold cross validation without and with data augmentation are given in Fig. 10(a) and 10(b)
respectively. Training loss and validation loss graphics for 10-fold cross validation without and with data
augmentation are given in Fig. 10(c) and 10(d) respectively.

As can be seen from the graphs, there are very high fluctuations in both accuracy and error curves in
some epochs in 10-fold cross validation without data augmentation. So the curves are not stable.
However, as a result of data augmentation with 10 folds, the curves are stable in both accuracy and loss.
Accuracy curves approach 1, while loss curves approach 0. Accuracy, precision, recall, F1-score and AUC
results obtained from the test data as a result of the training also support this. The obtained results are
shown in Table 1. As can be seen the table, the proposed method with data augmentation approach for
Mel spectrogram-based images classification gives the best performance according to accuracy,
prediction, recall, F1-score and AUC.

Table 1
The result of proposed CNN system terms of accuracy, precision, recall, F1 score and
AUC with/without data augmentation
10-fold Accuracy Precision Recall F1 score AUC

without data augmentation 0,95 0,91 0,94 0,98 0,96

with data augmentation 0,99 0,96 0,97 0,99 0,98

4.5 Results with post-processing operations

After the forged speech is created, various post-processing operations such as noise adding, compression
and filtering are applied to the forged audio to remove traces of forgery. For this purpose, post-processing
operations were applied to the test dataset to present the robustness of the proposed method to post-
processing operations. We applied to the forged audios from the Arabic Speech Corpus and TIMIT
databases some commonly used post-processing operations to create the audio copy-move forgery
databases with post-processing. These post-processing operations are adding noise (30 dB and 20 dB
Page 12/24
white Gaussian noise), filtering (median filter), and Mp3 compression (32 kbps and 64 kbps). As a result,
our copy-move forgery dataset contains a total of 1960 forged speeches with post-processing operations.
Table 2 presents the result of the proposed method in terms of Accuracy, Precision, Recall and F1-score
on forged database generated from Arabic Speech Corpus. (Prepared database is available at
[Link]

Table 2
The result of proposed method with post-processing terms of accuracy,
precision, recall, F1 score and AUC on forged database generated from
Arabic Speech Corpus.
Accuracy Precision Recall F1-score

Median filtering 0,91 0,87 0,93 0,9

64kbps compressing 0,9 0,88 0,93 0,91

20db noise adding 0,94 0,88 1 0,94

30db noise adding 0,93 0,87 1 0,93

As can be seen from the Table 2, the proposed method is highly robust to noise addition, compression,
and median filtering post-processing operations. The F1-score and accuracy values of proposed method
is obtained 0.94 because of the noise post-processing operation. This shows that the proposed method is
more resistant to noise attack than other attacks. We also experiment by training and testing on different
datasets. Our cross-dataset experiments are conducted with our copy-move forgery datasets generated
from Arabic Speech Corpus and TIMIT datasets.

We train our proposed approach on Arabic Speech Corpus and test on TIMIT. The results in terms of
Accuracy, Precision, Recall and F1-score of the proposed method on forged database generated from
TIMIT are provided in Fig. 11.

The experimental results presented in Fig. 11 shows the robustness of our method for unseen forged
audios in the training phase. At the same time, looking at the figure, the metric results obtained for all
post-processing operations are 0.99 and above. This shows that the method is also very robust to post-
processing operations.

The results of the proposed method on all databases (forged databases created from Arabic Speech
Corpus and TIMIT) are given in Table 3.

Page 13/24
 Table 3
The result of proposed method with post-processing terms of accuracy,
precision, recall, F1 score and AUC on all forged database generated
from Arabic Speech Corpus and TIMIT
Accuracy Precision Recall F1-score

No additional attack 0,98 0,97 0,98 0,99

Median filtering 0,98 0.96 0,99 0,98

64kbps compressing 0,98 0,96 0,99 0,98

32kbps compressing 0,98 0,95 0,99 0,97

20db noise adding 0,99 0,96 1 0,98

30db noise adding 0,99 0,96 0,99 0,97

When the results in the Table 3 are examined, the accuracy and recall values of the proposed method are
above 0.98, the Precision value is above 0.95, and the F1-score value is above 0.97.
4.5 Comparison with Traditional Results
In this part, we compared the proposed method with other studies in this field in the literature to present
the efficiency of our method. The results obtained as a result of the proposed method were compared
with the results of Lbp method [19], Pitch similarity method[20], Formant method[21], DFT method[16]
and DCT-SVD [17]. The proposed method and other studies were coded on the same machine and tested
in the audio database created by us. Table 4 summarizes the results of the comparison with post-
processing operations on database generated from TIMIT. When the results were examined, the accuracy
values of the proposed algorithm in all post-processing operations were 0.99 and above.

Page 14/24
 Table 4
Comparison of the proposed approach with other existing published studies on on database generated
from TIMIT
Lbp[19] Dft[16] Formant DCT-SVD Pitch-sim Proposed
[21] [17] [20]

No-additional 0,18 0,19 0,23 0,22 0,37 0,99

attack

Median filtering 0,1 0,21 0,28 0,22 0,36 1

20db noise adding 0,17 0,1 0,37 0,25 0,33 1

30db noise adding 0,19 0,16 0,31 0,29 0,35 1

32kbps 0,2 0,2 0,31 0,29 0,31 0,99

compressing

64kbps 0,2 0,19 0,3 0,3 0,34 0,99

compressing

Table 5 shows the results of the comparison with post-processing operations on database generated
from Arabic Speech Corpus. As can be seen from the table, the accuracy values of the studies in the
literature cannot exceed 0.63, while the accuracy values obtained from the proposed method are 0.90 and
above.

Table 5
Comparison of the proposed approach with other existing published studies on database generated from
Arabic Speech Corpus
Lbp[19] Dft[16] Formant DCT-SVD Pitch-sim Proposed
[21] [17] [20]

20db noise adding 0,5 0.2 0,63 0,6 0,5 0,94

30db noise adding 0,4 0,16 0,6 0,4 0,37 0,93

64kbps 0,3 0,2 0,57 0,3 0,3 0,9

compressing

Median filtering 0,1 0,2 0,57 0,4 0,3 0,91

The results show that the proposed ACMFD algorithm with deep learning on database generated from
TIMIT and Arabic Speech Corpus shows the highest performance in comparison with those of the other
ACMFD algorithms.

5. Conclusion
In this paper, we present a new ACMFD method with deep neural networks. The motivation of this method
is to buid up a CNN architecture to categorize the suspicious Mel spectrogram images into two classes:
Page 15/24
original and forged. The proposed CNN model is successfully trained on these Mel spectrogram images
feature extraction for audio copy-move forgery detection. The method also utilized data augmentation for
the proposed architecture. We tested the proposed model with our datasets generated from Arabic
Speech Corpus and TIMIT speech database. Several experiments have been conducted using data
augmentation, cross dataset, and post-processing operations. The obtained results showed that the best
performance was obtained using data augmentation. For datasets with post-processing operations
generated from Arabic Speech Corpus and TIMIT database, the best accuracies are 0,99 and 0,98
respectively with no additional attack. When the results of other studies in the literature are examined, it
will be seen that the proposed method gave the best and highest results in the our datasets.

Declarations
Funding

This study was supported by the Scientific and Technological Research Council of Turkey (TUBITAK) with
Project No: 122E013.

Ethics approval and consent to participate

The authors declare that this article is original, has not been published before, and is not currently being
considered for publication elsewhere. The authors confirm that the manuscript has been read and
approved by all named authors and that there are no other persons who satisfied the criteria for
authorship but are not listed. The authors further confirm that the order of authors listed in the
manuscript has been approved by all of them.

Competing interests

The authors declare the following financial interests/personal relationships which may be considered as
potential competing interests: This work was supported by the Scientific and Technological Research
Council of Turkey (TUBITAK) with Project No: 122E013

Authors' contributions

Arda USTUBIOGLU: Conceptualization, Methodology, Software Beste USTUBIOGLU.: Data curation,

Writing- Original draft preparation, Software. Guzin ULUTAS: Visualization, Investigation, Methodology.

Funding

This work was supported by the Scientific and Technological Research Council of Turkey (TUBITAK) with
Project No: 122E013

Availability of data and materials

Not applicable
Page 16/24
 Consent for publication

Not applicable

Acknowledgements

Not applicable

References
1. X Pan X Zhang S Lyu (2012) Detecting splicing in digital audios using local noise level estimation.
IEEE International Conference on acoustics speech and signal processing 1841–1844.
[Link] 10.1109/ICASSP.2012.6288260
2. J Chen S Xiang H Huang W Liu (2016) Detecting and locating digital audio forgeries based on
singularity analysis with wavelet packet. Multimed Tools Appl 75:2303–2325.
[Link]
3. V Gupta G Boulianne P Cardinal (2010) Content-based audio copy detection using nearest-neighbor
mapping. IEEE International Conference on Acoustics Speech and Signal Processing 261–264.
[Link] 10.1109/ICASSP.2010.5495963
4. G Muhammad Y A Alotaibi M Alsulaiman M N Huda (2010) Environment recognition using selected
MPEG-7 audio features and mel-frequency cepstral coefficients. 5th Int Conf Digit Telecommun 11–
16. [Link] 10.1109/ICDT.2010.10.
5. H Zhao Y Chen R Wang H Malik (2017) Audio splicing detection and localization using
environmental signature. Multimed Tools Appl 76:13897–13927. [Link]
016-3758-7
6. L Cuccovillo S Mann M Tagliasacchi P Aichroth (2013) Audio tampering detection via microphone
classification. IEEE Int Work Multimed Signal Process. 1:177–182.
[Link]
7. Q Liu A H Sung M Qiao (2010) Detection of double MP3 compression. Cognit Comput 2:291–296.
[Link]
8. D Luo R Yang B Li J Huang (2017) Detection of double compressed AMR audio using stacked
autoencoder. IEEE Trans Inf Forensics Secur 12:432–444. [Link]
10.1109/TIFS.2016.2622012.
9. X Lin X Kang (2017) Supervised audio tampering detection using an autoregressive model. IEEE
International Conference on Acoustics Speech and Signal Processing 2142–2146. [Link]
10.1109/ICASSP.2017.7952535
10. Y Zhang S Dai W Song L Zhang D Li (2020) Exposing speech resampling manipulation by local
texture analysis on spectrogram images. Electronics 9:1–16.
[Link]

Page 17/24
11. Q Liu A H Sung M Qiao (2010) Detection of double MP3 compression. Cognit Comput 2:291–296.
[Link]
12. D Luo R Yang B Li J Huang (2017) Detection of double compressed AMR audio using stacked
autoencoder. IEEE Trans Inf Forensics Secur 12:432–444. [Link]
10.1109/TIFS.2016.2622012.
13. X Lin X Kang (2017) Supervised audio tampering detection using an autoregressive model. IEEE
International Conference on Acoustics Speech and Signal Processing 2142–2146. [Link]
10.1109/ICASSP.2017.7952535
14. Y Zhang S Dai W Song L Zhang D Li (2020) Exposing speech resampling manipulation by local
texture analysis on spectrogram images. Electronics 9:1–16.
[Link]
15. X Ji-nian J I A Yun-zhe F U Er-dong H Zheng et al (2014) Audio Authenticity : Duplicated audio
segment detection in waveform audiofile. Journal of Shanghai Jiaotong University (Science) 4:392–
397. [Link] 10.1007/s12204-014-1515-5.
16. X Huang Z Liu W Lu et al Fast and effective copy-move detection of digital audio based on auto
segment (2019) Digital forensics and forensic Investigations: Breakthroughs in Research and
Practice 11:47–62. [Link]
17. F Wang C Li L Tian (2017) An algorithm of detecting audio copy-move forgery based on DCT and
SVD. IEEE 17th International Conference on Communication Technology 1652–1657. [Link]
10.1109/ICCT.2017.8359911
18. Z Xie W Lu X Liu et al (2018) Copy-move detection of digital audio based on multi-feature decision. J
Inf Secur Appl 43:37–46. [Link]
19. M Imran Z Ali S T Bakhsh et al (2017) Blind detection of copy-move forgery in digital audio forensics.
IEEE Access 5:12843–12855. [Link] 10.1109/ACCESS.2017.2717842.
20. Q Yan R Yang J Huang (2015) Copy-move detection of audio recording with pitch similarity. IEEE Int
Conf Acoust Speech Signal Proces 61202497:1782–[Link]://[Link]/
10.1109/ICASSP.2015.7178277.
21. Q Yan S Member R Yang J Huang (2019) Robust copy – move detection of speech recording using
similarities of pitch and formant. IEEE Trans. Inf. Forensics Secur 14:2331–2341.
[Link]
22. S A Zahorian H Hu (2008) A spectral / temporal method for robust fundamental frequency. The
Journal of the Acoustical Society of America 6:4559–4572. [Link]
23. J Garofolo L Lamel W Fisher et al (1993) Darpa TIMIT acoustic-phonetic continous speech corpus
CD-ROM. NIST Speech Disc 1-1.1 93: 27403. [Link]
24. D Talkin WB Kleijn (1995) A robust algorithm for pitch tracking (RAPT). Speech Coding and
Synthesis 495: 518. [Link]
25. [Link]

Page 18/24
Figures

Figure 1

(a) Original audio (b) forged audio with copy-move forgery

Page 19/24
Figure 2

(a) Original (b) forged spectrogram image

Figure 3

The schematic diagram of the CNN structure

Page 20/24
Figure 4

The structure of the proposed algorithm

Figure 5

An overview of creating databases

Page 21/24
Figure 6

(a) the waveform (b) pitch sequence of original speech

Figure 7

(a) the waveform (b) the pitch sequence of forged speech

Figure 8

An example of Mel spectrogram image

(a)of original audio (b) of forged audio with 20 dB white Gaussian noise addition attack (c)of forged
audio with 32-bit compressing attack (d) forged audio under median filtering attack

Page 22/24
Figure 9

Block diagram of the data augmentation method

Page 23/24
Figure 10

Training and validation accuracy (a)without data augmentation(b) with data augmentation, Training and
validation loss (c)without data augmentation(d) with data augmentation

Figure 11

The testing result of proposed method on unseen data(generated from TIMIT database)

Page 24/24