A Meta-Learning Approach for Few-Shot Face Forgery

Segmentation and Classification

Govind Raut
Department of Science and Technology, Department of Science and Technology,
G H Raisoni Institute of Engineering and G H Raisoni Institute of Engineering and
technology, Nagpur, India technology, Nagpur, India
[Link]@[Link] harshita3902@[Link]

Chandrakant Kottalwar
Department of Science and Technology, [Link] Chaube
G H Raisoni Institute of Engineering and Department of Science and Technology,
technology, Nagpur, India G H Raisoni Institute of Engineering and
[Link]@[Link] technology, Nagpur, India
[Link]@[Link]

Abstract: The technology for detecting forged images is good at detecting

known forgery methods. It trains neural networks using many original and
corresponding forged images created with known methods. However,
when encountering unseen forgery methods, the technology performs
poorly. Recently, one suggested approach to tackle this problem is to use a
hand-crafted generator of forged images to

create a range of fake images, which can then be used to train the neural
network. However, the aforementioned method has limited detection
performance when encountering unseen forging techniques that the hand-
craft generator has not accounted for. To overcome the limitations of
existing methods, in this paper, we adopt a meta-learning approach to
develop a highly adaptive detector for identifying new forging techniques.
The proposed method trains a forged image detector using meta-learning
techniques, making it possible to fine-tune the detector with only a few
new forged samples. The proposed method inputs a small number of the
forged images to the detector and enables the detector to adjust its weights
based on the statistical features of the input forged images, allowing the
detection of forged images with similar characteristics. The proposed
method achieves significant improvement in detecting forgery methods,
with IoU improvements ranging from 35.4% to 127.2% and AUC
improvements ranging from 2.0% to 48.9%, depending on the forgery
method. These results show that the proposed method significantly
improves detection performance with only a small number of samples and
demonstrates better performance compared to current state-of-the-art
methods in most scenarios.

1. Introduction
In recent years, the issue of forged facial images as a security
vulnerability issue has benefited greatly from the development of deep
learning. To perform forensic analysis on these types of forged images,
deep learning has also been utilized for the purpose of detection. The
technology for detecting forged images currently performs well in
detecting known methods of forging. This technology uses a large number
of original and corresponding forged images created using known forged
methods to train the neural networks as the detectors for learning the
characteristics of forged images. However, these methods have a
significant drop in detection performance when encountering forged
methods that have not been trained.
To address this issue, a novel method has recently been proposed [1].
This method utilizes a parameterizable forged image generator to create
diverse forged images, and then uses these forged images to train a neural
network. In Figure 1, the pristine image is input to the forged image
generator during the training stage. The forged image
generator/synthesizer G has various mechanisms, such as forging method
A and forging method B, for creating forged images. These synthetic
forged images are then used as training images to train the forged image
detector. Since the forged image generator is designed by
 pristine images

f training

test set f inference

Figure 1. Previous methods of fake sample syntheses. The pristine images

are used to generate fake images based on color jitter, resizing, sharpening,
and translation using a fake image synthesizer G. The orange dashed line
represents the use of trained weights during the inference stage.
To address the issue with the methods mentioned earlier, we propose a
meta-learning approach to train a detector that excels at identifying new
forging techniques. Our method for detecting forged images involves
utilizing meta-learning techniques to train a forged image detector that is
easy to train with a limited number of new, forged samples. The aim of
training the forged image detector is to enable it to take in a small number
of new, forged image samples and adjust their weights to identify forged
images with comparable statistical features to the provided small set of
forged image samples
2. Related Work
In the literature, there are various methods for creating fake faces. In this
paper, we will specifically introduce a few methods that are relevant to our
research. One such method is the NeuralTextures method, as proposed by
Thies et al. . This method improves the quality of a computer-generated
texture by using a special algorithm in combination with a rendering
network to create a realistic reenactment result. Another method, known as
the Face2Face facial reenactment system , maps 2D points of faces to 3D
models from source video streams and blends the altered faces from the
3D models with different facial features. A third method, called FaceSwap,
is a computer graphics-based approach that extracts facial landmarks of
source faces and maps these landmarks onto a 3D template model for
creating altered facial features. An additional method for creating fake
faces is the deep-learning-based DeepFakes technique. This method
involves the extraction of faces from original images, followed by the use
of a trained encoder and decoder for the source faces to generate the target
fake faces. This technique has been shown to produce highly realistic fake
faces and has received significant attention in recent years due to its
potential for nefarious usage in the generation of convincing fake videos.
Recent advances in deep learning have been utilized to develop automated
methods for detecting fake faces. In the literature, various CNN-based
approaches for forgery detection have been proposed . For instance,
Rössler et al. proposed the use of a CNN-based model, namely
XceptionNet, to address the forgery detection task as a binary
classification problem.
Nguyen et al. proposed a novel approach to the detection of fake faces.
They view it as more than just a classification problem, but also as a
segmentation problem. In this approach, the focus is on identifying and
marking the fake regions of a given image through the use of auto-
encoders and a specialized Y-shaped decoder. These techniques utilize
information sharing between classification and segmentation for the
purpose of detecting and segmenting manipulated facial images. Instead of
using the Y-shaped decoder to produce segmentation and classification,
the proposed method uses a U-Net model to produce the segmentation
result and accords the segmentation result to determine whether the input
image is fake.

3. The Proposed Scheme

3.1. Architecture of the Model
fori ← 1 to N do
Sample k images and their ground truth from support set
S = {(x1, y1), (x2, y2), . . . , (xk, yk)}
Sample q images and their ground truth from query set
Q ,...,
θi ← φ. set initialization weight for each task
while not done do . gradient descent for optimizing θ
Evaluate ∇θi L(fθi (xj), yj) for 1 ≤ j ≤ k 9 Update θi ← θi − ζ∇θi L(fθi (xj)) for
1≤j≤k
for 1 ≤ j ≤ q . count loss using query set
Update i end

To produce the segmentation of predicted manipulated regions, we use the

U-Net architecture, a variant of a fully convolutional network, to accept an
input image and predict the possibility of the fakeness of every pixel in the
input image. The U-Net consists of a sequence of convolutional blocks
and transposed convolutional blocks. Figure 3 shows the detailed
architecture of U-Net used in the proposed method. The input image x,
which contains only RGB channels, is sent to the U-Net, and the predicted
mask mˆ, which only contains one channel, is produced. The input images
are first resized to 256 × 256, and the value of each pixel is normalized
with a mean of 0.5 and a standard deviation of 0.5 for each R, G, and B
channel. That is, the output of each channel equals the input of the channel
minus the mean of the channel divided by the standard deviation of the
channel. Afterward, the normalized 256 × 256 × 3 images are sent to be
the input of U-Net. The sequence of convolutional blocks of U-Net is used
to extract the feature of fakeness, and the concatenated sequence of the
transposed convolutional block is used to synthesize the predicted mask
mˆ.
For each altered image, the training set also contains its altered area. This
altered area, also called a mask, is used to indicate the pixels being
modified in the forging procedure. Thus, the mask of an altered image is
used as the ground truth of the forged area prediction problem (also called
the segmentation problem). Because the images in the dataset include not
only facial features but also large areas of background, we use the mask to
identify the location of the face in the image, and the nearby area of this
location has been cut to be the cropped face. If face detection is used to
locate the face in the image and to crop the face part of the image, there
will be a failure due to the face detection algorithm being unable to detect
the forged image. The cropped images are resized to 256 × 256 and
normalized to a mean of 0.5 and a standard deviation of 0.5. The
rectangular area centered on the face in the original image is cropped as
the input image of U-Net.
3.2. The Meta-Learning Approach

A fake segmentation task trains the model to predict altered pixels of input
images using a training set generated by a specific forged method. There
are N fake segmentation tasks used to train the model, which can easily be
adjusted to unseen forgery methods. See the for loop of lines 3–10 in the
Algorithm 1. The goal of the algorithm is to find the parameters φ that can
be trained using only a few samples to detect fake images of unseen
methods. When training on task i, the inner loop of the meta-learning in
lines 7–9 uses gradient descent to adjust the weights, θi, of the model with
one or a few iterations using the support set, S = {(x1, y1), (x2, y2), . . . , (xk,
yk)}, which is randomly sampled from the dataset of the task. After
completing the whole loop in lines 7-9, line 10 calculates the loss
on the query set Q , . . . , using the
current
weights θi. After lines 4–10 complete the training of N tasks, line 11 uses
the average of all the loss `i from the tasks to update parameters φ using
gradient descent. This entire process is repeated until φ satisfies the loss
requirements. Meta-training is completed up to this point; the process now
enters the few-shot learning stage.
In the few-shot learning stage, the optimized model obtained from the
previous stage is fine-tuned by K fake images from an unseen forgery
method U using standard gradient descent optimization. After training is
finished, we test the model using forged images produced from forgery
method U.
For simplicity, here, we define the loss function using the notation of two
onedimensional vectors yˆ = yˆ1, yˆ2, . . . , yˆn, and y = y1, y2, . . . , yn. This
definition is easy to generalize to higher-dimensional arrays.
1 n
L(yˆ, y) = · ∑(−yi × log(σ(yˆi)) − (1 − yi) ×
log(1 − σ(yˆi))) (1) n i=1

where yˆ is the prediction, and y is the ground truth, e.g., the output of U-
Net fφ(x) = yˆ.
The logistic sigmoid function σ(·) is defined by σ(x) = 1+1ex .
The function σ(·) is utilized in order to introduce nonlinearity into the
output of the neurons. This function serves the additional purpose of
constraining the output range of the neurons between the values of 1 and
0, thereby enabling the interpretation of the output as a probability of
confidence in predicting the fakeness of pixels. Furthermore, σ(·) is
differentiable, and this property is of great importance in the Algorithm 1
used to calculate the derivative of the function. This derivative is then
employed to update the weights of the U-Net fφ, with the objective of
minimizing the error between the predicted and true outputs.
The basic idea behind Formula (1) is to measure the difference between
the predicted probability distribution and the true probability distribution
of the segmentation (binary classification of pixels). Specifically, the loss
function is calculated as the negative loglikelihood of the true class given
the predicted probability distribution of segmentation. Intuitively, the loss
function measures how well the predicted distribution matches the true
distribution of segmentation. If the predicted distribution is very different
from the true distribution of segmentation, then the loss will be high,
indicating a large amount of uncertainty or disorder in the prediction. If
the predicted distribution of segmentation is very similar to the true
distribution of segmentation, then the loss will be low.

[Link] and Comparison

4.1. Experimental Design and Data Collection
In order to verify the performance of the proposed method, images altered
by the methods DeepFakes, Face2Face, FaceSwap, and NeuralTextures
obtained from the FaceForensics++ dataset were used for the experiments.
Our experiment involved mixing real images that had not been tampered
with and fake images that had been forged using four methods—
DeepFakes, Face2Face, FaceSwap, and NeuralTextures—and then feeding
the resulting images into the proposed detector. In each experiment, the
ratio of forged images to real images is fifty-fifty. This resulting mixed set
of images is input into the detector we proposed to detect the forged
regions and determine whether the image is fake. As this dataset
FaceForensics++ has also been used by other related studies to evaluate
the detection capability of their algorithms and their ability to detect
forged images, we can compare our results with theirs based on the use of
the same dataset.
We utilized the C23 version images from the FaceForensics++ dataset,
which was compressed using H264 with a constant rate quantization
parameter equal to 23. The purpose of using C23 images is to simulate
real-world situations where manipulated images might have their quality
reduced by compression or other factors. If a high compression ratio, such
as c40, is used, the image will become very blurry. Although it is difficult
to determine if the image is forged, such a blurry image cannot be used in
general situations. The FaceForensics++ dataset comprises 1000 pristine
videos that were obtained from YouTube by the dataset’s creators. The
FaceForensics++ dataset is produced from 1000 pristine videos.

Figure 1 shows randomly selected images from DeepFakes and their

predicted segmentation results with one-shot fine-tuning. For each row, the
top sub-row shows the results obtained using the proposed method, while
the bottom sub-row displays the results without its use. Within each sub-
row, the images altered by DeepFakes appear on the most right, followed
by the ground truth of the altered region (mask), the binary predicted
output, and the gray-scale predicted output, moving from right to left.

DeepFakes

Figure 1. Four rows of randomly selected images from DeepFakes and their predicted results for the fake
region with one-shot fine-tuning. The top sub-row of each row is the result generated by the proposed method,
and the bottom sub-row of each row is the result without using the proposed method. In each sub-
Figure 2. Comparison of AUC between random initial weights (without the proposed method) and meta-
learning of detecting images altered by Face2Face manipulation methods. The x-axis is the size of the fine-
tuned training set and the y-axis is the value of AUC.
Figure 3. Comparison of AUC between random initial weights (without the proposed method) and meta-learning of
detecting images altered by FaceSwap manipulation methods. The x-axis is the size of the fine-tuned training set and the y-
axis is the value of AUC.

Figure 4. Comparison of AUC between random initial weights (without the proposed method) and meta-
learning of detecting images altered by NeuralTextures manipulation methods. The x-axis is the size of the fine-
tuned training set and the y-axis is the value of AUC.

Because our pioneering research aims to detect forgery regions and

determine whether a given image is forged using small samples, the study
closest to ours in published research is [10]. While their study also detects
forgery regions and determines whether an input image is forged, they
only experimented with two training sets and provided experimental
results on the pixel-wise accuracy of forgery region detection. In Section
4.2, we explained why pixel-wise accuracy is not an appropriate metric for
evaluating the detection of forgery regions. However, this paper also
provides detailed results on detecting forgery regions using the pixel-wise
accuracy and IoU metrics, which can serve as a reference for comparing
the effectiveness of subsequent research efforts.
Table 2 compares the zero-shot result of different detection methods
between [10,38] and the proposed method. In Table 2, the first two
methods FT_Res and FT are proposed by Cozzolino et al. [38]. The next
four methods Deeper_FT, MT_old, No_Recon, and MT_New are
proposed by Nguyen et al. Based on Table 2, the proposed method has the
best result in deciding the fakeness of the unseen methods DeepFakes,
Face2Face, and NeuralTextures, but MT_Old has the best result in
recognizing the unseen method FaceSwap.

5. Conclusions
This study utilized meta-learning to train a neural network that can detect
fake images generated from various unseen forgery techniques with a
small number of samples in contrast to the conventional approaches that
train a fake image detector with a comprehensive training dataset
encompassing various forgery techniques. The proposed method
prioritizes utilizing the information from a small number of samples to
quickly adapt the fake detector. The experimental results demonstrate that
the proposed method can significantly improve the performance metrics,
such as AUC, accuracy, and IoU, using a small number of samples. This
also indicates that this approach is a promising direction for further
research. Possible future directions include increasing the number of
forgery techniques (training tasks) and improving the method of extracting
features from a few samples.
This study demonstrates that the meta-learning approach enables us to
train a machine capable of detecting new forgery methods from small
sample sizes. Therefore, if a new forgery method emerges, it can be
detected as long as a few samples are collected. As a result, in the
competition with forgers, the response time of the detector can be reduced.
One constraint of our approach is that it necessitates the collection of a
small amount of training data. Nevertheless, considering the absence of a
technique capable of identifying all novel forgery methods without
additional training, obtaining a small set of training samples remains a
reasonable strategy. The future research direction is to compare the impact
of the number of training tasks on meta-training for detection
performance.

Author Contributions: Conceptualization, Y.-K.L.; Methodology, Y.-

K.L.; Software, T.-Y.Y.; Validation, Y.-K.L.; Formal analysis, Y.-K.L.;
Investigation, T.-Y.Y.; Writing—original draft, Y.-K.L.; Writing—review
& editing, Y.-K.L.; Project administration, Y.-K.L.; Funding acquisition,
Y.-K.L. All authors have read and agreed to the published version of the
manuscript.
Funding: This research was supported by the Ministry of Science and
Technology under grant number MOST-109-2221-E-153-003, Taiwan.
Data Availability Statement: Not applicable.
Conflicts of Interest: The authors declare no conflict of interest. The
funders had no role in the design of the study; in the collection, analyses,
or interpretation of data; in the writing of the manuscript; or in the
decision to publish the results.

References
1. Shiohara, K.; Yamasaki, T. Detecting Deepfakes with Self-Blended
Images. In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, New Orleans, LA, USA, 19–20 June
2022; pp. 18720–18729.
2. Thies, J.; Zollhöfer, M.; Nießner, M. Deferred neural rendering: Image
synthesis using neural textures. ACM Trans. Graph. (TOG) 2019, 38,
1–12. [CrossRef]
3. Thies, J.; Zollhofer, M.; Stamminger, M.; Theobalt, C.; Nießner, M.
Face2face: Real-time face capture and reenactment of rgb videos. In
Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, Las Vegas, NV, USA, 26–30 June 2016; pp. 2387–2395.
4. Thies, J. Face2Face: Real-time facial reenactment. IT-Inf. Technol.
2019, 61, 143–146. [CrossRef]
5. Faceswap. 2018. Available online:
[Link] (accessed on 3 February
2023).
6. Deepfakes. 2018. Available online:
[Link] (accessed on 3 February 2023).
7. Rössler, A.; Cozzolino, D.; Verdoliva, L.; Riess, C.; Thies, J.; Nießner,
M. Faceforensics++: Learning to detect manipulated facial images. In
Proceedings of the IEEE/CVF International Conference on Computer
Vision, Seoul, Republic of Korea, 27 October–2 November 2019; pp.
1–11.
8. Bayar, B.; Stamm, M.C. A deep learning approach to universal image
manipulation detection using a new convolutional layer. In
Proceedings of the 4th ACM Workshop on Information Hiding and
Multimedia Security, Vigo, Spain, 20–22 June 2016; pp. 5–10.
9. Cozzolino, D.; Poggi, G.; Verdoliva, L. Recasting residual-based local
descriptors as convolutional neural networks: An application to image
forgery detection. In Proceedings of the 5th ACM Workshop on
Information Hiding and Multimedia Security, Philadelphia, PA, USA,
20–22 June 2017; pp. 159–164.
10. Nguyen, H.H.; Fang, F.; Yamagishi, J.; Echizen, I. Multi-task learning
for detecting and segmenting manipulated facial images and videos.
arXiv 2019, arXiv:1906.06876.
11. Zhou, Z.; Rahman Siddiquee, M.M.; Tajbakhsh, N.; Liang, J. Unet++:
A nested u-net architecture for medical image segmentation.
In Deep Learning in Medical Image Analysis and Multimodal
Learning for Clinical Decision Support: 4th International Workshop,
DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in
Conjunction with MICCAI 2018, Granada, Spain, September 20, 2018,
Proceedings 4; Springer: Berlin/Heidelberg, Germany, 2018; pp. 3–11.
12. Feng, S.; Fan, Y.; Tang, Y.; Cheng, H.; Zhao, C.; Zhu, Y.; Cheng, C. A
Change Detection Method Based on Multi-Scale Adaptive Convolution
Kernel Network and Multimodal Conditional Random Field for Multi-
Temporal Multispectral Images. Remote Sens. 2022, 14, 5368.
[CrossRef]