JOURNAL OF LATEX CLASS FILES, VOL. 14, NO.

8, AUGUST 2015 1

Pixel-Inconsistency Modeling for Image

Manipulation Localization
Chenqi Kong, Member, IEEE, Anwei Luo, Shiqi Wang, Senior Member, IEEE, Haoliang Li, Member, IEEE,
Anderson Rocha, Fellow, IEEE, and Alex C. Kot, Life Fellow, IEEE

Abstract—Digital image forensics plays a crucial role in image authentication and manipulation localization. Despite the progress
powered by deep neural networks, existing forgery localization methodologies exhibit limitations when deployed to unseen datasets and
perturbed images (i.e., lack of generalization and robustness to real-world applications). To circumvent these problems and aid image
integrity, this paper presents a generalized and robust manipulation localization model through the analysis of pixel inconsistency artifacts.
arXiv:2310.00234v2 [[Link]] 19 Nov 2024

The rationale is grounded on the observation that most image signal processors (ISP) involve the demosaicing process, which introduces
pixel correlations in pristine images. Moreover, manipulating operations, including splicing, copy-move, and inpainting, directly affect such
pixel regularity. We, therefore, first split the input image into several blocks and design masked self-attention mechanisms to model the
global pixel dependency in input images. Simultaneously, we optimize another local pixel dependency stream to mine local manipulation
clues within input forgery images. In addition, we design novel Learning-to-Weight Modules (LWM) to combine features from the two
streams, thereby enhancing the final forgery localization performance. To improve the training process, we propose a novel
Pixel-Inconsistency Data Augmentation (PIDA) strategy, driving the model to focus on capturing inherent pixel-level artifacts instead of
mining semantic forgery traces. This work establishes a comprehensive benchmark integrating 16 representative detection models across
12 datasets. Extensive experiments show that our method successfully extracts inherent pixel-inconsistency forgery fingerprints and
achieve state-of-the-art generalization and robustness performances in image manipulation localization.

Index Terms—Image forensics, image manipulation localization, image manipulation detection, generalization, robustness.

✦
1 I NTRODUCTION

I MAGE manipulation has been carried out since photog-

raphy was born [3]. In recent decades, there has been
significant advances in image manipulation techniques,
people. Image manipulation certainly undermines the trust
in media content. Moreover, the proliferation of fakes has
raised pressing security concerns for the public. Therefore,
including splicing, copy-move, and inpainting, which are designing effective image forgery localization models to
three pervasive but notorious attack types [88], as shown address these issues is paramount.
in Fig. 1. These techniques can produce forgery content Early attempts at image manipulation localization mainly
with a very high level of realism, blurring the boundaries focused on extracting features based on prior knowledge,
between authentic and forgery images. Manipulation traces such as lens distortions [29], [33], [47], [68], [97], [97], Color
are very subtle and can hardly be perceived by the naked eye. Filter Array (CFA) artifacts [10], [26], [30], [39], [77], noise
With the widespread use of digital images on the internet, patterns [17], [23], [49], [65], [66], [76], compression artifacts
it has become much easier for malicious attackers to launch [6], [9], [15], [24], [28], [44], [74]. However, these traditional
manipulation attacks using off-the-shelf yet powerful image methods demonstrate limited accuracy and generalizability.
editing tools, such as Photoshop, After Effects Pro, GIMP, and In turn, learning-based detectors have been proposed thanks
more recently, Firefly. The produced sophisticated content can to recent advancements in deep learning and artificial
be used to commit fraud, generate fake news, and blackmail intelligence. These methods exhibit promising performance
in image forgery localization under the intra-domain setting.
Nonetheless, data-driven methods are typically prone to
• C. Kong and A. Kot are with the Rapid-Rich Object Search (ROSE) Lab,
School of Electrical and Electronic Engineering, Nanyang Technology
overfitting the training data, resulting in limited robustness
University, Singapore, 639798. and generalization performance. Namely, they are fragile
E-mail: [Link]@[Link], eackot@[Link]. to image perturbations and vulnerable to unseen image
• A. Luo is with the School of Computer Science and Engineering, Sun manipulation datasets.
Yat-sen University, Guangzhou, China. He is also with the Rapid-Rich
Object Search (ROSE) Lab, School of Electrical and Electronic Engineering, Extracting inherent forgery fingerprints for generalized
Nanyang Technology University, Singapore, 639798. and robust image forgery localization remains a challenging
E-mail: luoanw@[Link]. problem. This paper recasts the typical image manipulation
• S. Wang is with the Department of Computer Science, City University of
Hong Kong, Hong Kong.
pipeline and proposes a new forgery localization framework
E-mail: shiqwang@[Link]. that captures the pixel inconsistencies in manipulated images.
• H. Li is with the Department of Electrical Engineering, City University of Fig. 2 shows the typical forgery image construction chain.
Hong Kong, Hong Kong. The filter and lens eliminate undesired light and focus light
E-mail: [Link]@[Link].
• A. Rocha is with the Artificial Intelligence Laboratory ([Link]), Institute onto the sensor. Subsequently, the Color Filter Array (CFA)
of Computing, University of Campinas, Campinas 13084-851, Brazil is applied to extract single-color components. A series of
E-mail: arrocha@[Link] software operations is carried out during the in-camera
• Corresponding author: Haoliang Li.
processing. Demosaicing, also known as color interpolation,
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 2

Fig. 2: Typical forgery image construction pipeline.

Fig. 1: Illustration of manipulation types: splicing, copy-

move, and inpainting. The top, middle, and bottom rows
show the real, forgery, and ground-truth images.
Fig. 3: Typical Color Filter Array (CFA) types. (a). Bayer CFA;
(b). RGBE; (c). CMY; (d). CMYG.
is performed to reconstruct full-color pixels from surround-
ing single-color pixels. Some internal processing steps, such
as color correction, noise reduction, and compression, are dependency within input images. Furthermore, we design
subsequently conducted to generate the final processed a Difference Convolution (DC) stream to capture local pixel
RGB image. In turn, malicious attackers can utilize image inconsistency artifacts within local image regions. In addition,
editing tools to manipulate pristine images during the we introduce a novel Learning-to-Weight Modules (LWM) to
out-camera processing. These manipulations can disrupt combine global and local pixel-inconsistency features from
such pixel correlation (i.e., perturb the periodic patterns) these two streams.
introduced by the demosaicing operation, leaving distinctive We design three decoders to predict the potential ma-
pixel inconsistency artifacts for forensics analyses [10], [77], nipulated regions, forgery boundaries, and reconstructed
[88]. images. We finally introduce the Pixel-Inconsistency Data
Augmentation (PIDA) strategy to explore the pixel-level
Fig. 3 showcases four typical CFA types: (a). Bayer CFA,
forgery traces. PIDA is an effective approach that relies upon
(b). RGBE, (c). CMY, and (d). CMYG. Color filtering allows
only real images for data augmentation. It guides the model
the capture of a specific color at each pixel. Consequently, in
to focus on capturing pixel-inconsistency artifacts rather
the resulting RAW image, only one color is present at each
than semantic forgery traces. The designed framework is
pixel, and the demosaicing process reconstructs the missing
trained end-to-end, jointly supervised by the binary mask
color samples. Some existing forensics analysis techniques
and boundary labels.
for forgery fingerprint extraction focus on mathematically
The key contributions of our work are:
modeling different image regularities. For instance, Popescu
et al. [77] quantifies the specific correlations introduced by • We establish a comprehensive benchmark assessing
CFA interpolation and describes how these correlations can the generalization capabilities of 16 representative
be automatically detected. Ferrara et al. [26] proposes a novel image forgery localization methods across 12 datasets.
feature that measures the presence or absence of these image We further extend this benchmark to evaluate the
regularities at the smallest 2×2 block level, thus predicting robustness performance across six unseen image
a forgery probability map. In [11] and [56], the intra-block perturbation types, each with nine severity levels.
fingerprint is modeled using a linear regression approach. Additionally, we evaluate our designed model on
Despite the effectiveness of these pixel correlation modeling sophisticated and advanced manipulations generated
approaches in forensic analysis, most require knowledge by modern Artificial Intelligence Generated Content
of the CFA type as prior information. Furthermore, these (AIGC) techniques.
methods cannot sufficiently capture more complex regular- • We design a two-stream image manipulation localiza-
ities introduced by smart image signal processors (ISPs) in tion framework comprising a local pixel dependency
modern AI cameras [2]. encoder, a global pixel dependency encoder, four
Different from the prior arts, we propose a learning-based feature fusion modules, and three decoders. The
method to capture inherent pixel inconsistencies within proposed model can effectively extract the pixel-
forged images based on this insight. We design a two-stream inconsistency forgery fingerprints, leading to more
pixel-dependency modeling framework for image manipu- generalized and robust manipulation localization
lation localization to achieve this. Drawing inspiration from performance.
recent success of autoregressive models (e.g., PixelCNN • We introduce a Pixel-Inconsistency Data Augmenta-
[86], [87]) in various computer vision tasks, we design a tion strategy that exclusively utilizes real images to
masked self-attention mechanism to model the global pixel create the generated data. The proposed data aug-
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 3

Fig. 4: Proposed image manipulation localization framework. The input image is split into several patches, which are
simultaneously fed forward to the Local Pixel Dependency Encoder and Global Pixel Dependency Encoder. The upper stream
comprises four Difference Convolution (DC) blocks to capture local pixel inconsistencies in forged images. Meanwhile, the
Global Pixel Dependency Encoder, which incorporates four masked self-attention (Masked SA) blocks, focuses on modeling
long-range statistics within the input images. Four Learning-to-Weight Modules (LWM) have been devised to combine
global and local features extracted by the two encoders. The Forgery Decoder and Boundary Decoder take the aggregated
features as inputs and predict the final forgery and boundary maps.

mentation drives the model to focus on capturing the detection [47], [68], [97]. Besides, many methods [10], [11],
inherent pixel-level artifacts rather than the semantic [26], [77] propose to capture color filter array (CFA) artifacts
forgery clues, contributing to a forgery localization to detect manipulations. These techniques demonstrated
performance boost. that manipulation operations can disrupt periodic patterns
• Extensive quantitative and qualitative experimental introduced by the demosaicing process. Additionally, since
results demonstrate that our proposed method consis- photo-response nonuniformity (PRNU) is specific to each
tently outperforms state-of-the-art in generalization camera model, some methods [49], [65], [66] extract noise
and robustness evaluations. Comprehensive ablation patterns from query images for detecting digital tampering
experiments further illustrate the effectiveness of the traces. Furthermore, extensive research has been dedicated
designed components. to studying JPEG compression artifacts that persist in the
discrete cosine transform (DCT) domain [9], [15], [24],
Sec. 2 overviews prior work in image forgery localization
[25], [74] for forgery detection. While these traditional
and pixel dependency modeling. Sec. 3 elaborates on the de-
image manipulation detection methods are explainable and
signed framework. Sec. 4 presents comprehensive evaluation
computationally efficient, most suffer from poor detection
results under diverse experimental settings. Finally, Sec. 5
accuracy and limited generalization. To achieve an accurate,
concludes this paper and discusses current limitations and
generalized, and interpretive image forgery localization, we
possible future research directions.
introduce a learning-based framework in this work designed
to capture low-level pixel inconsistency artifacts.
2 R ELATED W ORK
In this section, we broadly review existing works on image 2.2 Learning-based Manipulation detection and localiza-
forgery detection and localization, including both hand- tion methods
crafted and learning-based methodologies. Additionally, we
Recent years have witnessed significant progress in image
review the studies related to pixel dependency modeling and
forensics, with various learning-based methods proposed to
their applications.
solve the forgery localization problem, which substantially
improved detection performances. Many of these methods
2.1 Manipulation detection and localization methods leverage a wide range of prior knowledge, such as noise tell-
using low-level traces tales [18], [35], [104], CFA artifacts [4], and JPEG features [53],
Image manipulation detection is no new problem. Early [79], [91] to perform the forgery detection. High-frequency
methods focus on detecting low-level artifacts derived from (HF) filters [55], [106], such as steganalysis rich model (SRM)
in-camera processing traces. For example, lens distortions filter [95], [104] and Bayer filter [19], [95] have also been used
[29], [33], [47], [68], [97], [97], introduced by the imperfection to capture abundant HF forgery artifacts. Besides, detecting
of complex optical systems, can be regarded as unique the forgery boundary [19], [82] has effectively improved
fingerprints for forensics purposes. Chromatic aberration pixel-level forgery detection performance. In turn, some
is a typical lens distortion cue widely studied for forgery methods [19], [27], [31], [62] utilize multi-scale learning
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 4

to extract forgery features from different levels, thereby 3.1 Overall Framework
achieving increased detection accuracy. While SPAN [41] As Fig. 4 depicts, this paper designs a two-stream image ma-
models relationships between image patches or pixels at nipulation localization framework, which draws inspiration
multiple scales using a pyramid of local self-attention blocks, from the observation that manipulation processes, such as
our method innovatively employs a local pixel dependency splicing, copy-move, and inpainting, inevitably disrupt the
encoder to capture local pixel-difference, a masked self- pixel regularity introduced by the demosaicing operation.
attention global pixel dependency encoder to model long- The framework relies upon a Local Pixel Dependency En-
range pixel correlations, and feature fusion modules to coder and a Global Pixel Dependency Encoder to explore
combine the forgery fingerprints. These components are pixel inconsistency and context for manipulation localization.
designed to better capture inherent pixel-inconsistency The input image is firstly split into patches, which are then
artifacts within forgery images. Thanks to the advent of concurrently processed by the two encoders. In the patch
vision transformer (ViT), ViT-based detectors [59], [90] take embedding process, we segment the input image into 4×4-
advantage of long-range interaction and no inductive bias, pixel patches. The raw pixel RGB values of each patch are
yielding outstanding detection performance in different flattened into a dimension of 4×4×3=48, and each patch
problems, including forensics. However, these data-driven token is subsequently projected to the embedding dimension.
methods suffer from limited generalization and robustness Intuitively, embedding individual RGB pixel values into
capability. This paper argues that pixel inconsistency within single tokens can help model pixel dependency. However,
forgery images represents a more ubiquitous artifact across this would significantly increase computational costs, as
different manipulations and datasets. As such, we devise a the number of tokens would equal the image’s height (H )
novel image forgery localization framework that captures and width (W ) (H, W = 512 in this work). To address
pixel inconsistency artifacts to achieve more generalized and this and achieve a reasonable tradeoff, we relax the patch
robust forgery localization performance. size to 4×4, balancing computational efficiency with the
effective capture of pixel inconsistencies in manipulated
2.3 Pixel Dependency Modeling images. Compared to individual pixels, the 4×4-pixel token
Autoregressive (AR) models [14], [16], [32], [54], [73], [81], provides a more expressive representation. Our experiments
[86] have achieved remarkable success across various com- show that the proposed method, with the adopted patch
puter vision tasks, including image generation [32], [50], [86], embedding strategy, successfully captures global and local
completion [14], [45], [73], and segmentation [72]. These AR pixel inconsistencies within manipulated images. Addition-
methods aim to model the joint probability distribution of ally, we adopt MLP layers in the designed transformer blocks
each pixel as follows: to enhance the learning of pixel dependencies within each
token [57], [85].
âi ∼ pθ (ai |a1 , ..., ai−1 ). (1) To explore long-range interaction and no inductive bias,
These models employ specific mask convolution or mask we adopt transformer architectures as backbones of the
self-attention strategies, such that the probability distribution two streams. The upper Local Pixel Dependency Encoder
of the current pixel depends on all previous pixels in the comprises four Difference Convolution (DC) Blocks designed
generation order. Pioneering AR models like PixelCNN to capture pixel inconsistencies in local regions. In turn, we in-
[86] and PixelRNN [87] demonstrate their effectiveness in troduce a Global Pixel Dependency Encoder comprising four
modeling long-range pixel dependencies for natural images novel masked self-attention blocks. The designed masked
in the context of image generation. Follow-up variations, self-attention mechanism models global pixel dependencies
such as PixelCNN++ [81], have been introduced to enhance within input images. Additionally, we design four Learning-
image generation performance further. Furthermore, masked to-Weight Modules (LWM) to complementarily combine
self-attention can also aid dependency modeling, such as global features [fg1 , fg2 , fg3 , fg4 ] and local features [fl1 ,
image transformer [73] and sparse transformer [16]. Pixel- fl2 , fl3 , fl4 ] at multiple levels. The designed framework also
SNAIL [14] combines causal convolutions with self-attention, incorporates a Boundary Decoder, a Forgery Decoder, and
improving image generation. Inspired by the success of pixel- an Image Decoder.
dependency modeling in various generative tasks, we seek to Notably, pixel inconsistency is most prominent in the
extend upon this concept to the domain of forensic analysis. boundary region. We, therefore, integrate the boundary auxil-
This paper introduces novel pixel-difference convolutions iary supervision to enhance the final forgery localization per-
and masked self-attention mechanisms to capture local and formance. The Forgery Decoder takes the combined features
global pixel inconsistency artifacts. [f1 , f2 , f3 , f4 ] as inputs to predict potential manipulated
regions of input images, while the Image Decoder takes [fg1 ,
fg2 , fg3 , fg4 ] as inputs and aims to reconstruct the original
3 P ROPOSED M ETHOD
input image. Finally, we propose a novel Pixel-Inconsistency
This section presents the proposed manipulation localization Data Augmentation (PIDA) strategy that focuses on pixel
method. We first introduce the overall framework. Subse- inconsistency rather than semantic forgery traces. This
quently, we delve into the details and underlying rationales strategy further enhances the model’s generalization and
of the designed components, including the Global Pixel robustness capabilities.
Dependency Modeling Module, the Local Pixel Dependency
Modeling Module, and the Learning-to-Weight Module. 3.2 Global Pixel Dependency Modeling
Lastly, we introduce the proposed Pixel-Inconsistency Data In this part, our goal is to model the global pixel dependency
Augmentation strategy and its advantages. across image blocks, with each token conditioned on the
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 5

rewritten as [14]:
X
out
zm = γmn yvalue (znin ), (3)
n≤m

where elements γmn in row m can be formulated as:

γm = softmax[ykey (z1in )⊤ yquery (zm
in in ⊤
), ..., ykey (zm in
) yquery (zm )],
(4)
out
In Eq. (3), we can readily observe that each output token zm
in
is conditioned on the previous seen tokens zn (n ≤ m) in the
input z in , and the scan order follows a raster-scan ordering.
This mechanism also facilitates modeling more complex
pixel dependencies in real-world applications, such as the
dependency introduced by smart image signal processors in
modern AI cameras. As such, each conditional can access
any pixel within its context through the attention operator,
as indicatedPby the summation over all available context,
denoted as n≤m .
This designed module enables the access of far-away pix-
els, thereby enhancing the modeling of long-range statistics.
Fig. 5: (a). Illustration of the proposed masked attention As such, the extracted features [fg1 , fg2 , fg3 , fg4 ] can carry
mechanism. ⊗ indicates the matrix multiplication. Q, K, and abundant global pixel dependency information. Experimen-
V stand for Query, Key and Value. We designed the Raster- tal results demonstrate that the captured pixel correlations
scan mask to model the pixel dependency. (b). The mask between real and manipulated images are distinctive for
and corresponding pixel scan ordering. The green squares image forgery localization.
indicate the value ‘1’ while the red squares indicate the value
‘0’.
3.3 Local Pixel Dependency Modeling
According to the nature of demosaicing algorithms, the pixel
correlation regularity of a given pixel largely depends on its
previous ones in a raster scan ordering. Consequently, each neighboring pixels [10], [11]. Moreover, the pixel regularity
output token highly depends on all previous “seen” pixels. can be modeled by linear demosaicing formulas [10], [56].
Compared to processing individual pixels, this design is However, these traditional methods exhibit limited forgery
more computational-efficient. Given the spatial redundancy detection performance. Inspired by [60], [83], [102], we pro-
in images [37], the proposed method can also effectively pose to model the local pixel dependency by integrating the
model global pixel dependencies. traditional demosaicing ideas into convolutional operations.
In the Local Pixel Dependency Encoder, we place Differ-
Inspired by [14] and [73] that model long-term pixel de- ence Convolution (DC) heads on top of each transformer
pendency using attention mechanism, we introduce Masked block to model pixel dependency in local image regions in
Self-Attention (Masked SA) blocks, in a style similar to the a learning-based fashion. Our designed Difference Convo-
self attention, into pixel global dependency modeling. Fig. 4 lutions (DC) are performed at the token level, with each
depicts the combination of the global pixel dependency token representing a very small image block. Compared to
encoder and the image decoder that forms an auto-encoder. processing individual pixels, the 4×4 image block provides
Fig. 5 (a) illustrates the details of the proposed masked self- a more expressive representation for performing difference
attentions and the corresponding mask design, with Q, K, convolutions. Our method significantly reduces computa-
and V representing Query, Key, and Value, respectively. (We tional costs while effectively capturing pixel inconsistencies
omit the normalization and MLP layers for conciseness). z in in local image regions. Moreover, we adopt MLP layers in
and z out indicate the input and output features. ⊗ denotes each transformer block to further enhance the learning of
the matrix multiplication operator. The masked self-attention local pixel dependencies within each block.
mechanism can be formulated as: Fig. 6 (a). depicts the architecture of the designed Dif-
yquery (z in )ykey (z in )⊤ ference Convolution (DC) head. The input feature flin is
z out = Mask[softmax( √ )]yvalue (z in ), firstly fed forward to two difference convolution modules:
dim
(2) Central Difference Convolution (CDC) and Radial Difference
where yquery (·), ykey (·), and yvalue (·) represent the learnable Convolution (RDC). By exploiting CDC and RDC, the local
parameters, and yquery (z in ), ykey (z in ), and yvalue (z in ) are pixel dependencies can be effectively modeled, enhancing
equivalent to Q, K, and V. As Fig. 5 (b) shows, we employ the final forgery localization performance. Fig. 6. (b) presents
a raster-scan mask to model the global pixel dependency, the details of CDC and RDC. The input tokens, which
corresponding to the raster-scan sampling ordering for the are the output of transformer blocks in the Local Pixel
input image [72]. If we name the input z in ∈ RN ×dim Dependency Encoder, are reshaped into a 2D feature flin .
in ⊤
as z in =[z1in , z2in ,..., zN ] , then each row zm in
represents a We first calculate the difference within local feature map
out N ×dim
input token. For the output z ∈R of the proposed regions for a given input feature map. Then, we respectively
out
masked attention mechanism, each output token zm can be convolve the two pixel-difference feature maps with the
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 6

the fused feature fl is obtained by adding the concatenated

feature to the weighted feature.
Similarly, as depicted in Fig. 4, we further employ LWM
to fuse the local pixel-dependency features [fl1 , fl2 , fl3 , fl4 ]
and the global pixel-dependency features [fg1 , fg2 , fg3 , fg4 ].
The fused features [f1 , f2 , f3 , f4 ] are then delivered to the
boundary and forgery decoder for boundary and forgery
map prediction.

3.5 Pixel-Inconsistency Data Augmentation

Previous methods [90] mainly focus on discovering semantic-
level (or object-level) inconsistencies in forgery images. Some
methods [19], [105] also propose randomly pasting objects to
pristine real images to perform data augmentation. However,
as image manipulation techniques advance, forgery content’s
sophistication grows in tandem. Consequently, the methods
Fig. 6: (a). Difference Convolution (DC) head; (b). Details of designed to capture semantic-level inconsistencies struggle to
Central Difference Convolution (CDC) and Radial Difference generalize well to the advanced manipulations. We introduce
Convolution (RDC); (c). Details of Learning-to-Weight Mod- a Pixel-Inconsistency Data Augmentation (PIDA) strategy to
ule (LWM). capture pixel-level inconsistencies instead of semantic forgery
traces. Fig. 7 (a) illustrates the proposed PIDA pipeline. ➀ For
a given real pristine image Ip , we apply image perturbations
corresponding convolutional weights, resulting in CDC and (e.g., compression, noise, and blurriness) to generate the
RDC feature maps. The CDC operation can be formulated corrupted image Ic ; ➁ We can readily use built-in OpenCV
as: X function to extract the foreground mask M of Ip . The
flC = wiC (xi − xc ). (5) Blending Module takes Ip , Ic , and M as inputs and produces
(xi ,xc )∈ΩC the self-blended forge image Ib ; ➂ The boundary label B
Here, xc represents the center element in the local region of the manipulated image can be easily derived from M .
ΩC , and xi denotes the corresponding surrounding elements. Fig. 7 (b) details the blending module. We combine the donor
Each element in ΩC or ΩR depicted in Fig. 6 (b) represents image’s foreground with the target image’s background to
a token. The wiC values represent learnable convolutional generate the self-blended forgery sample.
weights. Similarly, the RDC operation can be expressed as: The proposed PIDA method bears the following advan-
tages: (1) It exclusively utilizes pristine images to generate
X ′
flR = wiR (xi − xi ), (6) examples of forgeries. Real data is considerably more acces-
′
(xi ,xi )∈ΩR sible than image forgeries, facilitating training data-hungry
′
detectors; (2) As the generated forgery samples maintain
where xi and xi are element pairs in region ΩR , as illustrated semantic consistency, the PIDA strategy directs the model’s
in Fig. 6 (b). attention toward capturing pixel inconsistencies, enhancing
We complementarily combine CDC features flC and RDC detection performance; (3) The generated forgery samples
features flR using a Learning-to-Weight Module (LWM), can be regarded as harder samples, effectively increasing the
which shall be elaborated in Sec. 3.4. Our designed model difficulty of the training set. More PIDA details can be found
aims at extracting local pixel-dependency features. Com- in the Appendix.
pared to the vanilla convolution, CDC and RDC benefit
from their difference operations, exposing more pixel in- 3.6 Objective Function
consistency artifacts and boosting the final image forgery The whole framework is trained in an end-to-end manner,
localization performance. and the overall objective function consists of the following
four components: mask prediction loss LM , boundary predic-
tion loss LB , compactness loss LC , and image reconstruction
3.4 Learning-to-Weight Module
loss LR :
As Fig. 6 (a) shows, the features flC and flR generated by
CDC and RDC are combined and sequentially delivered L = LM + λB LB + λC LC + λR LR , (7)
to the Learning-to-Weight Module (LWM). The designed where LM and LB are cross-entropy losses between pre-
LWM fuses these two input features using learned weights, dicted results and the corresponding labels. The boundary
enabling more effective feature integration. Fig. 6 (c) show- loss LB can be considered as an auxiliary supervision for
cases the Learning-to-Weight process for local CDC features better forgery localization performance. Based on the obser-
vation that most manipulated regions are rather compact,
flC and RDC features flR , where FC and EltMul represent
we further apply compactness constraint LC to predicted
the fully-connected layer and element-wise multiplication. masks:
In this process, the concatenated feature goes through one Nimg Nimg P 2
average pooling layer and one FC layer. The learned weights 1 X P eri2 1 X j∈B̂ B̂j
LC = = .
A1 ⊕ A2 are then sequentially applied to the concatenated Nimg 4πS Nimg i=1 4π( k∈M̂ |M̂k | + ϵ)
P
i=1
feature flC ⊕ flR via element-wise multiplication. Finally, (8)
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 7

Fig. 7: (a). Pixel-Inconsistency Data Augmentation pipeline. ➀ For a given real pristine image Ip , we firstly apply common
image perturbations to obtain the corrupted image Ic ; ➁ We use built-in OpenCV function to extract the foreground mask
M of Ip ; The Blending Module takes Ip , Ic , and M as inputs and outputs the self-blended forge image Ib . ➂ The boundary
label B of the manipulated image can be obtained from M . (b). Details of the blending module in (a). The output blended
image is the combination of the donor image’s foreground and the target image’s background.

In this equation, P eri and S denote the perimeter and area images with diverse contents. Furthermore, we employ the
of the predicted forgery region, respectively, while Nimg DEF-12k-val [67] as the validation set, consisting of 6,000
represents the number of images. B̂ and M̂ refer to the challenging fake images with three forgery types and 6,000
predicted boundaries and masks. As such, the nominator real images collected from the MS-COCO [58] dataset. For
of Eq. (8) calculates the sum of the squared pixel values the testing phase, we select 11 challenging datasets, including
2
B̂j in the predicted boundary map B̂ . The denominator is Columbia [40], IFC [1], CASIAv1+1 [21], WildWeb [103],
proportional to the sum of the absolute pixel values M̂k in COVER [92], NIST2016 [34], Carvalho [12], Korus [52], In-
the predicted mask map M̂ . Here, ϵ is set to a very small the-wild [43], DEF-12k-test [67], and IMD2020 [71], sorted
value. Utilizing LC makes the predicted image forgery map by released dates. In all datasets, we uniformly label forgery
more compact and improves the manipulation localization regions as ‘1’ and authentic regions as ‘0’.
performance.
The image reconstruction loss LR calculates the l1 −norm
4.2 Evaluation metrics
of the difference between the reconstructed images Iˆi and
the corresponding input images Ii : This paper evaluates state-of-the-art models’ pixel-level
N forgery detection performances using four metrics: F1, MCC,
1 X
IoU, and AUC.
LR = ||Ii − Iˆi ||1 . (9)
N i=1 F1 Score is a pervasive metric in binary classification,
employed in image forgery detection and localization. It
By using LR , the global pixel dependency can be modeled in
calculates the harmonic mean of precision and recall:
[fg1 , fg2 , fg3 , fg4 ], which is used in the LWMs for the forgery
map and boundary map prediction. P recision × Recall 2 × TP
F1 = 2 · = ,
P recision + Recall 2 × TP + FP + FN
(10)
4 E XPERIMENTS AND R ESULTS
where T P , T N , F P , and F N represent True Positives, True
Herein, we first introduce the datasets, evaluation metrics, as Negatives, False Positives, and False Negatives.
well as baseline models involved in this work. Subsequently, Matthews Correlation Coefficient (MCC) measures the
we evaluate our model in terms of generalization and correlation between the predicted and true values. MCC
robustness under different experimental settings. We also value falls within -1 and 1, where a higher MCC indicates
visualize the forgery localization results to illustrate the better performance. The calculation of MCC is derived from
superiority of our method. Finally, we conduct ablation the formula below:
studies to demonstrate the effectiveness of the designed
components.
TP × TN − FP × FN
M CC = p .
4.1 Datasets (T P + F P )(T P + F N )(T N + F P )(T N + F N )
(11)
This paper adopts 12 image manipulation datasets with Intersection over Union (IoU) is a widely used metric in
varying properties, images resolutions and quality. We semantic segmentation. The numerator of the IoU metric
summarize these datasets in Table 1, where CM, SP, and IP measures the area of intersection between prediction P and
denote three common image manipulation types: copy-move,
splicing, and inpainting. Consistent with previous research
1. CASIAv1+ and the training set CASIAv2 share 782 identical real
[19], [82], [104], we utilize the CASIAv2 [20] dataset as the images. To prevent data leakage, CASIAv1+ relaces these real images
training set due to its extensive collection of over 12,000 with the equal number of images from COREL [89].
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 8

TABLE 1: Summary of image manipulation datasets involved H-LSTM [5] is a forgery detection model that integrates
in this paper. CM, SP, and IP indicate three common image both a CNN encoder and LSTM networks. This combination
manipulation types: copy-move, splicing, and inpainting. enables the model to capture and analyze spatial and
frequency domain artifacts in forgery images.
Dataset Year Venue #Real #Fake #CM #SP #IP
SPAN [41] is a framework that constructs a pyramid attention
CASIAv2 [20] 2013 ChinaSIP 7,491 5,123 3,295 1,828 0
network to capture the interdependencies between image
DEF-12k-val [67] 2019 EUSIPCO 6,000 6,000 2,000 2,000 2,000
patches across multiple scales. It builds upon the foundation
Columbia [40] 2006 ICME 183 180 0 180 0
of the pre-trained MantraNet and offers the flexibility to
IFC [1] 2013 IFC-TC 1050 450 - - - fine-tune its parameters on specific training sets.
CASIAv1+ [21] 2013 ChinaSIP 800 920 459 461 0 PSCC [62] is a progressive spatial-channel correlation net-
WildWeb [103] 2015 ICMEW 99 9,657 0 9,657 0 work, which extracts local and global features at multiple
COVER [92] 2016 ICIP 100 100 100 0 0 scales with dense cross-connections. The progressive learning
NIST2016 [34] 2016 OpenMFC 0 564 68 288 208 mechanism enables the model to predict the forgery mask
Carvalho [12] 2016 IEEE TIFS 100 100 0 100 0 in a coarse-to-fine manner, thereby empowering the final
Korus [52] 2016 WIFS 220 220 - - - detection performance.
In-the-wild [43] 2018 ECCV 0 201 0 201 0 MVSS-Net++ [19] designs a two-stream network to capture
DEF-12k-test [67] 2019 EUSIPCO 6,000 6,000 2,000 2,000 2,000 boundary and noise artifacts using multi-scale features.
IMD2020 [71] 2020 WACVW 404 2010 - - - Incorporating two streams effectively analyzes different
aspects of the image to detect manipulations at both pixel
and image levels.
ground-truth G, while the denominator calculates the area CAT-NET [53] is a CNN-based model that leverages dis-
of the union between P and G: crete cosine transform (DCT) coefficients to capture JPEG
P ∩G compression artifacts in manipulated images.
IoU = . (12) EVP [61] presents a unified low-level structure detection
P ∪G
framework for images. ViT Adaptors and visual prompt-
Area Under Curve (AUC) measures the area under the
ings enable the EVP model to achieve outstanding forgery
Receiver Operating Characteristic (ROC) curve. Unlike the
localization accuracy.
other metrics, the AUC does not require threshold selection.
TruFor 2 [35] concurrently captures high-level RGB artifacts
It quantifies the overall performance of the model across all
and low-level noise forgery traces through a transformer-
possible thresholds.
based fusion architecture based on a learned noise-sensitive
fingerprint.
4.3 Baseline Models In this work, for a fair and reproducible comparison, we
This paper incorporates 16 representative baseline detectors follow MVSS-Net++ [19], selecting baseline models that meet
from top journals and conferences, including five data- one of the following three criteria: (1) official training code
driven architectures and 11 state-of-the-art image forgery is publicly available; (2) the model uses the same training
detectors. The goal is to evaluate the detection performance protocol as ours, i.e., CASIAv2 is used as the training dataset;
of different network architectures and facilitate a head-to- or (3) official pretrained models are released. During testing,
head comparison. The baselines include three pervasive CNN we follow the protocols of MVSS-Net++ [19] and JPEG-
architectures (FCN [64], U-Net [80], and DeepLabv3 [13]) SSDA [79], testing the trained models on forgery images
and two vision transformers (ViT-B [22] and Swin-ViT [63]). and reporting the image-level detection results for all testing
Furthermore, this benchmark incorporates ten state-of-the-art datasets. The selected manipulation methods encompass
image forgery detection models: a wide variety of forgery fingerprints, such as boundary
MFCN [82] casts the image splicing localization as a multi- artifacts (MFCN [82], MVSS-Net++ [19]), multi-scale features
task problem. It exploits the two-branch FCN VGG-16 (PSCC [62], MVSS-Net++ [19], TruFor [35]), high-frequency
network to predict the forgery map and boundary map artifacts (HPFCN [55], MVSS-Net++ [19], MantraNet [95]),
simultaneously. and compression artifacts (CAT-NET [53]).
RRU-Net [8] is an end-to-end ringed residual U-Net archi-
tecture specifically designed for image splicing detection. 4.4 Implementation Details
It leverages residual propagation to address the issue of
Our models are implemented in PyTorch [75] and trained on
gradient perturbation in deep networks effectively. By incor-
two Quadro RTX 8000 GPUs. The input image size is 512 ×
porating this mechanism, RRU-Net strengthens the learning
512. We use Adam optimizer [48] with β1 =0.9 and β2 =0.999
process of forgery clues.
to train the designed model with batch size 28. The learning
MantraNet [95] is an end-to-end image forgery detection and
rate and weight decay are 6e-5 and 1e-5, respectively. The
localization framework trained on a dataset consisting of 385
model is trained for 20 epochs and validated every 1,600
manipulation types. To achieve robust image manipulation
global steps. Following the experimental setting of [19], we
detection, MantraNet introduces a novel long short-term
train our model on CASIAv2 [20] dataset and validate it
memory solution specifically designed to detect local anoma-
on DEF-12k-val [67] dataset. Besides the proposed Pixel-
lies.
Inconsistency Data Augmentation, we follow [19] to use
HPFCN [55] ensembles the ResNet blocks and a learnable
high-pass filter to perform the pixel-wise inpainting localiza- 2. For a head-to-head comparison, we align the TruFor training,
tion. validation, and testing sets with ours.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 9

TABLE 2: Image manipulation localization performance (F1 score with fixed threshold: 0.5).

Method Venue NIST Columbia CASIAv1+ COVER DEF-12k IMD Carvalho IFC In-the-Wild Korus WildWeb AVG
FCN [64] CVPR15 .167 .223 .441 .199 .130 .210 .068 .079 .192 .122 .110 .176
U-Net [80] MICCAI15 .173 .152 .249 .107 .045 .148 .124 .070 .175 .117 .056 .129
DeepLabv3 [13] TPAMI18 .237 .442 .429 .151 .068 .216 .164 .081 .220 .120 .098 .202
MFCN [82] JVCIP18 .243 .184 .346 .148 .067 .170 .150 .098 .161 .118 .102 .162
RRU-Net [8] CVPRW19 .200 .264 .291 .078 .033 .159 .084 .052 .178 .097 .092 .139
MantraNet [95] CVPR19 .158 .452 .187 .236 .067 .164 .255 .117 .314 .110 .224 .208
HPFCN [55] ICCV19 .172 .115 .173 .104 .038 .111 .082 .065 .125 .097 .075 .105
H-LSTM [5] TIP19 .357 .149 .156 .163 .059 .202 .142 .074 .173 .143 .141 .160
SPAN [41] ECCV20 .211 .503 .143 .144 .036 .145 .082 .056 .196 .086 .024 .148
ViT-B [22] ICLR21 .254 .217 .282 .142 .062 .154 .169 .071 .208 .176 .117 .168
Swin-ViT [63] ICCV21 .220 .365 .390 .168 .157 .300 .183 .102 .265 .134 .040 .211
PSCC [62] TCSVT22 .173 .503 .335 .220 .072 .197 .295 .114 .303 .114 .112 .222
MVSS-Net++ [19] TPAMI22 .304 .660 .513 .482 .095 .270 .271 .080 .295 .102 .047 .284
CAT-NET [53] IJCV22 .102 .206 .237 .210 .206 .257 .175 .099 .217 .085 .170 .179
EVP [61] CVPR23 .210 .277 .483 .114 .090 .233 .060 .081 .231 .113 .099 .181
TruFor [35] CVPR23 .268 .829 .532 .280 .148 .359 .213 .127 .361 .122 .169 .310
PIM Ours .280 .680 .566 .251 .167 .419 .253 .155 .418 .234 .236 .333

TABLE 3: Image manipulation localization performance (IoU score with fixed threshold: 0.5).

Method Venue NIST Columbia CASIAv1+ COVER DEF-12k IMD Carvalho IFC In-the-Wild Korus WildWeb AVG
FCN [64] CVPR15 .114 .177 .367 .117 .089 .158 .043 .058 .140 .089 .084 .131
U-Net [80] MICCAI15 .128 .097 .204 .072 .031 .105 .082 .048 .121 .082 .044 .092
DeepLabv3 [13] TPAMI18 .191 .353 .361 .106 .050 .159 .112 .058 .162 .084 .073 .155
MFCN [82] JVCIP18 .193 .123 .291 .100 .050 .124 .103 .074 .112 .083 .080 .121
RRU-Net [8] CVPRW19 .156 .196 .244 .057 .024 .119 .057 .039 .131 .068 .080 .106
MantraNet [95] CVPR19 .098 .301 .111 .139 .039 .098 .153 .068 .201 .061 .146 .129
HPFCN [55] ICCV19 .126 .076 .137 .070 .026 .076 .054 .045 .084 .064 .057 .074
H-LSTM [5] TIP19 .276 .090 .101 .108 .037 .131 .084 .047 .106 .094 .095 .106
SPAN [41] ECCV20 .156 .390 .112 .105 .024 .100 .049 .037 .132 .055 .015 .107
ViT-B [22] ICLR21 .197 .164 .232 .101 .045 .192 .121 .051 .152 .130 .094 .134
Swin-ViT [63] ICCV21 .167 .297 .356 .124 .129 .243 .132 .078 .214 .103 .033 .171
PSCC [62] TCSVT22 .108 .360 .232 .130 .042 .120 .185 .067 .193 .066 .070 .143
MVSS-Net++ [19] TPAMI22 .239 .573 .397 .384 .076 .200 .188 .055 .219 .075 .034 .222
CAT-NET [53] IJCV22 .062 .140 .165 .141 .152 .183 .110 .062 .144 .049 .107 .120
EVP [61] CVPR23 .160 .213 .421 .083 .070 .183 .043 .062 .182 .084 .071 .143
TruFor [35] CVPR23 .212 .781 .481 .215 .121 .297 .159 .100 .303 .095 .138 .264
PIM Ours .225 .604 .512 .188 .133 .340 .194 .119 .338 .182 .193 .275

common data augmentation for training, including flipping, dataset and validate it on DEF-12k-val [67] dataset. To
blurriness, compression, noise, pasting, and inpainting. facilitate a comprehensive interpretation of the results, we
report two key metrics, namely F1 and IoU, in Table 2 and
4.5 Cross-Dataset Evaluation Table 3, which have been widely used in image forgery
Pixel-level evaluation. Localizing manipulated regions in localization. We further provide the AUC and MCC results
forgery images is crucial as it provides evidence regarding in the Appendix. We highlight the best localization results
the regions that have been manipulated. Predicted forgery in bold and underline the second-best results. Unlike in
regions can unveil the potential intents of attackers [51]. [19] where optimal thresholds are determined individually
However, most detectors suffer from poor localization perfor- for each model and dataset, we set the default decision
mance in cross-dataset evaluations due to substantial domain threshold of F1, MCC, and IoU as 0.5 for the following
gaps between the training and testing sets. Herein, we two reasons: (1). In real-world application scenarios, it is
evaluate the generalization capability of different detectors unlikely to predefine different optimal threshold values for
in terms of pixel-level forgery detection (i.e., manipula- each testing data sample, and (2). Unifying the decision
tion localization). In line with the cross-dataset evaluation threshold as 0.5 enables us to compare all baseline models
protocols in [19], we train our model on CASIAv2 [20] fairly. The pixel-level evaluation at different thresholds is
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 10

TABLE 4: Image-level manipulation detection performance (F1 score with fixed threshold: 0.5).

Method Venue NIST Columbia CASIAv1+ COVER DEF-12k IMD Carvalho IFC In-the-Wild Korus WildWeb AVG
FCN [64] CVPR15 .897 .702 .713 .653 .607 .827 .566 .441 .908 .627 .769 .701
U-Net [80] MICCAI15 .945 .692 .673 .660 .633 .878 .662 .466 .972 .637 .715 .721
DeepLabv3 [13] TPAMI18 .939 .724 .746 .660 .626 .867 .646 .441 .974 .610 .827 .733
RRU-Net [8] CVPRW19 .871 .678 .661 .553 .564 .798 .646 .387 .877 .587 .602 .657
HPFCN [55] ICCV19 .893 .664 .580 .624 .615 .824 .636 .446 .902 .632 .715 .685
ViT-B [22] ICLR21 .969 .707 .653 .671 .646 .870 .664 .448 .972 .644 .829 .734
PSCC [62] TCSVT22 .953 .698 .577 .660 .646 .866 .674 .463 .972 .649 .812 .725
MVSS-Net++ [19] TPAMI22 .831 .735 .758 .659 .646 .863 .613 .472 .953 .613 .540 .698
CAT-NET [53] IJCV22 .982 .687 .548 .641 .642 .885 .662 .464 .992 .668 .685 .714
EVP [61] CVPR23 .878 .623 .746 .569 .563 .813 .554 .418 .828 .573 .888 .678
TruFor [35] CVPR23 .858 .740 .743 .643 .569 .821 .610 .414 .886 .530 .760 .689
PIM Ours .973 .702 .779 .655 .651 .896 .669 .458 .977 .657 .932 .759

presented in the Appendix. and challenging inpainting datasets include CA [99], EC

F1-score is the most widely used metric in this field [69], GC [100], LB [94], LR [36], NS [7], PM [38], RN [101],
[21], [78], [79], [98]. In Table 2, our method achieves the SG [42], SH [96], and TE [84], which are widely used in
best detection F1-score on six datasets and the second-best previous inpainting detection works [93]. From Table 5, it
performance on two datasets. In comparison to the state-of- can be observed that CAT-NET and TruFor benefit from their
the-art method TruFor [35], the proposed Pixel-Inconsistency extensive training data and their ability to capture low-level
Modelling (PIM) method demonstrates superior forgery artifacts, achieving promising average forgery localization
localization F1-scores across nine datasets, with an aver- performance. However, our proposed method PIM achieves
age improvement of 2.3% average F1-score improvement, the highest F1 scores on eight inpainting datasets, with F1
increasing from 31.0% to 33.3%. In Table 3, our method Pixel- score of 0.649 on average, outperforming previous methods
Inconsistency Modelling (PIM) consistently achieves the best by a significant margin.
or second-best detection performance on unseen testing Our method’s superior generalizability to unforeseen
datasets. Even though the 11 unseen datasets exhibit diverse manipulation techniques can be attributed to two key
distributions, our method’s average IoU score outperform all designs: (1) The Pixel-Inconsistency Data Augmentation
previous approaches by a significant margin. The superiority (PIDA) strategy enables the model to capture more general
of the proposed method can be attributed to its ability and subtle artifacts, effectively mitigating overfitting during
to capture pixel inconsistency artifacts, which serve as a training; (2) The designed network effectively captures both
common fingerprint across different forgery datasets. global and local pixel inconsistency artifacts, enabling the
Image-level evaluation. In this subsection, we further eval- model to reveal more inherent pixel-level artifacts rather than
uate the image-level forgery detection under cross-dataset semantic traces.
evaluation. Ideally, the tampering probability map should
all be zero for a pristine real image. To this end, we employ 4.7 Generalization to Sophisticated Manipulations
maximum pooling on the tampering probability map and To examine our model’s generalizability to sophisticated
utilize the resulting output score as the overall prediction for manipulations, we test our model on two datasets: Dall-
the input image [79]. We present the key metric F1 score in E2 (DE2) and Stable Diffusion (SD). DE2 and SD include
Table 4. We highlight the best results in bold and underline 60 and 328 sophisticated fake images, respectively. The
the second-best results. Notably, our method achieves the forgery images exhibit high-level harmonization, with the
top-2 image-level detection performance on eight datasets: forgery regions having compatible illumination, reasonable
NIST, CASIAv1+, DEF-12k, IMD, Carvalho, In-the-Wild, size, semantic consistency, and appropriate position. The
Korus, and WildWeb. Even in cases where our method ranks generation pipelines of the two sophisticated datasets are
6th on the COVER dataset and 5th on the IFC datasets, detailed in the Appendix. The forgery localization per-
it closely approaches the best detection results (COVER: formances (F1, IoU, AUC, and MCC scores) on unseen
Ours: .655 v.s. Best: .671; IFC: Ours: .458 v.s. Best: .472). Our sophisticated manipulation techniques are shown in Table 6.
method achieves the best average results, demonstrating its While the state-of-the-art TruFor achieves decent localization
outstanding forgery detection generalization performance. performance in terms of the listed metrics, our proposed
method, Pixel-Inconsistency Modelling (PIM), outperforms
4.6 Cross-Manipulation Evaluation all other methods across most metrics. For both DE2 and
To evaluate the model’s generalization capability to SD datasets, PIM achieves the highest F1, IoU, and MCC
unseen manipulation techniques, we train our model on scores, indicating superior generalization capability in image
the CASIAv2 dataset and test it on the unseen Inpainting forgery localization for sophisticated manipulations. The
(IP) manipulation. The cross-manipulation F1 score on 10 superiority of PIM on sophisticated manipulations generated
inpainting techniques is presented in Table 5, and the IoU by advanced AIGC technologies suggests that PIM is highly
performance can be found in the Appendix. The 10 typical effective at generalizing to unseen and complex manipu-
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 11

TABLE 5: Image manipulation localization performance (F1 score with fixed threshold: 0.5) on the unseen manipulation
type: Inpainting.

Method Venue CA EC GC LB LR NS PM RN SG SH TE AVG

FCN [64] CVPR15 .089 .032 .009 .026 .468 .136 .230 .120 .304 .106 .063 .144
U-Net [80] MICCAI15 .010 .011 .007 .004 .334 .543 .104 .060 .066 .044 .507 .154
DeepLabv3 [13] TPAMI18 .105 .069 .011 .021 .566 .648 .265 .185 .467 .131 .593 .278
MFCN [82] JVCIP18 .012 .018 .003 .011 .169 .588 .044 .059 .042 .057 .574 .143
RRU-Net [8] CVPRW19 .036 .054 .029 .021 .452 .538 .194 .096 .177 .078 .444 .193
MantraNet [95] CVPR19 .270 .419 .272 .395 .070 .425 .045 .294 .107 .355 .354 .273
HPFCN [55] ICCV19 .011 .012 .008 .008 .154 .490 .020 .035 .017 .030 .447 .112
H-LSTM [5] TIP19 .049 .033 .043 .039 .117 .059 .043 .062 .038 .048 .049 .053
SPAN [41] ECCV20 .009 .031 .009 .005 .357 .432 .116 .108 .184 .017 .224 .136
ViT-B [22] ICLR21 .021 .018 .016 .029 .103 .354 .020 .035 .030 .049 .339 .092
Swin-ViT [63] ICCV21 .206 .221 .005 .071 .377 .218 .402 .296 .335 .266 .064 .224
PSCC [62] TCSVT22 .314 .314 .108 .201 .292 .652 .191 .279 .349 .238 .613 .323
MVSS-Net++ [19] TPAMI22 .087 .049 .012 .020 .575 .814 .313 .233 .390 .192 .809 .318
CAT-NET [53] IJCV22 .547 .530 .382 .757 .335 .459 .244 .550 .572 .623 .469 .497
EVP [61] CVPR23 .277 .375 .058 .398 .484 .312 .350 .340 .499 .534 .300 .357
TruFor [35] CVPR23 .181 .158 .166 .301 .162 .200 .066 .145 .104 .123 .199 .164
PIM Ours .628 .660 .080 .790 .774 .836 .537 .457 .890 .631 .853 .649

lation techniques, ensuring a robust model for real-world TABLE 6: Image manipulation localization performance
applications. on unseen sophisticated manipulations. (DE2: Dall-E2, SD:
Stable Diffusion)
4.8 Generalization to Advanced Manipulations
F1 IoU AUC MCC
With the rapid development of AIGC technologies, Method
DE2 SD DE2 SD DE2 SD DE2 SD
forgery images are becoming increasingly photorealistic, and FCN [64] .122 .248 .065 .141 .708 .847 .137 .250
the barrier to using AIGC tools is much lower. Therefore, it
U-Net [80] .314 .173 .186 .095 .921 .834 .314 .170
is crucial to detect these emerging advanced manipulations.
We adapt our model to two image manipulation datasets: DeepLabv3 [13] .116 .171 .062 .094 .825 .807 .110 .166
Autosplice [46] and CocoGlide [70], which are generated MFCN [82] .178 .171 .097 .093 .806 .692 .180 .166
by advanced AIGC methodologies. Autosplice [46] is a text- RRU-Net [8] .253 .118 .145 .063 .922 .802 .262 .113
prompt manipulated image dataset generated by powerful MantraNet [95] .021 .012 .011 .006 .839 .770 .000 .000
large vision language models. It includes 2,273 real images HPFCN [55] .122 .087 .065 .045 .831 .694 .112 .082
and 3,621 manipulated images, with each forgery image
H-LSTM [5] .255 .068 .181 .042 .822 .713 .262 .069
having three JPEG compression quality factors: 75, 90, and
SPAN [41] .131 .178 .070 .098 .905 .859 .122 .178
100 (with higher values indicating better image quality).
CocoGlide includes 512 photorealistic forgery images, gen- ViT-B [22] .245 .156 .142 .085 .862 .804 .241 .161
erated from the COCO 2017 validation set using the text- Swin-ViT [63] .214 .174 .120 .095 .923 .903 .232 .170
guided GLIDE diffusion model. The image forgery localiza- PSCC [62] .020 .013 .010 .007 .609 .547 .000 .000
tion scores (AUC and MCC) are reported in Table 7. Our MVSS-Net++ [19] .067 .264 .035 .152 .741 889 .063 .261
method PIM consistently achieves the best AUC and MCC
CAT-NET [53] .089 .178 .068 .141 .588 .787 .088 .185
performances across the Autosplice 100, Autosplice 90, and
EVP [61] .028 .164 .014 .089 .916 .923 .074 .196
CocoGlide datasets. The SOTA method, TruFor, benefits from
its Noiseprint++ extractor trained on extensive extra data, TruFor [35] .234 .221 .133 .124 .891 .875 .249 .240
achieving the highest scores on the low-quality Autosplice PIM .357 .288 .217 .168 .953 .914 .351 .300
75 dataset. Nonetheless, PIM exhibits superior average AUC
and MCC across all advanced AIGC datasets.
4.9 Robustness Evaluation Results
This study introduced six common image perturbations,
Due to uncontrollable variables in real-world applications brightness, contrast, darkening, dithering, pink noise, and
(e.g., black-box compression via social media platforms), JPEG2000 compression, on the CASIAv1+ [21] dataset, which
detectors may encounter unseen image perturbations, result- was unknown during the training process. We further set
ing in significant performance drops. Although regular data nine severity levels for each perturbation type to accom-
augmentations have been considered during the training modate various environmental variations. We showcase
process, it is challenging to foresee all perturbation types examples of raw images and the corresponding perturbed
under the deployment circumstance. versions in the Appendix. The pixel-level AUC detection
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 12

TABLE 7: Image manipulation localization performance on unseen advanced manipulation techniques.

AutoSplice 100 AutoSplice 90 AutoSplice 75 CocoGlide Avgerage

Method Venue
AUC MCC AUC MCC AUC MCC AUC MCC AUC MCC
FCN [64] CVPR15 .681 .150 .619 .078 .589 .048 .618 .079 .627 .089
U-Net [80] MICCAI15 .616 .072 .585 .047 .570 .034 .578 .047 .587 .050
DeepLabv3 [13] TPAMI18 .864 .223 .812 .153 .759 .099 .730 .103 .791 .145
MFCN [82] JVCIP18 .565 .072 .547 .049 .534 .035 .551 .052 .549 .052
RRU-Net [8] CVPRW19 .781 .159 .737 .114 .714 .083 .620 .051 .713 .102
MantraNet [95] CVPR19 .664 .189 .626 .160 .660 .177 .806 .190 .689 .179
HPFCN [55] ICCV19 .646 .092 .633 .082 .622 .067 .586 .048 .622 .072
H-LSTM [5] TIP19 .639 .162 .643 .162 .634 .145 .643 .137 .640 .152
SPAN [41] ECCV20 .645 .020 .549 .005 .566 .007 .776 .198 .634 .058
ViT-B [22] ICLR21 .662 .131 .658 .126 .651 .118 .631 .105 .651 .120
Swin-ViT [63] ICCV21 .700 .233 .590 .072 .570 .046 .648 .126 .627 .119
PSCC [62] TCSVT22 .749 .275 .657 .195 .630 .156 .566 .051 .651 .169
MVSS-Net++ [19] TPAMI22 .836 .280 .751 .101 .714 .054 .819 .309 .780 .186
CAT-NET [53] IJCV22 .887 .578 .720 .301 .607 .163 .587 .133 .700 .294
EVP [61] CVPR23 .762 .226 .697 .124 .637 .078 .686 .114 .696 .136
TruFor [35] CVPR23 .827 .382 .818 .358 .820 .367 .757 .253 .806 .340
PIM Ours .940 .715 .852 .402 .729 .151 .817 .372 .835 .410

Fig. 8: Robustness evaluation results (AUC) on six unseen perturbation types: (a). Brightness, (b). Contrast, (c). Darkening,
(d). Dithering, (e). Pink noise, (f). JPEG2000. The x-axis indicates the perturbation severity level.

scores are illustrated in Fig. 8. The x dimension indicates the evaluations.

severity levels, where Severity ‘0’ indicates no perturbation
applied. We can observe that all detection models suffer
4.10 Qualitative Experimental Results
certain performance drops due to these unforeseen pertur-
bation types. The proposed method consistently achieves In Fig. 9, we qualitatively evaluate the image manipulation
the best AUC across different perturbation levels on all localization performance across 11 unseen test sets, where
unseen perturbation types, demonstrating the robustness the leftmost three columns show the input images, the
of our method. As most image perturbations encountered corresponding ground-truth masks, and the predicted results
in real-world scenarios are uniformly applied to images, the of our method. Besides, we show the forgery localization
pixel dependencies within unaltered images and the pixel results of SOTA methods in the right 11 columns. Our
inconsistencies within manipulated images remain consistent. method can accurately localize the manipulated regions for
Therefore, our proposed method continues to exhibit the forgery images with diverse image quality, scenes, occlusions,
best forgery localization performance in such robustness and illumination conditions. Our localization results are
superior to previous methods, regardless of whether the
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 13

TABLE 8: Ablation study for image manipulation localization.

Setting BB BDD RDA PIDA CDC RDC LWM LC GPDE LR AVG. F1 AVG. IoU
1 ✓ - - - - - - - - - .211 .171
2 ✓ ✓ - - - - - - - - .220 .178
3 ✓ ✓ ✓ - - - - - - - .233 .190
4 ✓ ✓ - ✓ - - - - - - .260 .209
5 ✓ ✓ ✓ ✓ - - - - - - .283 .237
6 ✓ ✓ ✓ ✓ ✓ - - - - - .304 .252
7 ✓ ✓ ✓ ✓ - ✓ - - - - .308 .258
8 ✓ ✓ ✓ ✓ ✓ ✓ - - - - .312 .262
9 ✓ ✓ ✓ ✓ ✓ ✓ ✓ - - - .317 .271
10 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ - - .323 .269
11 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ - .330 .272
12 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ .333 .275

forgery regions are relatively substantial (e.g., Columbia and localization methods according to our experimental results
WildWeb) or subtle (e.g., DEF-12k and Korus) in fake images. in Sec. 4.5-4.8. We observe that PIM (Ours), MVSS-Net++,
The boundaries of predicted results are much sharper and TruFor can successfully predict manipulated regions in
for the proposed method than in previous arts. This can be the unshuffled images.
attributed to the global pixel dependency modeling module Next, we present prediction results on shuffled images in
and local pixel difference convolution module that can Fig. 10 (h)-(n). These randomly shuffled images inherently
highlight pixel inconsistency in forgery boundary regions. contain limited semantic information. In column (k), the
As the predicted results in CASIAv1+ and In-the-Wild show, proposed method effectively localizes the forgery regions
the proposed method can successfully localize extremely within each patch. Column (l) showcases the predicted
subtle forgery details. This can be attributed to the local pixel sharp boundaries of forgery patches. In contrast, forgery
difference convolution module, which allows the model to prediction results of MVSS-Net++ in column (m) reveal
capture local pixel inconsistency artifacts. Our method main- struggling performance, marked by numerous false alarms
tains accurate localization performance for more challenging and undetected forgery regions. While TruFor aims to capture
images, such as the one in the IMD row that contains multiple generic noise artifacts in forgery images, column (n) shows
tiny forgery regions. Finally, the proposed method results in that it still performs poorly in such a challenging setting. The
fewer false alarms, as evidenced in the predictions of COVER localization results of shuffled images further demonstrate
and NIST. This characteristic can ensure a more dependable the superiority of our method. Therefore, we conclude the
forgery detection for real-world deployment. Compared to proposed method focuses more on pixel-level artifacts than
TruFor, our method PIM exhibits more accurate forgery semantic-level forgery traces.
localization, fewer false alarms, sharper forgery boundaries,
and superior capability in capturing subtle forgery traces. 4.12 Ablation Experiments
We provide more visualization results in the Appendix.
The qualitative experimental results demonstrate that In this subsection, we present comprehensive ablation studies
the proposed formulation effectively deals with various to evaluate the effectiveness of the components designed
challenging forgery situations. This is primarily attributed to in our framework. Table. 8 shows the average forgery
the dedicated module designs to extract inherent pixel-level localization performance in the cross-dataset evaluations,
forgery fingerprints. where ‘✓’ denotes the used component.
BB indicates the ensemble of the transformer backbone
and the mask decoder. BDD denotes the utilization of the
4.11 Visualization Results on Shuffled Images boundary decoder. RDA and PIDA represent the regular data
To demonstrate the effectiveness of the proposed model in augmentation and the proposed Pixel-Inconsistency Data
capturing pixel inconsistency artifacts for forgery localization, Augmentation. CDC, RDC, and LWM stand for central pixel
we split the input image into 3×3 patches and shuffle them difference convolution, radial pixel difference convolution,
randomly. This random shuffling effectively suppresses the and the learning to weight module, respectively. LC indicates
semantic information within the input images and allows the usage of the compactness loss. GPDE and LR represent
us to assess whether our model can still accurately localize the designed Global Pixel Dependency Encoder and the
the forgery regions. We present results for unshuffled and reconstruction loss, respectively.
shuffled images in Fig. 10, denoted as (a)-(g) and (h)-(n), From Table. 8, we can observe that using a boundary
respectively. Columns (a)-(c) show the original input images, decoder can boost the forgery localization performance. A
their respective mask, and boundary labels. Columns (d)-(g) comparison between Setting 3 and 4 highlights the supe-
present our forgery localization maps, boundary predictions, riority of the proposed PIDA over RDA, suggesting that
localization results of MVSS-Net++, and localization results PIDA encourages the detector to focus on more general
of TruFor. In this evaluation, we select the MVSS-Net++ artifacts. Intuitively, the combination of RDA and PIDA in
and TruFor as the baselines as they are the SOTA forgery Setting 5 is expected to enhance pixel-level forgery detection
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 14

Fig. 9: Forgery localization results on the 11 unseen test sets. The three left columns show the input images, corresponding
ground-truth, and the localization results of our method. The right 11 columns present the results of SOTA methods.

performance, primarily because the model has been fed In summary, the ablation studies exhibit the critical role
more data. The CDC and RDC modules (Settings 6-8) effec- of the designed components in our framework. The ensemble
tively capture local pixel difference features, contributing of these components jointly enhances the forgery localization
to enhanced localization results. Furthermore, Setting 9 performance.
demonstrates the effectiveness of the LWM, which learns
the weights more smartly and performs a better feature 5 C ONCLUSIONS AND F UTURE W ORK
fusion. Using the compactness loss LC in Setting 10 produces
This paper presented a generalized and robust image manip-
more compact outputs, improving the final performance.
ulation localization model by capturing pixel inconsistency in
The use of GPDE in Setting 11 successfully models global
forgery images. The method is underpinned by a two-stream
pixel dependency, thereby achieving superior image forgery
pixel dependency modeling framework for image forgery
localization performance. Compared to Setting 11, Setting
localization. It incorporates a novel masked self-attention
12 adopts the reconstruction loss LR to further enhance
mechanism to model the global pixel dependencies within
global pixel dependency modeling while revealing pixel
input images effectively. Additionally, two customized convo-
inconsistency artifacts in manipulated images. This con-
lutional modules, the Central Difference Convolution (CDC)
tributes significantly to the overall localization performance.
and the Radial Difference Convolution (RDC), better capture
The detailed ablation experimental results across all testing
pixel inconsistency artifacts within local regions. We find that
datasets, the experiments regarding the impacts of multi-
modeling pixel interrelations can effectively mine intrinsic
head self-attention, and the visualization ablation results can
forgery clues. To enhance the overall performance, Learning-
be found in the Appendix.
to-Weight Modules (LWM) complementarily combines global
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 15

Fig. 10: Visualization results on shuffled images. (a) Input unshuffled images. (b) Forgery localization labels. (c) Forgery
boundary labels. (d) Our forgery localization results. (e) Our boundary prediction results. (f) MVSS-Net++ forgery localization
results. (g) TruFor forgery localization results. (h) Input shuffled images. (i) Shuffled forgery localization labels. (j) Shuffled
forgery boundary labels. (k) Our forgery localization results on shuffled images. (l) Our boundary prediction results on
shuffled images. (m) MVSS-Net++ forgery localization results on shuffled images. (n) TruFor forgery localization results on
shuffled images.

and local features. The usage of the dynamic weighting [5]Jawadul H Bappy, Cody Simons, Lakshmanan Nataraj, BS Man-
scheme can lead to a better feature fusion, contributing to a junath, and Amit K Roy-Chowdhury. Hybrid lstm and encoder–
decoder architecture for detection of image forgeries. IEEE
more robust and generalized image forgery localization. Transactions on Image Processing, 28(7):3286–3300, 2019.
Furthermore, a novel Pixel-Inconsistency Data Augmen- [6] Mauro Barni, Andrea Costanzo, and Lara Sabatini. Identification
tation (PIDA) that exclusively employs pristine images to of cut & paste tampering by means of double-jpeg detection and
generate augmented forgery samples, guides the focus on image segmentation. In Proceedings of 2010 IEEE International
Symposium on Circuits and Systems, pages 1687–1690. IEEE, 2010.
pixel-level artifacts. The proposed PIDA strategy can shed
[7] Marcelo Bertalmio, Andrea L Bertozzi, and Guillermo Sapiro.
light on improving the generalization for future forensics Navier-stokes, fluid dynamics, and image and video inpainting.
research. Extensive experimental results demonstrated the In Proceedings of the 2001 IEEE Computer Society Conference on
state-of-the-art performance of the proposed framework in Computer Vision and Pattern Recognition. CVPR 2001, volume 1,
pages I–I. IEEE, 2001.
image manipulation detection and localization, both in gen-
[8] Xiuli Bi, Yang Wei, Bin Xiao, and Weisheng Li. Rru-net: The
eralization and robustness evaluations. Our designed model ringed residual u-net for image splicing forgery detection. In
also exhibits outstanding performance on unseen, advanced, Proceedings of the IEEE/CVF Conference on Computer Vision and
and sophisticated manipulation images, underscoring its Pattern Recognition Workshops, pages 0–0, 2019.
potential in challenging real-world scenarios. The ablation [9] Tiziano Bianchi and Alessandro Piva. Image forgery localization
via block-grained analysis of jpeg artifacts. IEEE Transactions on
studies further validated the effectiveness of the designed Information Forensics and Security, 7(3):1003–1017, 2012.
components. [10] Hong Cao and Alex C Kot. Accurate detection of demosaicing
While our method is robust against unseen image pertur- regularity for digital image forensics. IEEE Transactions on
bations, it remains susceptible to recapturing attacks. This Information Forensics and Security, 4(4):899–910, 2009.
[11] Hong Cao and Alex C Kot. Accurate detection of demosaicing reg-
vulnerability stems from the framework’s primary objective: ularity from output images. In 2009 IEEE International Symposium
to identify pixel inconsistency artifacts resulting from the dis- on Circuits and Systems, pages 497–500. IEEE, 2009.
ruption of CFA regularity during the manipulation process. [12] Tiago Carvalho, Fabio A Faria, Helio Pedrini, Ricardo da S Torres,
Recapturing operations reintroduce the pixel dependencies and Anderson Rocha. Illuminant-based transformed spaces for
image forensics. IEEE transactions on information forensics and
initially constructed during the demosaicing process, con- security, 11(4):720–733, 2015.
cealing the pixel inconsistency artifacts and leading to failed [13] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin
forgery detection. In future research, developing an effective Murphy, and Alan L Yuille. Deeplab: Semantic image segmenta-
recapturing detection module becomes a crucial research tion with deep convolutional nets, atrous convolution, and fully
connected crfs. IEEE transactions on pattern analysis and machine
direction to ensure more secure manipulation detection. intelligence, 40(4):834–848, 2017.
R EFERENCES [14] Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel.
[1] Ieee ifs-tc image forensics challenge dataset. https: Pixelsnail: An improved autoregressive generative model. In
//[Link]/newsletter/2013/06/ International Conference on Machine Learning, pages 864–872. PMLR,
ifs-tc-image-forensics-challenge, 2013. 2018.
[2] Ai camera and its advantages. [Link] [15] Yi-Lei Chen and Chiou-Ting Hsu. Detecting recompression of
what-is-an-ai-camera, 2023. jpeg images via periodicity analysis of compression artifacts for
[3] Available. [EB/OL], 2023. https:// tampering detection. IEEE Transactions on Information Forensics and
[Link]/news/article-2107109/ Security, 6(2):396–406, 2011.
[16] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever.
Iconic-Abraham-Lincoln-portrait-revealed-TWO-pictures-stitched-together.
html. Generating long sequences with sparse transformers. arXiv
[4] Quentin Bammey, Rafael Grompone von Gioi, and Jean-Michel preprint arXiv:1904.10509, 2019.
Morel. An adaptive neural network for unsupervised mosaic [17] Davide Cozzolino, Giovanni Poggi, and Luisa Verdoliva. Splice-
consistency analysis in image forensics. In Proceedings of the buster: A new blind image splicing detector. In 2015 IEEE
IEEE/CVF Conference on Computer Vision and Pattern Recognition, International Workshop on Information Forensics and Security (WIFS),
pages 14194–14204, 2020. pages 1–6. IEEE, 2015.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 16

[18] Davide Cozzolino and Luisa Verdoliva. Noiseprint: A cnn- rank approximation. IEEE transactions on visualization and computer
based camera model fingerprint. IEEE Transactions on Information graphics, 24(6):2023–2036, 2017.
Forensics and Security, 15:144–159, 2019. [37] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár,
[19] Chengbo Dong, Xinru Chen, Ruohan Hu, Juan Cao, and Xirong Li. and Ross Girshick. Masked autoencoders are scalable vision
Mvss-net: Multi-view multi-scale supervised networks for image learners. In Proceedings of the IEEE/CVF Conference on Computer
manipulation detection. IEEE Transactions on Pattern Analysis and Vision and Pattern Recognition, pages 16000–16009, 2022.
Machine Intelligence, 2022. [38] Jan Herling and Wolfgang Broll. High-quality real-time video
[20] Jing Dong, Wei Wang, and Tieniu Tan. CASIA image tampering inpaintingwith pixmix. IEEE Transactions on Visualization and
detection evaluation database. In 2013 IEEE China Summit and Computer Graphics, 20(6):866–879, 2014.
International Conference on Signal and Information Processing. IEEE, [39] John S Ho, Oscar C Au, Jiantao Zhou, and Yuanfang Guo. Inter-
July 2013. channel demosaicking traces for digital image forensics. In 2010
[21] Jing Dong, Wei Wang, and Tieniu Tan. Casia image tampering IEEE International Conference on Multimedia and Expo, pages 1475–
detection evaluation database. In 2013 IEEE China Summit and 1480. IEEE, 2010.
International Conference on Signal and Information Processing, pages [40] J Hsu and SF Chang. Columbia uncompressed image splicing
422–426. IEEE, 2013. detection evaluation dataset. Columbia DVMM Research Lab, 2006.
[22] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk [41] Xuefeng Hu, Zhihan Zhang, Zhenye Jiang, Syomantak Chaudhuri,
Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa De- Zhenheng Yang, and Ram Nevatia. Span: Spatial pyramid atten-
hghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob tion network for image manipulation localization. In Computer
Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August
Transformers for image recognition at scale. ICLR, 2021. 23–28, 2020, Proceedings, Part XXI 16, pages 312–328. Springer,
[23] Jiayuan Fan, Hong Cao, and Alex C Kot. Estimating exif 2020.
parameters based on noise features for image manipulation [42] Jia-Bin Huang, Sing Bing Kang, Narendra Ahuja, and Johannes
detection. IEEE Transactions on Information Forensics and Security, Kopf. Image completion using planar structure guidance. ACM
8(4):608–618, 2013. Transactions on graphics (TOG), 33(4):1–10, 2014.
[24] Zhigang Fan and Ricardo L De Queiroz. Identification of bitmap [43] Minyoung Huh, Andrew Liu, Andrew Owens, and Alexei A
compression history: Jpeg detection and quantizer estimation. Efros. Fighting fake news: Image splice detection via learned self-
IEEE Transactions on Image Processing, 12(2):230–235, 2003. consistency. In Proceedings of the European conference on computer
[25] Hany Farid. Exposing digital forgeries from jpeg ghosts. IEEE vision (ECCV), pages 101–117, 2018.
transactions on information forensics and security, 4(1):154–160, 2009. [44] Chryssanthi Iakovidou, Markos Zampoglou, Symeon Papadopou-
los, and Yiannis Kompatsiaris. Content-aware detection of jpeg
[26] Pasquale Ferrara, Tiziano Bianchi, Alessia De Rosa, and Alessan-
grid inconsistencies for intuitive image forensics. Journal of Visual
dro Piva. Image forgery localization via fine-grained analysis of
Communication and Image Representation, 54:155–170, 2018.
cfa artifacts. IEEE Transactions on Information Forensics and Security,
[45] Ajay Jain, Pieter Abbeel, and Deepak Pathak. Locally masked con-
7(5):1566–1577, 2012.
volution for autoregressive models. In Conference on Uncertainty
[27] Anselmo Ferreira, Siovani C Felipussi, Carlos Alfaro, Pablo in Artificial Intelligence, pages 1358–1367. PMLR, 2020.
Fonseca, John E Vargas-Munoz, Jefersson A Dos Santos, and
[46] Shan Jia, Mingzhen Huang, Zhou Zhou, Yan Ju, Jialing Cai, and
Anderson Rocha. Behavior knowledge space-based fusion for
Siwei Lyu. Autosplice: A text-prompt manipulated image dataset
copy–move forgery detection. IEEE Transactions on Image Process-
for media forensics. In Proceedings of the IEEE/CVF Conference on
ing, 25(10):4729–4742, 2016.
Computer Vision and Pattern Recognition, pages 893–903, 2023.
[28] Dongdong Fu, Yun Q Shi, and Wei Su. A generalized benford’s [47] Micah K Johnson and Hany Farid. Exposing digital forgeries
law for jpeg coefficients and its applications in image forensics. In through chromatic aberration. In Proceedings of the 8th workshop on
Security, Steganography, and Watermarking of Multimedia Contents Multimedia and security, pages 48–55, 2006.
IX, volume 6505, pages 574–584. SPIE, 2007.
[48] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic
[29] Huazhu Fu and Xiaochun Cao. Forgery authentication in extreme optimization. arXiv preprint arXiv:1412.6980, 2014.
wide-angle lens using distortion cue and fake saliency map. IEEE [49] Michihiro Kobayashi, Takahiro Okabe, and Yoichi Sato. Detecting
Transactions on Information Forensics and Security, 7(4):1301–1314, forgery from static-scene video based on inconsistency in noise
2012. level functions. IEEE Transactions on Information Forensics and
[30] Andrew C Gallagher and Tsuhan Chen. Image authentication by Security, 5(4):883–892, 2010.
detecting traces of demosaicing. In 2008 IEEE Computer Society [50] Alexander Kolesnikov and Christoph H Lampert. Pixelcnn
Conference on Computer Vision and Pattern Recognition Workshops, models with auxiliary variables for natural image modeling.
pages 1–8. IEEE, 2008. In International Conference on Machine Learning, pages 1905–1914.
[31] Zan Gao, Shenghao Chen, Yangyang Guo, Weili Guan, Jie Nie, PMLR, 2017.
and Anan Liu. Generic image manipulation localization through [51] Chenqi Kong, Baoliang Chen, Haoliang Li, Shiqi Wang, Anderson
the lens of multi-scale spatial inconsistence. In Proceedings of the Rocha, and Sam Kwong. Detect and locate: Exposing face manip-
30th ACM International Conference on Multimedia, pages 6146–6154, ulation by semantic-and noise-level telltales. IEEE Transactions on
2022. Information Forensics and Security, 17:1741–1756, 2022.
[32] Mathieu Germain, Karol Gregor, Iain Murray, and Hugo [52] Paweł Korus and Jiwu Huang. Evaluation of random field models
Larochelle. Made: Masked autoencoder for distribution estimation. in multi-modal unsupervised tampering localization. In 2016 IEEE
In International conference on machine learning, pages 881–889. international workshop on information forensics and security (WIFS),
PMLR, 2015. pages 1–6. IEEE, 2016.
[33] Thomas Gloe, Karsten Borowka, and Antje Winkler. Efficient [53] Myung-Joon Kwon, Seung-Hun Nam, In-Jae Yu, Heung-Kyu Lee,
estimation and large-scale evaluation of lateral chromatic aberra- and Changick Kim. Learning jpeg compression artifacts for image
tion for digital image forensics. In Media Forensics and Security II, manipulation detection and localization. International Journal of
volume 7541, pages 62–74. SPIE, 2010. Computer Vision, 130(8):1875–1895, 2022.
[34] Haiying Guan, Mark Kozak, Eric Robertson, Yooyoung Lee, [54] Hugo Larochelle and Iain Murray. The neural autoregressive
Amy N Yates, Andrew Delgado, Daniel Zhou, Timothee distribution estimator. In Proceedings of the fourteenth international
Kheyrkhah, Jeff Smith, and Jonathan Fiscus. Mfc datasets: Large- conference on artificial intelligence and statistics, pages 29–37. JMLR
scale benchmark datasets for media forensic challenge evaluation. Workshop and Conference Proceedings, 2011.
In 2019 IEEE Winter Applications of Computer Vision Workshops [55] Haodong Li and Jiwu Huang. Localization of deep inpainting
(WACVW), pages 63–72. IEEE, 2019. using high-pass fully convolutional network. In proceedings of
[35] Fabrizio Guillaro, Davide Cozzolino, Avneesh Sud, Nicholas the IEEE/CVF international conference on computer vision, pages
Dufour, and Luisa Verdoliva. Trufor: Leveraging all-round clues 8301–8310, 2019.
for trustworthy image forgery detection and localization. In [56] Haoliang Li, Alex C Kot, and Leida Li. Color space identification
Proceedings of the IEEE/CVF Conference on Computer Vision and from single images. In 2016 IEEE International Symposium on
Pattern Recognition, pages 20606–20615, 2023. Circuits and Systems (ISCAS), pages 1774–1777. IEEE, 2016.
[36] Qiang Guo, Shanshan Gao, Xiaofeng Zhang, Yilong Yin, and [57] Sihao Lin, Pumeng Lyu, Dongrui Liu, Tao Tang, Xiaodan Liang,
Caiming Zhang. Patch-based image inpainting via two-stage low Andy Song, and Xiaojun Chang. Mlp can be a good transformer
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 17

learner. In Proceedings of the IEEE/CVF Conference on Computer [77] Alin C Popescu and Hany Farid. Exposing digital forgeries in
Vision and Pattern Recognition, pages 19489–19498, 2024. color filter array interpolated images. IEEE Transactions on Signal
[58] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Processing, 53(10):3948–3959, 2005.
Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. [78] Shuren Qi, Yushu Zhang, Chao Wang, Jiantao Zhou, and Xiaochun
Microsoft coco: Common objects in context. In Computer Vision– Cao. A principled design of image representation: Towards
ECCV 2014: 13th European Conference, Zurich, Switzerland, September forensic tasks. IEEE Transactions on Pattern Analysis and Machine
6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014. Intelligence, 45(5):5337–5354, 2022.
[59] Xun Lin, Shuai Wang, Jiahao Deng, Ying Fu, Xiao Bai, Xinlei Chen, [79] Yuan Rao, Jiangqun Ni, Weizhe Zhang, and Jiwu Huang. Towards
Xiaolei Qu, and Wenzhong Tang. Image manipulation detection by jpeg-resistant image forgery detection and localization via self-
multiple tampering traces and edge artifact enhancement. Pattern supervised domain adaptation. IEEE Transactions on Pattern
Recognition, 133:109026, 2023. Analysis and Machine Intelligence, 2022.
[60] Li Liu, Lingjun Zhao, Yunli Long, Gangyao Kuang, and Paul [80] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net:
Fieguth. Extended local binary patterns for texture classification. Convolutional networks for biomedical image segmentation.
Image and Vision Computing, 30(2):86–99, 2012. In Medical Image Computing and Computer-Assisted Intervention–
[61] Weihuang Liu, Xi Shen, Chi-Man Pun, and Xiaodong Cun. Explicit MICCAI 2015: 18th International Conference, Munich, Germany,
visual prompting for low-level structure segmentations. In CPVR, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer,
2023. 2015.
[62] Xiaohong Liu, Yaojie Liu, Jun Chen, and Xiaoming Liu. Pscc- [81] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma.
net: Progressive spatio-channel correlation network for image Pixelcnn++: Improving the pixelcnn with discretized logistic
manipulation detection and localization. IEEE Transactions on mixture likelihood and other modifications. arXiv preprint
Circuits and Systems for Video Technology, 32(11):7505–7517, 2022. arXiv:1701.05517, 2017.
[63] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, [82] Ronald Salloum, Yuzhuo Ren, and C-C Jay Kuo. Image splicing
Stephen Lin, and Baining Guo. Swin transformer: Hierarchical localization using a multi-task fully convolutional network (mfcn).
vision transformer using shifted windows. In Proceedings of the Journal of Visual Communication and Image Representation, 51:201–
IEEE/CVF international conference on computer vision, pages 10012– 209, 2018.
10022, 2021. [83] Zhuo Su, Wenzhe Liu, Zitong Yu, Dewen Hu, Qing Liao, Qi Tian,
[64] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully Matti Pietikäinen, and Li Liu. Pixel difference networks for
convolutional networks for semantic segmentation. In Proceedings efficient edge detection. In Proceedings of the IEEE/CVF international
of the IEEE conference on computer vision and pattern recognition, conference on computer vision, pages 5117–5127, 2021.
pages 3431–3440, 2015. [84] Alexandru Telea. An image inpainting technique based on the
[65] Siwei Lyu, Xunyu Pan, and Xing Zhang. Exposing region splicing fast marching method. Journal of graphics tools, 9(1):23–34, 2004.
forgeries with blind local noise estimation. International journal of [85] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas
computer vision, 110:202–221, 2014. Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas
[66] Babak Mahdian and Stanislav Saic. Using noise inconsistencies Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An
for blind image forensics. Image and Vision Computing, 27(10):1497– all-mlp architecture for vision. Advances in neural information
1503, 2009. processing systems, 34:24261–24272, 2021.
[67] Gaël Mahfoudi, Badr Tajini, Florent Retraint, Frederic Morain- [86] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol
Nicolier, Jean Luc Dugelay, and PIC Marc. Defacto: Image and Vinyals, Alex Graves, et al. Conditional image generation with
face manipulation dataset. In 2019 27th European Signal Processing pixelcnn decoders. Advances in neural information processing systems,
Conference (EUSIPCO), pages 1–5. IEEE, 2019. 29, 2016.
[68] Owen Mayer and Matthew C Stamm. Accurate and efficient [87] Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu.
image forgery detection using lateral chromatic aberration. IEEE Pixel recurrent neural networks. In International conference on
Transactions on information forensics and security, 13(7):1762–1777, machine learning, pages 1747–1756. PMLR, 2016.
2018. [88] Luisa Verdoliva. Media forensics and deepfakes: an overview.
[69] Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Z Qureshi, and IEEE Journal of Selected Topics in Signal Processing, 14(5):910–932,
Mehran Ebrahimi. Edgeconnect: Generative image inpainting 2020.
with adversarial edge learning. arXiv preprint arXiv:1901.00212, [89] James Ze Wang, Jia Li, and Gio Wiederhold. Simplicity: Semantics-
2019. sensitive integrated matching for picture libraries. IEEE Trans-
[70] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, actions on pattern analysis and machine intelligence, 23(9):947–963,
Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. 2001.
Glide: Towards photorealistic image generation and editing with [90] Junke Wang, Zuxuan Wu, Jingjing Chen, Xintong Han, Abhinav
text-guided diffusion models. arXiv preprint arXiv:2112.10741, Shrivastava, Ser-Nam Lim, and Yu-Gang Jiang. Objectformer for
2021. image manipulation detection and localization. In Proceedings of
[71] Adam Novozamsky, Babak Mahdian, and Stanislav Saic. Imd2020: the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
a large-scale annotated dataset tailored for detecting manipulated pages 2364–2373, 2022.
images. In Proceedings of the IEEE/CVF Winter Conference on [91] Menglu Wang, Xueyang Fu, Jiawei Liu, and Zheng-Jun Zha. Jpeg
Applications of Computer Vision Workshops, pages 71–80, 2020. compression-aware image forgery localization. In Proceedings of the
[72] Yassine Ouali, Céline Hudelot, and Myriam Tami. Autoregressive 30th ACM International Conference on Multimedia, pages 5871–5879,
unsupervised image segmentation. In Computer Vision–ECCV 2022.
2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, [92] Bihan Wen, Ye Zhu, Ramanathan Subramanian, Tian-Tsong Ng,
Proceedings, Part VII 16, pages 142–158. Springer, 2020. Xuanjing Shen, and Stefan Winkler. Coverage—a novel database
[73] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, for copy-move forgery detection. In 2016 IEEE international
Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. conference on image processing (ICIP), pages 161–165. IEEE, 2016.
In International conference on machine learning, pages 4055–4064. [93] Haiwei Wu and Jiantao Zhou. Iid-net: Image inpainting detection
PMLR, 2018. network via neural architecture search and attention. IEEE
[74] Cecilia Pasquini, Giulia Boato, and Fernando Pérez-González. Sta- Transactions on Circuits and Systems for Video Technology, 32(3):1172–
tistical detection of jpeg traces in digital images in uncompressed 1185, 2021.
formats. IEEE Transactions on Information Forensics and Security, [94] Haiwei Wu, Jiantao Zhou, and Yuanman Li. Deep generative
12(12):2890–2905, 2017. model for image inpainting with local binary pattern learning and
[75] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James spatial attention. IEEE Transactions on Multimedia, 24:4016–4027,
Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia 2021.
Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, [95] Yue Wu, Wael AbdAlmageed, and Premkumar Natarajan. Mantra-
high-performance deep learning library. In NeurIPS, 2019. net: Manipulation tracing network for detection and localization
[76] Alin C Popescu and Hany Farid. Statistical tools for digital of image forgeries with anomalous features. In Proceedings of the
forensics. In International workshop on information hiding, pages IEEE/CVF Conference on Computer Vision and Pattern Recognition,
128–147. Springer, 2004. pages 9543–9552, 2019.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 18

[96] Zhaoyi Yan, Xiaoming Li, Mu Li, Wangmeng Zuo, and Shiguang Shiqi Wang received the B.S. degree in computer
Shan. Shift-net: Image inpainting via deep feature rearrangement. science from the Harbin Institute of Technology
In Proceedings of the European conference on computer vision (ECCV), in 2008 and the Ph.D. degree in computer ap-
pages 1–17, 2018. plication technology from Peking University in
[97] Ido Yerushalmy and Hagit Hel-Or. Digital image forgery detection 2014. From 2014 to 2016, he was a Post-Doctoral
based on lens and sensor aberration. International journal of Fellow with the Department of Electrical and
computer vision, 92:71–91, 2011. Computer Engineering, University of Waterloo,
[98] Qichao Ying, Hang Zhou, Zhenxing Qian, Sheng Li, and Xinpeng Waterloo, ON, Canada. From 2016 to 2017, he
Zhang. Learning to immunize images for tamper localization and was a Research Fellow with the Rapid-Rich
self-recovery. IEEE Transactions on Pattern Analysis and Machine Object Search Laboratory, Nanyang Technolog-
Intelligence, 2023. ical University, Singapore. He is currently an
[99] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Associate Professor with the Department of Computer Science, City
Thomas S Huang. Generative image inpainting with contextual University of Hong Kong. He has proposed over 40 technical proposals to
attention. In Proceedings of the IEEE conference on computer vision ISO/MPEG, ITU-T, and AVS standards, and authored/coauthored more
and pattern recognition, pages 5505–5514, 2018. than 200 refereed journal articles/conference papers. He received the
[100] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Best Paper Award from IEEE VCIP 2019, ICME 2019, IEEE Multimedia
Thomas S Huang. Free-form image inpainting with gated 2018, and PCM 2017 and is the coauthor of an article that received
convolution. In Proceedings of the IEEE/CVF international conference the Best Student Paper Award in the IEEE ICIP 2018. His research
on computer vision, pages 4471–4480, 2019. interests include video compression, image/video quality assessment,
[101] Tao Yu, Zongyu Guo, Xin Jin, Shilin Wu, Zhibo Chen, Weiping and image/video search and analysis.
Li, Zhizheng Zhang, and Sen Liu. Region normalization for
image inpainting. In Proceedings of the AAAI conference on artificial Haoliang Li received the B.S. degree in commu-
intelligence, volume 34, pages 12733–12740, 2020. nication engineering from University of Electronic
[102] Zitong Yu, Chenxu Zhao, Zezheng Wang, Yunxiao Qin, Zhuo Su, Science and Technology of China (UESTC) in
Xiaobai Li, Feng Zhou, and Guoying Zhao. Searching central 2013, and his Ph.D. degree from Nanyang Tech-
difference convolutional networks for face anti-spoofing. In nological University (NTU), Singapore in 2018.
Proceedings of the IEEE/CVF Conference on Computer Vision and He is currently an assistant professor in Depart-
Pattern Recognition, pages 5295–5305, 2020. ment of Electrical Engineering, City University of
[103] Markos Zampoglou, Symeon Papadopoulos, and Yiannis Kompat- Hong Kong. His research mainly focuses on AI
siaris. Detecting image splicing in the wild (web). In 2015 IEEE security, multimedia forensics and transfer learn-
International Conference on Multimedia & Expo Workshops (ICMEW), ing. His research works appear in international
pages 1–6. IEEE, 2015. journals/conferences such as TPAMI, IJCV, TIFS,
[104] Peng Zhou, Xintong Han, Vlad I Morariu, and Larry S Davis. NeurIPS, CVPR and AAAI. He received the Wallenberg-NTU presidential
Learning rich features for image manipulation detection. In postdoc fellowship in 2019, doctoral innovation award in 2019, and VCIP
Proceedings of the IEEE conference on computer vision and pattern best paper award in 2020.
recognition, pages 1053–1061, 2018.
[105] Peiyu Zhuang, Haodong Li, Shunquan Tan, Bin Li, and Jiwu
Huang. Image tampering localization using a dense fully convo-
lutional network. IEEE Transactions on Information Forensics and
Security, 16:2986–2999, 2021. Anderson Rocha received his Ph.D. degree
[106] Long Zhuo, Shunquan Tan, Bin Li, and Jiwu Huang. Self- in computer science. He is a full professor of
adversarial training incorporating forgery attention for image artificial intelligence and digital forensics at the
forgery localization. IEEE Transactions on Information Forensics and Institute of Computing, University of Campinas,
Security, 17:819–834, 2022. Campinas 13083-852, Brazil, where he is the
coordinator of the Artificial Intelligence Lab. A
Microsoft and Google Faculty Fellow, he is a for-
mer chair of the IEEE Information Forensics and
Security Technical Committee (2019–2020) and
an affiliated member of the Brazilian Academy of
Chenqi Kong received the B.S. and M.S. de- Sciences and the Brazilian Academy of Forensics
grees in the College of Science and the College Sciences. His research interests include artificial intelligence, digital
of Electrical Engineering and Automation, Harbin forensics, and reasoning for complex data. He is a Fellow of IEEE.
Institute of Technology, Harbin, China, in 2017
and 2019, respectively. He received the Ph.D.
degree in the Department of Computer Science,
City University of Hong Kong, Hong Kong, China
(Hong Kong SAR) in 2023. He is currently a Prof. Alex Kot has been with the Nanyang
research fellow in the School of Electrical and Technological University, Singapore since 1991.
Electronic Engineering, Nanyang Technological He was Head of the Division of Information
University, Singapore. He is a recipient of Na- Engineering and Vice Dean Research at the
tional Scholarship and Research Tuition Scholarship. His research School of Electrical and Electronic Engineering.
interests include AI security and multimedia forensics. Subsequently, he served as Associate Dean for
College of Engineering for eight years. He is
currently Professor and Director of Rapid-Rich
Object SEarch (ROSE) Lab and NTU-PKU Joint
Research Institute. He has published extensively
in the areas of signal processing, biometrics,
Anwei Luo received the B.S. degree from Jilin
image forensics and security, and computer vision and machine learning.
University, Changchun, China, in 2013. He is
Prof. Kot served as Associate Editor for more than ten journals, mostly
currently pursuing the Ph. D. degree from Sun
for IEEE transactions. He served the IEEE SP Society in various
Yat-sen University, Guangzhou, China. His cur-
capacities such as the General Co-Chair for the 2004 IEEE International
rent research interests include digital multimedia
Conference on Image Processing and the Vice-President for the IEEE
forensics, watermarking and security.
Signal Processing Society. He received the Best Teacher of the Year
Award and is a co-author for several Best Paper Awards including ICPR,
IEEE WIFS and IWDW, CVPR Precognition Workshop and VCIP. He was
elected as the IEEE Distinguished Lecturer for the Signal Processing
Society and the Circuits and Systems Society. He is a Fellow of IEEE,
and a Fellow of Academy of Engineering, Singapore.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 19
TABLE 9: Image manipulation localization performance (AUC score).
Method Venue NIST Columbia CASIAv1+ COVER DEF-12k IMD Carvalho IFC In-the-Wild Korus WildWeb AVG
FCN [64] CVPR15 .675 .696 .819 .694 .628 .748 .686 .605 .690 .644 .651 .685
U-Net [80] MICCAI15 .668 .645 .759 .622 .587 .703 .653 .598 .654 .626 .591 .646
DeepLabv3 [13] TPAMI18 .720 .853 .861 .763 .667 .815 .807 .631 .752 .675 .709 .750
MFCN [82] JVCIP18 .691 .634 .740 .614 .576 .664 .631 .591 .621 .621 .575 .633
RRU-Net [8] CVPRW19 .715 .749 .800 .676 .593 .754 .661 .586 .704 .669 .633 .685
MantraNet [95] CVPR19 .734 .734 .733 .722 .696 .760 .644 .592 .719 .646 .626 .691
HPFCN [55] ICCV19 .688 .607 .725 .591 .583 .683 .583 .564 .642 .607 .626 .627
H-LSTM [5] TIP19 .696 .571 .634 .634 .581 .656 .586 .553 .611 .588 .630 .613
SPAN [41] ECCV20 .751 .855 .756 .777 .641 .763 .671 .602 .749 .649 .582 .709
ViT-B [22] ICLR21 .705 .689 .763 .665 .602 .693 .674 .580 .692 .653 .605 .666
Swin-ViT [63] ICCV21 .723 .750 .777 .740 .669 .793 .668 .641 .710 .701 .572 .704
PSCC [62] TCSVT22 .676 .731 .822 .660 .600 .762 .700 .589 .696 .646 .558 .676
MVSS-Net++ [19] TPAMI22 .791 .818 .845 .871 .683 .817 .731 .635 .794 .659 .646 .754
CAT-NET [53] IJCV22 .522 .524 .668 .662 .818 .588 .603 .442 .504 .531 .536 .582
EVP [61] CVPR23 .775 .791 .855 .716 .697 .811 .688 .648 .748 .715 .695 .740
TruFor [35] CVPR23 .745 .916 .889 .827 .629 .832 .739 .634 .802 .670 .724 .764
PIM Ours .752 .884 .889 .809 .687 .870 .760 .669 .831 .725 .725 .782

TABLE 10: Image manipulation localization performance (MCC score with fixed threshold: 0.5).
Method Venue NIST Columbia CASIAv1+ COVER DEF-12k IMD Carvalho IFC In-the-Wild Korus WildWeb AVG
FCN [64] CVPR15 .151 .194 .425 .154 .113 .212 .083 .078 .192 .126 .162 .172
U-Net [80] MICCAI15 .155 .119 .263 .073 .036 .137 .098 .058 .140 .105 .053 .112
DeepLabv3 [13] TPAMI18 .226 .404 .428 .132 .065 .214 .173 .071 .203 .119 .091 .193
MFCN [82] JVCIP18 .230 .172 .351 .118 .062 .165 .145 .090 .152 .119 .102 .155
RRU-Net [8] CVPRW19 .190 .228 .292 .068 .028 .154 .054 .041 .155 .094 .087 .126
MantraNet [95] CVPR19 .107 .156 .120 .134 .061 .118 .090 .020 .157 .038 .087 .099
HPFCN [55] ICCV19 .155 .074 .180 .069 .028 .094 .052 .047 .093 .081 .068 .086
H-LSTM [5] TIP19 .354 .140 .140 .130 .044 .187 .114 .053 .155 .131 .133 .144
SPAN [41] ECCV20 .195 .454 .153 .142 .031 .141 .077 .046 .166 .075 .023 .137
ViT-B [22] ICLR21 .242 .193 .285 .114 .052 .196 .151 .053 .185 .163 .099 .158
Swin-ViT [63] ICCV21 .208 .321 .392 .159 .158 .303 .175 .098 .260 .136 .039 .204
PSCC [62] TCSVT22 .131 .338 .319 .110 .056 .166 .184 .035 .156 .085 .046 .148
MVSS-Net++ [19] TPAMI22 .289 .545 .503 .464 .097 .265 .170 .068 .265 .105 .063 .258
CAT-NET [53] IJCV22 .023 .055 .147 .135 .216 .208 .125 .043 .109 .040 .042 .104
EVP [61] CVPR23 .205 .266 .478 .103 .090 .236 .055 .082 .228 .118 .096 .178
TruFor [35] CVPR23 .257 .795 .536 .270 .147 .358 .210 .117 .344 .117 .149 .300
PIM Ours .264 .630 .565 .230 .162 .415 .229 .142 .396 .228 .212 .318

A PPENDIX inconsistencies rather than semantic-level inconsistencies.

Details of PIDA. In our work, we exclusively use the Additional evaluations. In Table 9, we report the cross-
CASIAv2 dataset for training, which includes 7,491 real im- dataset forgery localization performance using the threshold-
ages and 5,123 fake images. Only real images from CASIAv2 free metric AUC. Notably, our method achieves an out-
are used for Pixel-Inconsistency Data Augmentation (PIDA). standing 78.2% AUC performance. Compared with MVSS-
Fig. 7 illustrates the PIDA pipeline. We apply four common Net++ [19], the proposed method achieves a 2.8% average
perturbation types to the pristine real images (Ip ): Gaussian AUC-score improvement, increasing from 75.4% to 78.2%. In
blurriness, compression, noise, and color channel shuffling. Table 10, our method consistently achieves the best or second-
By combining the corrupted image (Ic ), the pristine real best detection performance on unseen testing datasets. And
images (Ip ), and the foreground mask (M ), we generate our average MCC performance outperforms SOTA methods
the augmented forgery sample (Ib ) and the corresponding by a clear margin.
label (M ). For Gaussian blurriness, each Ip in CASIAv2 is Pixel-level evaluation at different thresholds. The determi-
blurred with a kernel size ∈ {3, 5, 7, 9, 11}. Each Ip is nation of threshold values is crucial for the final localization
also compressed with a random Quality Factor (QF) ∈ [71, performance [19]. We assess the effectiveness of our model’s
95], and the standard deviation σ of the Gaussian noise is forgery localization across a range of threshold values from
randomly sampled from σ ∈ (0.01, 0.20). Additionally, we 0.1 to 0.9. We classify a pixel as a forgery if its predicted
randomly shuffle the RGB color channels of Ip to obtain Ic . probability exceeds the specified threshold. Fig. 13 presents
Consequently, we obtain 7,491 × 4 PIDA forgery images. the average localization performance on the 11 unseen
Each image is randomly horizontally flipped before being datasets using F1, MCC, and IoU metrics. Namely, we plot
passed to the model during training. The purpose of PIDA is the average results under the cross-dataset setting with
to drive the model to focus on extracting inherent pixel-level varying thresholds. Our proposed method consistently out-
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 20

TABLE 11: Ablation experiments on MHSA. TABLE 13: Trained on DEF-84k dataset.
LPDE GPDE Avg. F1 Avg. IoU DEF-12k CASIAv1+
Method
MH MH .333 .275 F1 IoU F1 IoU
Swin-ViT [63] .477 .423 .058 .048
SH MH .318 .270 TruFor [35] .514 .456 .152 .087
MH SH .312 .261 PIM (Ours) .542 .483 .168 .102
SH SH .298 .245

TABLE 12: Selection of loss weights. Description:

A cute dog
with black
λB λC λR F1 AUC background.
Generated Extracted Original
1.0 0.01 0.1 .320 .752 Stable Diffusion image Object image
Generator
1.0 0.1 0.1 .312 .740
1.0 0.001 0.1 .333 .782
1.0 0.001 1.0 .331 .775 ‘‘trike’’
1.0 0.001 0.01 .327 .769 Harmonization
Sophisticated Blender Copy-paste generated
Forgery image anomalous image

Fig. 11: Generation pipelines of Sophisticated manipulation

performs existing models across all thresholds, underscoring pipelines. (a). Dall-E2 cityscape manipulation dataset. (b).
its superiority regardless of the threshold selection. Stable Diffusion cityscape manipulation dataset.
We observe that most detectors’ performance continu-
ously decreases with higher threshold values. This phe- we present the detailed ablation experimental results (F1
nomenon may be attributed to subtle artifacts in challenging and IoU scores) across the 11 testing datasets in Table 15
forgery regions, where detectors struggle to make confident and Table 16. The 12 listed settings indicate the use of
decisions, resulting in reduced true positives (TP) at higher different designed components, details of which can be found
thresholds. This finding indicates the importance of selecting in Table 8. The overall localization performances further
a lower threshold when deploying a forgery detector in demonstrate the effectiveness of the designed components.
real-world scenarios. Compared to Setting 1 which only uses the transformer
IoU score on the unseen manipulation: Inpainting. We backbone (BB) to perform forgery localization, Setting 2
further report Image manipulation localization performance incorporates a boundary decoder, achieving a superior
(IoU score with fixed threshold: 0.5) on the unseen Inpainting performance, particularly on the more challenging DEF-12k,
data in Table 14. Our designed method PIM consistently IFC, and In-the-Wild datasets. This improvement highlights
achieves outstanding IoU scores across different inpainting the significance of boundary information in enhancing the
techniques. Furthermore, PIM significantly overperforms the model’s generalizability to practical scenarios. Setting 3
best model CAT-NET, demonstrating our method’s supe- and 4 employ regular data augmentation (RDA) and the
rior generalizability to unseen manipulations from another proposed pixel-inconsistency data augmentation (PIDA),
perspective. respectively, further improving overall forgery localization
Showcases of perturbed images for robustness evaluation. performance. Notably, PIDA outperforms RDA in F1 and
To mimic uncontrollable real-world scenarios, we incorporate IoU scores across 8 out of 11 datasets, demonstrating its
six common image perturbation types with nine severity effectiveness. Furthermore, combining RDA and PIDA in
levels to examine the robustness of the image forgery Setting 5 yields additional performance gains, as the joint use
localization models. The showcase examples of Severity ‘1’, of these augmentations enables the model to better handle
‘5’, and ‘9’ are shown in Fig. 12. complex forgeries. Setting 6 and 7 introduce central difference
Generation details of the sophisticated datasets. Fig. 11 convolution (CDC) and radial difference convolution (RDC),
illustrates the generation pipelines of the two sophisticated respectively. Both modules consistently enhance performance
datasets. In Fig. 11 (a), we manually select the appropriate across most datasets, as they effectively model local pixel
position and size for the generated object and then pass dependencies critical for generalized forgery localization.
a reasonable object prompt to Dall-E2 (DE2) to obtain a In Setting 8, the naive concatenation of CDC and RDC
photo-realistic image with a high level of harmonization. features increases the diversity of captured local pixel-
Since manually generating sophisticated forgery images is inconsistency features, resulting in overall improvements.
costly, we further apply existing algorithms in Fig. 11 (b) To further optimize feature fusion, Setting 9 incorporates
to automatically produce sophisticated forgery images. The a learning-to-weight module (LWM), which dynamically
resulting images exhibit high-level harmonization, with the adjusts the weights of CDC and RDC features based on
forgery object having compatible illumination, reasonable different input images. This strategy significantly enhances
size, semantic consistency, and appropriate position. Con- generalizability on unseen datasets. Setting 10 integrates a
sequently, the DE2 and SD datasets include 60 and 328 compactness loss LC , which delivers notable improvements
sophisticated fake images, respectively. on challenging CASIAv1+ and COVER datasets, likely due
Additional visualization results. Fig. 14 shows additional to their compact forgery regions. Setting 11 introduces a
forgery localization results under the cross-dataset experi- global pixel dependency encoder (GPDE), which significantly
mental setting. Our method accurately identifies the manip- boosts F1 and IoU scores on the Columbia and Carvalho
ulated regions. In comparison with state-of-the-art (SOTA) datasets with large forgery regions. This demonstrates that
methods, the proposed method demonstrates a superior the proposed GPDE successfully models long-range pixel
forgery localization performance. inconsistencies. However, relying heavily on GPDE causes
Detailed ablation experimental results. To enhance clarity, slight performance drops on datasets with small manipulated
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 21

Fig. 12: Showcases of (a). raw real image, raw forgery image, and ground-truth mask. The corresponding six image
perturbation types of the raw forgery image: (b). Brightness; (c). Contrast; (d). Darkening; (e). Dithering; (f). Pink noise; (g).
JPEG2000 compression. The top, middle, and bottom rows show Severity ‘1’, ‘5’, and ‘9’ for all perturbation types.

Fig. 13: Image forgery localization performance with varying thresholds. (a). F1; (b). MCC; (c). IoU.

regions. To address this limitation, Setting 12 employs a CASIAv1+, WildWeb, COVER, NIST2016, Carvalho, Korus,
reconstruction loss, LR , which encourages the model to In-the-wild, DEF-12k-test, and IMD2020), unseen inpainting
also pay attention to image contents. This regularization datasets (CA, EC, GC, LB, LR, NS, PM, RN, SG, SH, and TE),
effectively mitigates the issue, leading to overall performance and recent AIGC datasets (Dall-E2 (DE2), Stable Diffusion
enhancements. (SD), Autosplice, and CocoGlide). These experimental results
Impacts of MHSA on image forgery localization. In our verify the adaptability of the selected loss weights from
designed model, we adopt the Multi-Head Self-Attention another point of view.
(MHSA) strategy in both the Local Pixel Dependency Encoder
To further validate our method’s adaptability, we train
(LPDE) and the Global Pixel Dependency Encoder (GPDE),
our model on the DEF-84k image manipulation dataset [67]
using head numbers of [3, 6, 12, 24] across the four trans-
using the same loss weights and compare it with previous
former blocks. To examine the impact of MHSA, we conduct
methods, as shown in Table 13. Note that all listed methods
ablation experiments in Table. 11, where SH and MH refers to
are trained on DEF-84k to ensure a fair comparison. It can be
the Single-Head and Multi-Head Self-Attention mechanism,
observed that our method PIM still achieves the best perfor-
respectively. We report the average F1 and IoU scores across
mance on the DEF-12k test set and best generalizability to the
11 unseen datasets in Table. 11. MHSA effectively scales the
unseen CASIAv1+ dataset, demonstrating the adaptability of
model’s capacity and enables the model to search in larger
the selected loss weights.
feature space, resulting in superior image forgery localization
performance compared to SHSA. In addition, it is observed Visualization ablation experiments on GPIM & LPIM. To
that MHSA has a greater impact on GPDE than on LPDE. The demonstrate the efficacy of the Global Pixel-Inconsistency
potential reason could be that accurately modeling global Modeling (GPIM) and Local Pixel-Inconsistency Modeling
pixel dependency for input images requires larger feature (LPIM) strategies, we visualize the ablation results of image
space. forgery localization maps in Fig. 15. The top three rows
Impacts of loss weights. Table 12 shows the average F1 and represent the input images, the corresponding ground-truth
AUC across all 11 test datasets using different loss weights. masks, and the predicted results of our proposed Pixel-
We first fix λB at 1.0, assigning equal importance to mask Inconsistency Modeling (PIM) method. The fourth row
and boundary predictions. We then initialize λC and λR at presents the predicted forgery maps without using the raster-
0.01 and 0.1, respectively, to balance the scale of the loss scan mask in the attention mechanism, while the bottom
components in the early iterations. Subsequently, we tune row shows the results without the designed difference
λC and λR and report the image forgery localization results convolutions in the local pixel dependency encoder. From
in Table 12. The model achieves the highest F1 and AUC the highlighted red boxes, we observe that our proposed
scores on unseen datasets when λC is 0.001 and λR is 0.1. PIM method can more accurately localize forgery pixels,
Our trained model using the determined loss weights regardless of whether the forgery regions are substantial or
has been demonstrated effective on multiple forgery image subtle. This finding evidences that PIM indeed benefits from
datasets. The proposed method achieves strong general- the designed GPIM and LPIM strategies, thereby achieving
izability across unseen traditional forgery datasets (IFC, superior pixel-level forgery detection performance.
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 22

TABLE 14: Image manipulation localization performance (IoU score with fixed threshold: 0.5) on the unseen manipulation
type: Inpainting.

Method Venue CA EC GC LB LR NS PM RN SG SH TE AVG

FCN [64] CVPR15 .065 .024 .005 .019 .385 .083 .161 .095 .243 .083 .037 .109
U-Net [80] MICCAI15 .006 .008 .004 .003 .267 .448 .073 .047 .048 .030 .422 .123
DeepLabv3 [13] TPAMI18 .075 .050 .006 .015 .467 .541 .180 .144 .378 .095 .493 .222
MFCN [82] JVCIP18 .008 .013 .001 .009 .136 .490 .031 .048 .032 .046 .480 .118
RRU-Net [8] CVPRW19 .022 .037 .017 .014 .356 .433 .129 .071 .128 .054 .357 .147
MantraNet [95] CVPR19 .182 .308 .183 .269 .037 .335 .023 .201 .058 .262 .259 .192
HPFCN [55] ICCV19 .006 .007 .004 .005 .120 .400 .013 .026 .012 .021 .365 .089
H-LSTM [5] TIP19 .028 .018 .024 .021 .076 .034 .025 .038 .022 .028 .028 .031
SPAN [41] ECCV20 .005 .022 .006 .004 .289 .361 .077 .085 .146 .012 .184 .108
ViT-B [22] ICLR21 .012 .010 .009 .019 .075 .281 .012 .023 .020 .033 .270 .069
Swin-ViT [63] ICCV21 .158 .177 .003 .057 .308 .144 .304 .245 .277 .221 .039 .176
PSCC [62] TCSVT22 .208 .207 .066 .126 .186 .519 .112 .181 .226 .148 .479 .223
MVSS-Net++ [19] TPAMI22 .063 .036 .007 .016 .489 .735 .229 .189 .329 .153 .731 .271
CAT-NET [53] IJCV22 .450 .429 .286 .658 .244 .354 .167 .426 .470 .509 .361 .396
EVP [61] CVPR23 .207 .290 .035 .300 .393 .212 .245 .267 .393 .434 .206 .271
TruFor [35] CVPR23 .119 .102 .105 .210 .119 .126 .042 .102 .064 .079 .125 .108
PIM Ours .530 .567 .052 .702 .690 .758 .416 .370 .832 .523 .782 .566

TABLE 15: Ablation study for image manipulation localization (F1 score with fixed threshold: 0.5).

Setting NIST Columbia CASIAv1+ COVER DEF-12k IMD Carvalho IFC In-the-Wild Korus WildWeb AVG
1 .220 .365 .390 .168 .157 .300 .183 .102 .265 .134 .040 .211
2 .186 .339 .358 .124 .166 .346 .251 .112 .336 .168 .036 .220
3 .233 .485 .515 .132 .126 .281 .175 .132 .267 .166 .046 .233
4 .275 .433 .331 .260 .127 .308 .177 .110 .387 .201 .249 .260
5 .237 .614 .525 .175 .168 .380 .129 .137 .372 .200 .177 .283
6 .272 .628 .506 .235 .168 .392 .211 .138 .414 .199 .177 .304
7 .290 .636 .525 .221 .164 .388 .200 .142 .406 .192 .225 .308
8 .294 .670 .526 .222 .169 .393 .174 .135 .418 .213 .220 .312
9 .269 .754 .507 .235 .162 .395 .232 .149 .408 .183 .194 .317
10 .264 .677 .543 .282 .171 .405 .239 .160 .404 .210 .200 .323
11 .284 .720 .516 .286 .142 .393 .312 .141 .426 .201 .207 .330
12 .280 .680 .566 .251 .167 .419 .253 .155 .418 .234 .236 .333

TABLE 16: Ablation study for image manipulation localization (IoU score with fixed threshold: 0.5).

Setting NIST Columbia CASIAv1+ COVER DEF-12k IMD Carvalho IFC In-the-Wild Korus WildWeb AVG
1 .167 .297 .356 .124 .129 .243 .132 .078 .214 .103 .033 .171
2 .150 .271 .328 .092 .136 .281 .186 .088 .268 .131 .026 .178
3 .180 .395 .471 .098 .101 .226 .135 .104 .212 .128 .035 .190
4 .222 .367 .281 .200 .101 .242 .126 .085 .305 .161 .208 .209
5 .190 .542 .474 .138 .138 .316 .094 .109 .305 .158 .145 .237
6 .216 .552 .459 .179 .137 .324 .154 .108 .340 .158 .144 .252
7 .235 .562 .475 .179 .133 .319 .147 .112 .332 .154 .188 .258
8 .242 .598 .473 .178 .136 .324 .129 .109 .339 .172 .183 .262
9 .229 .692 .461 .190 .132 .329 .179 .119 .335 .146 .165 .271
10 .216 .602 .490 .223 .139 .332 .184 .126 .321 .164 .167 .269
11 .228 .642 .465 .226 .110 .319 .229 .106 .343 .155 .167 .272
12 .225 .604 .512 .188 .133 .340 .194 .119 .338 .182 .193 .275
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 23

Fig. 14: Additional forgery localization results on the 11 unseen test sets. The three left columns show the input images,
corresponding ground-truth, and the localization results of our method. The right 11 columns present the results of SOTA
methods.

Fig. 15: Visualization ablation experiments on the designed Global Pixel-Inconsistency Modeling (GPIM) and Local Pixel-
Inconsistency Modeling (LPIM).