International Journal of Multimedia Information Retrieval (2025) 14:8

[Link]

REGULAR PAPER

Image forgery classification and localization through vision

transformers
Digambar Pawar1 · Raghavendra Gowda2 · Krishna Chandra1

Received: 10 June 2024 / Accepted: 14 February 2025

© The Author(s), under exclusive licence to Springer-Verlag London Ltd., part of Springer Nature 2025

Abstract
Due to the easy availability of software over the Internet, any naive user can tamper the images for entertainment purposes
or to defame a personality by circulating over social media networks. The practice of image tampering is a serious issue and
can attract legal action if proven guilty. Forensic researchers employ various methods to detect and localize image forgeries.
In this research, we use a Vision transformer (ViT) as a method for binary classification of images distinguishing forged
and unforged images. Further, we use a pre-trained Segment Anything Model(SAM) which is fine-tuned with custom data to
adaptively recognize patterns indicating forged regions within the images. SAM can localize these forged areas and is leveraged
to create templates by extracting the identified regions. The proposed method is rigorously tested across various datasets,
including CASIA v1.0, CASIA v2.0, MICC-F2000, MICC-F600, and Columbia. Through comprehensive experimentation,
our approach showcases considerable promise yielding accuracy in image forgery classification and localization. Our model’s
robustness and adaptability make it an attractive tool for forensic analysis in diverse scenarios, contributing to the advancement
of multimedia forensics security research.

Keywords Image forensics · Forgery detection · Forgery localization · Vision transformer · Segment anything model ·
Binary mask

1 Introduction or concealing past events. Common tampering operations

include deleting, adding, and modifying contents on both
The widespread availability of user-friendly software on the images and videos.
Internet and smart devices has led to an increase in image Image forgery encompasses a range of attacks such as
tampering. Image tampering is a specific form of forgery that copy-move, resampling, noise variations, splicing, retouch-
involves altering parts of image contents. Tampered images, ing, and JPEG compression. This paper focuses on two
especially those disseminated on the Internet evoke public prevalent image forgery techniques: image splicing and copy-
reactions leading to serious consequences, including their move exploring their characteristics and detection methods.
use to mislead evidence in legal proceedings. Tampering an By focusing on these commonly employed forgery tech-
image involves creating fake content within it by distorting niques, the aim is to enhance our understanding of image
tampering and develop robust methods to detect and coun-
B Raghavendra Gowda teract these manipulations.
goudru@[Link]
The approaches for detecting forgeries are divided into
Digambar Pawar active and passive categories. The process of detecting and
dpr@[Link]
combating useful and explicit image modifications is known
Krishna Chandra as active forgery detection. To actively embed or detect
k2021@[Link]
manipulation in an image, these techniques frequently make
1 School of Computer and Information Sciences, University of use of watermarking, adding extra information, or involving
Hyderabad, Central University, Prof. C R Rao Road, cryptographic techniques. Active forgery detection is a more
Hyderabad, Telangana 500046, India proactive technique because the aim is to prevent or discover
2 Department of Computer Science and Engineering, forgeries proactively. On the other hand, passive forgery
Vardhaman College of Engineering, Kacharam, Shamshabad, detection uses the image’s natural qualities and properties to
Hyderabad, Telangana 501218, India

0123456789().: V,-vol 123

 8 Page 2 of 11 International Journal of Multimedia Information Retrieval (2025) 14:8

Fig. 2 Copy-move image forgery [22]

Fig. 1 Image splicing forgery attack [26]

the image and detect copy-move forgeries. These methods

identify possible alterations. Instead of adding new informa- include content-based picture retrieval, feature extraction,
tion, these methods examine the inherent characteristics of and pattern recognition [3]. Figure 2 shows copy-move image
the images, such as statistical anomalies, noise patterns, and forgery where the left side of the image is the original and
inconsistencies. Finding anomalies like texture and abnor- the right side of the image is Copy-Move.
malities that can point to post-mortem tampering is the main Maintaining the integrity of digital communication, guard-
goal of passive forgery detection. Analyzing textures within ing against false information, and guaranteeing the validity
an image is to identify inconsistencies and suggest the detec- of visual evidence in forensic and legal settings all depend
tion of tampering attacks (image splicing and copy-move). on the timely detection of forgeries. To combat the issues
Ensuring that the model remains reliable and effective associated with image forgeries, deep learning and complex
across various types of image forgeries across varied con- manipulation techniques are applied with cutting-edge tech-
ditions requires robust training, testing, and post-processing nologies. Various research works are ongoing to handle enor-
approaches: Diverse & augmented training data, Context- mous image forgery detection techniques. As anti-forensic
Aware feature extraction, adaptive learning & transfer learn- techniques (e.g., noise addition, compression artifacts, adver-
ing, and Continuous model evaluation & Testing. sarial attacks) evolve to obscure forgeries, detection methods
must continuously adapt and improve with the following
1.1 Image splicing strategies to ensure that forensic models remain robust and
effective against emerging threats.
Splicing is a kind of digital image fraud in which a new image
is created by combining one or more regions taken from
other images. This is frequently done to produce a composite 1. Adapt self-learning models.
image that combines elements from various sources to look 2. Multi-scale & cross-domain feature learning.
authentic. Sophisticated methods are needed to detect image 3. Adversarial training against anti-forensic attacks.
splicing which examines variations in the texture, color, and 4. Hybrid detection using frequency & spatial analysis.
lighting of various image components. Figure 1 shows image 5. Real-world data collection & active learning.
splicing forgery involving composite forgery attack [22, 26].
In Fig. 1, the left side of the image shows a Badri bird
whose body shares similar shades with a tiger. The right side 2 Background
of the Badri bird’s face is cleverly replaced with the face
of the tiger’s face, creating a funny hybrid bird called the Vision transformers (ViTs) have transformed the field of
"tiger bird". Social media users share the composite image computer vision with their capacity to capture global con-
claiming the new species exists. This highlights how image text and long-range interdependencies in an image. ViTs
manipulation can lead to misinformation and impact people’s have achieved state-of-the-art performance in a variety of
emotions. tasks, including picture segmentation, object identification,
and image classification. Thanks to their distinctive architec-
1.2 Copy-move ture, which is built on self-attention mechanisms. ViTs have
proven useful in detecting tampered regions and differenti-
Copy-move is a type of image fraud taking a portion of a ating between real and fake images, indicating potential in
picture, copying it, and then pasting it somewhere else in the the field of image forgery detection. Further research intends
same image to hide or copy. This method is frequently used to to address the constraints and improve the capabilities of
hide undesirable components or to generate many instances ViTs for image forgery detection and other computer vision
of a person or object in a picture. Advanced digital image pro- tasks. Figure 3 shows the architecture of Vision transformers
cessing methods can be applied to spot duplicate elements in (ViTs).

123
International Journal of Multimedia Information Retrieval (2025) 14:8 Page 3 of 11 8

Fig. 3 Vision transformer (ViTs) architecture [2]

Fig. 4 Segment anything model (SAM) architecture [15]

3 Related work
Segment Anything Model (SAM), is a powerful deep-
learning architecture designed specifically for image seg- Bi et al. [4] provides a thorough overview of Evolutionary
mentation tasks. It leverages a novel transformer-based Computation (EC) methods used in computer vision and
encoder-decoder structure to achieve state-of-the-art results image processing. The survey delves into the noteworthy
in various segmentation applications. The Segment Any- accomplishments of EC in tackling image-related problems
thing Model (SAM), introduced by Kirillov et al. [15], is including edge detection, picture segmentation, feature anal-
a novel model for image segmentation. In addition, the ysis, classification, and object detection. The authors provide
authors have created the largest segmentation dataset to date, insights into the history, current state, and future directions
with over 1 billion masks extracted from 11 million autho- of Evolutionary Computer Vision (ECV) by discussing the
rized and privacy-respecting photos by utilizing an effective contributions of several EC techniques. This survey is a great
model within a data-collecting loop. The Segment Anything resource for learning about the state of EC in the context of
Model (SAM) can move zero-shot to new image distribu- computer vision and image analysis.
tions and tasks. The authors assessed SAM’s performance Han et al. [10] provide a two-stage CNN transfer learning
on a range of tasks and found that it frequently matches or and web data augmentation approach for image categoriza-
outperforms previous fully supervised approaches in terms tion. This method effectively transfers pre-trained network
of zero-shot performance. Integrating the Segment Anything features, which help deep CNNs overcome the problem of
Model (SAM) with Focal Loss for accurate identification having insufficient training data. The strategy efficiently
and localization of forged regions in images requires Model addresses overfitting on small datasets and minimizes the
Selection & Adaptation, Improving Localization Accuracy, requirement for large amounts of training data by supple-
Computational Considerations, and Evaluation metrics. The menting the original dataset with images from the Internet.
SAM dataset (SA-1B), which includes 1 billion masks and Fine-tuning and hyper-parameter tuning is done via Bayesian
11 million images is publicly available to support future optimization. Experiments conducted on six small datasets
research in foundation models for computer vision. Integrat- showed better performance, especially with ResNet surpass-
ing the Segment Anything Model (SAM) with Focal Loss for ing the performance of the most advanced models.
accurate identification and localization of forged regions in Mehrjardi et al. [18] examine deep learning-based image
images requires Model selection & adaptation, Enhancing fraud detection in their survey, highlighting the growing sig-
SAM with Focal Loss, Training pipeline & dataset aug- nificance of image authenticity and reliability especially in
mentation, Improving localization accuracy, Computational applications related to medicine, forensics, and the legal sys-
considerations, and Evaluation & metric. Figure 4 shows the tem. The survey examines several forms of image forgeries,
architecture of the Segment Anything Model (SAM). evaluation criteria, benchmark datasets, and conventional

123
 8 Page 4 of 11 International Journal of Multimedia Information Retrieval (2025) 14:8

techniques for detecting forgeries. While conceding the forgery detection were found by the enhanced SIFT descrip-
shortcomings of older methodologies, the authors underline tor.
the robustness and automatic recognition capabilities of deep In their extensive review, Han et al. [11] provide an in-
learning methods, placing them as a popular alternative in depth synopsis of the developments in ViT architecture and
computer vision. address the important topics including training methods,
The review paper by Taneja et al. [24] examines digital applications, and architecture design, providing insightful
picture anti-forensics and highlights how it affects forensic information about how the Vision Transformers market is
difficulties. The article discusses artifacts like JPEG block- developing. This survey is a great way to learn about the
ing and contrast enhancement that are essential for forensic current state of the art in virtual reality research.
methods. By removing traces of their activities, anti-forensic Asiri et al. [2] provides a thorough investigation into
techniques make detection more difficult. The review also the classification of brain tumors utilizing fine-tuned vision
describes attacks such as JPEG anti-forensics that help foren- transformers (ViTs). Several pre-trained ViT models (R50-
sic analysts create reliable forgery detection for applications ViT-l16, ViT-l16, ViT-l32, ViT-b16, and ViT-b32) are used in
like cybercrimes by summarizing contemporary methodolo- the study to improve the state-of-the-art in brain tumor clas-
gies. sification. With 4855 training images and 857 testing images,
He et al. [12], introduced a unique method titled "Image ViT-b32 was found to be the best-performing model trained
Copy-Move Forgery Detection via Deep Cross-Scale Patch- on four different tumor classes. The outcomes outperform
Match", which addresses the practical limitations of existing current techniques, demonstrating the usefulness of the sug-
deep learning algorithms. By combining traditional and gested strategy and setting a standard for further study on the
deep learning techniques [20, 21], end-to-end deep cross- categorization of brain tumors.
scale patch-match is introduced and designed specifically The difficulties of accurately detecting and localizing
for copy-move forgery detection (CMFD). The method face forgeries due to lack of pixel-by-pixel supervision are
stresses explicit point-to-point matching using features from tackled by well-trained Segment-Anything Model(SAM) in
high-resolution scales, boosting generalizability to different Detect Any Deepakes(DAD) framework [16]. DAD frame-
copy-move contents. Comprehensive trials show that the sug- work effectively captures both short and long-range forging
gested framework performs better than the current methods. contexts with the help of the Multiscale Adapter and the
A novel CrossViT model was developed by Chen et al. Reconstruction Guided Attention (RGA) module, increasing
[6] which is a type of vision transformer (ViT) that investi- the sensitivity of the model to forged regions.
gates the multi-scale feature representations in transformer In the work titled "Segment Anything", Kirillov et al. [15]
models for image classification. The dual-branch transformer introduced the Segment Anything (SA) project. For image
uses attention mechanisms to fuse images by processing segmentation, the project presents a new task, model, and
image patches of different widths. A cross-attention-based dataset. To create the largest segmentation dataset to date-
token fusion module is proposed to improve model efficiency, which includes over 1 billion masks on 11 million licensed,
which results in a considerable reduction in computation time privacy-respecting images-the authors employed an effective
when compared to quadratic computation. Numerous exper- model in the data gathering loop. Segment Anything Model
imental tests demonstrate that CrossViT performs better than (SAM) is a model that can be zero-shot transferred to new
current works and CNN models. image distributions and jobs since it is promptable by design
In the thorough analysis of attention mechanisms in com- and training. On a variety of tasks, the zero-shot performance
puter vision, Guo et al. [9] examined how well the attention of SAM is shown to be impressive and frequently on par with
mechanisms work for various tasks, including semantic seg- or better than previous fully supervised results.
mentation, object recognition, and image classification. The
attention mechanisms dynamically modify weights based on
input image elements, drawing inspiration from the human
visual system. The survey provides a comprehensive analysis
of attention divided into methods like temporal, geograph- 4 Challenges and contributions
ical, branch, and channel and also suggests future paths of
inquiry. Overall, the major challenges encountered in optimizing
A feature-based copy-move forgery detection (CMFD) models for classification and localization were instrumental
technique was introduced by Bin Yang et al. [28] in which in refining the proposed methodologies. The contributions
key points are found using a modified Scale Invariant Fea- lie in the successful implementation of ViT-l32 for supe-
ture Transform (SIFT) detector. To distribute the key points rior image classification and the introduction of SAM with
throughout the image, a key-point distribution approach focal loss for accurate object localization, addressing critical
was developed. In the end, the critical spots for copy-move aspects in the field of forensic image analysis.

123
International Journal of Multimedia Information Retrieval (2025) 14:8 Page 5 of 11 8

5 Proposed methodology Table 1 Details of different types of vision transformers (ViT)

Model Layer (Type) Output Shape Parameters
The proposed methodology includes the classification and
localization discussed in the following subsections. The R50-ViT-l16 KerasLayer (None, 2048) 23,500,352
dataset comprises images along with the corresponding Flatten (None, 2048) 0
masks and labels. Initially, the images, along with labels are Dense (None, 11) 22,539
inputted into the selected Vision Transformer model chosen ViT-b16 Functional (None, 768) 85,798,656
from R50-ViT-l16, ViT-b16, ViT-l16, ViT-l32, and ViT-b32. Flatten (None, 768) 0
This model effectively determines the authenticity of each Dense (None, 4) 48
image, classifying it as either forged or genuine. Subse- ViT-l16 Functional (None, 1024) 303,301,632
quently, the forged images are directed to the SAM (Segment Flatten (None, 1024) 0
Anything Model) for precise forgery localization, leveraging Dense (None, 4) 48
its advanced segmentation capabilities. This comprehensive ViT-l32 Functional (None, 1024) 305,510,400
approach adeptly addresses image forgery classification and Flatten (None, 1024) 0
localization challenges, showcasing the synergy between Dense (None, 4) 48
Vision Transformer models and SAM for robust perfor- ViT-b32 Functional (None, 768) 87,455,232
mance. Flatten (None, 768) 0
Dense (None, 4) 48
5.1 Classification

1. The Vision_Transformer R50-vit-l16 model, which uses

resentations. Similar to vit_l16, batch normalization and
a transformer with an output shape of (None, 1024)
a Dense layer with one neuron facilitate binary classifica-
and 303,301,632 parameters, excels in capturing com-
tion. While computationally more intensive, this model
plex visual patterns. Its flexibility for binary classification
is well-suited for tasks that demand a high level of feature
applications is increased with the addition of batch nor-
abstraction.
malization and a Dense layer with a single neuron. This
5. The Vision_Transformer vit_b32 model utilizes a trans-
model can capture more subtle information than vit_b16
former with an output shape of (None, 768) and
due to its increased feature dimensionality, which may
87,455,232 parameters. Combining efficiency with effec-
be useful to capture detailed image information.
tiveness, this model strikes a balance between computa-
2. For the categorization of fake images, the Vision_
tional resources and feature representation. With batch
Transformer vit_b16 model is a strong option thanks
normalization and a Dense layer for binary classifi-
to its transformer architecture, which is tuned for scal-
cation, vit_b32 provides a scalable solution for image
ability and efficiency. The 768-dimensional output it
forgery detection tasks. The 768-dimensional output cap-
produces achieves a balance between expressive power
tures diverse visual features, making it suitable for our
and processing economy by capturing hierarchical visual
problem. Tuning the Vision Transformer (ViT-L/32) for
information. Vit_b16 is a good contender for our classi-
superior performance in image classification requires
fication task since it can distinguish between counterfeit
careful selection of hyperparameters, data augmentation
photos with batch normalization and a Dense layer for
strategies, and optimization techniques.
binary classification.
3. The Vision_Transformer vit_l16 model, featuring a
transformer with an output shape of (None, 1024) and In conclusion, the ViT_l32 model emerges as the optimal
303,301,632 parameters, excels in capturing intricate choice for forgery image classification offering a balance
visual patterns. The inclusion of batch normalization between accuracy, efficiency, and scalability. Table 1 shows
and a Dense layer with one neuron enhances its adapt- the details of ViT_l32 offering a harmonious blend of effi-
ability for binary classification tasks. The larger feature ciency, scalability, and effective feature representation. Its
dimensionality compared to vit_b16 allows this model to balanced architecture and transformer-based design make it
potentially capture more nuanced details, which can be a standout performer among the considered models.
advantageous to understanding image patterns clearly.
4. The Vision_Transformer vit_l32 model, with an out- 5.2 Localization
put shape of (None, 1024) and 305,510,400 parameters,
extends the capabilities of vit_l16 by incorporating a 1. Focal Loss: The Focal Loss is defined as:
deeper transformer architecture. This increased depth
allows the model to learn more complex hierarchical rep- Focal Loss = −(1 − pt )γ · log( pt )

123
 8 Page 6 of 11 International Journal of Multimedia Information Retrieval (2025) 14:8

where pt is the predicted probability, and γ is the focus- Table 2 Details of datasets used
ing parameter [17]. Dataset Authentic Forged Type Mask
2. Dice Focal Loss: The Dice Focal Loss is defined as:
CASIA v1.0 800 459+462 CM,SP Yes
CASIA v2.0 7491 5123 CM,SP Yes
Dice Focal Loss = (1 − Dice Coefficient)γ · Focal Loss
MICC-F2000 1300 700 CM No
MICC-F600 440 160 CM Yes
where the Dice Coefficient is used in conjunction with
Columbia 183 180 SP No
the Focal Loss [19].
3. Adam Optimizer: The Adam Optimizer is an adaptive
learning rate optimization algorithm. The update rule is
given by: by a wide array of copy-move image forgery instances, ulti-
mately advancing the field of image forensics.
m̂ t MICC-F600 introduces complexity in the realm of copy-
θt+1 = θt − lr ·  move image forgery detection. With a collection of 600
v̂t +  challenging images, this dataset has intricate geometric trans-
formations and introduces additional noise. Researchers turn
where θt is the parameter at time t, m̂ t and v̂t are the bias- to MICC-F600 to test the robustness and efficacy of their
corrected moving averages of the gradient and its square, algorithms under more demanding conditions, pushing the
lr is the learning rate (lr = 1 × 10−5 in the provided boundaries of copy-move image forgery detection capabili-
program), and  is a small constant to prevent division ties.
by zero [14]. Columbia The Columbia dataset emerges as an extensive
resource, comprising copy-move forgery images, splicing
images, and a set of pristine images. With a total of 3876
6 Results and analysis images, this dataset provides a diverse array of forgery and
non-forgery scenarios. Researchers leverage the Columbia
6.1 Dataset dataset for training and testing purposes, benefitting from
its inclusion of various image manipulation techniques. This
CASIA 1.0 stands as a pivotal dataset in the realm of diversity contributes to the dataset’s significance in advanc-
image forgery detection, focusing on copy-move forgery ing the field of image forensics. Table 2 shows details of the
images. Comprising a substantial collection of 4000 images, datasets used.
this dataset offers a diverse array of realistic scenarios, Figure 5 illustrates samples from the datasets, showcasing
showcasing varying degrees of manipulation. The images two images identified as forged and one as authentic. This
within CASIA 1.0 are sourced from different origins and visual representation provides a glimpse into the diversity of
exhibit transformations such as rotation, scaling, and bright- the dataset and the varying characteristics between forged
ness adjustments. Researchers widely employ this dataset to and authentic images shown in Table 2.
assess and enhance the effectiveness of algorithms in detect-
ing copy-move forgery within digital images. 6.2 Computational metrics
CASIA 2.0 extends the complexity of image forgery
scenarios. This dataset exclusively consists of copy-move 6.2.1 Precision
forgery images, totaling 5000 in number. Notably, CASIA
2.0 introduces more intricate operations, including multi- Precision measures the accuracy of positive predictions made
ple copy-moved regions, geometric transformations, and the by the model. It is calculated as the ratio of true positive
intentional addition of noise. The increased complexity chal- predictions to the sum of true positives and false positives.
lenges of image forgery detection algorithms, make CASIA A high precision indicates that the model has fewer false
2.0 a valuable resource for researchers striving to improve positives.
the robustness and accuracy of their methodologies.
MICC-F2000 Claiming the title of the largest publicly True Positives
Precision =
available dataset for copy-move forgery detection, MICC- True Positives + False Positives
F2000 is a rich resource comprising 2000 images. This
dataset spans a diverse spectrum of complexity levels and 6.2.2 Recall (sensitivity or true positive rate)
manipulation scenarios, making it a comprehensive bench-
mark for algorithm evaluation. Researchers engage with Recall measures the ability of the model to capture all positive
MICC-F2000 to explore and address the challenges posed instances. It is calculated as the ratio of true positive predic-

123
International Journal of Multimedia Information Retrieval (2025) 14:8 Page 7 of 11 8

6.2.5 Intersection over union (IoU)

IoU evaluates the spatial overlap between predicted and

ground truth regions. It is calculated as the intersection area
divided by the union area of the predicted and ground truth
regions.

Intersection Area
IoU =
Union Area

High IoU values indicate accurate localization.

6.3 Results

In our initial phase of experimentation, we rigorously

assessed the performance of diverse models for image clas-
sification across five distinct datasets: CASIA v1.0, CASIA
v2.0, MICC-F2000, MICC-F600, and Columbia. The pri-
Fig. 5 Authentic and forged image samples of all Dataset mary metrics considered were training accuracy (Tr. Acc),
test accuracy (Te Acc), training loss (Tr Loss), and test
loss (Te Loss). Additionally, precision, recall, and F1-score
tions to the sum of true positives and false negatives. High were calculated to provide a comprehensive evaluation. The
recall indicates that the model effectively identifies positive ViT-l32 model consistently demonstrated superior perfor-
instances. mance across all datasets, achieving high accuracy, robust
precision, and notable F1-score. The experimental results
True Positives obtained are shown in Table 3. For example, on CASIA
Recall = v1.0, ViT-l32 exhibited a remarkable training accuracy of
True Positives + False Negatives
98%, test accuracy of 93%, and precision, recall, and F1-
score values of 91.18, 90.02, and 94.55%, respectively.
6.2.3 F1-score
Similar commendable results were observed across CASIA
v2.0, MICC-F2000, MICC-F600, and Columbia datasets,
The F1-score is the harmonic mean of precision and recall,
highlighting the model’s versatility and efficacy in diverse
providing a balanced measure of a model’s performance. It
scenarios.
ranges from 0 to 1, where 1 indicates perfect precision and
Sample localized results for each dataset have been suc-
recall.
cessfully incorporated, enriching our findings with visual
  representations of the segmentation outputs. These examples
Precision × Recall
F1-Score = 2 × provide a qualitative glimpse into the localization accuracy
Precision + Recall achieved by our model, demonstrating its ability to precisely
identify and delineate objects of interest. Figure 6 shows the
6.2.4 Accuracy forged image with the predicted mask and its probability map
from the CASIA v1.0 dataset. The forged image with the pre-
Accuracy measures the overall correctness of the model’s dicted mask and its probability map from the CASIA v1.0,
predictions. It is calculated as the ratio of correct predictions VASIA v2.0, MICC-F2000, MICC-F6000, and Columbia
(true positives and true negatives) to the total number of pre- dataset are shown in Figs. 6, 7, 8, 9, and 10.
dictions. The comprehensive classification and localization results
have been organized in Tables 3 and 4 respectively, offering
True Positives + True Negatives readers an easily navigable summary of our experimental
Accuracy =
Total Predictions outcomes. Building on the success of ViT-l32 in classifica-
tion, we transitioned to the localization experiments using
Effectively balancing accuracy and computational efficiency the Segment Anything model. Two loss functions, Dice
in classification and localization tasks requires a combination Focal Loss and Focal Loss, were employed, and their per-
of model selection, optimization techniques, and hardware formance was evaluated across varying epochs. In the Dice
considerations. Focal Loss experiments, notable outcomes were observed.

123
 8 Page 8 of 11 International Journal of Multimedia Information Retrieval (2025) 14:8

Table 3 Image classification performance results on diverse models

Dataset Model Tr. Acc Te Acc Tr Loss Te Loss Precision Recall F1-score

CASIAv1.0 R50-ViT-l16 0.97 0.89 0.03 0.05 0.87041 0.85172 0.86044

ViT-b16 0.92 0.91 0.01 0.1 0.90075 0.93172 0.91522
ViT-l16 0.98 0.92 0.04 0.12 0.88889 0.82 0.85368
ViT-l32 0.98 0.93 0.08 0.15 0.91184 0.90025 0.94559
ViT-b32 0.94 0.92 0.04 0.12 0.83333 0.90109 0.86578
CASIAv2.0 R50-ViT-l16 0.97 0.92 0.06 0.08 0.87041 0.86172 0.86667
ViT-b16 0.96 0.93 0.02 0.07 0.90075 0.94172 0.92044
ViT-l16 0.95 0.91 0.09 0.11 0.88889 0.82 0.85368
ViT-l32 0.98 0.94 0.03 0.1 0.91184 0.90025 0.90559
ViT-b32 0.97 0.94 0.05 0.07 0.83333 0.91109 0.87189
MICC-F2000 R50-ViT-l16 0.95 0.91 0.04 0.06 0.865 0.845 0.85417
ViT-b16 0.91 0.87 0.02 0.07 0.89 0.915 0.90244
ViT-l16 0.94 0.9 0.07 0.13 0.872 0.803 0.83636
ViT-l32 0.96 0.91 0.03 0.09 0.9 0.92 0.90948
ViT-b32 0.93 0.89 0.05 0.08 0.828 0.89 0.8573
MICC-F600 R50-ViT-l16 0.96 0.91 0.15 0.07 0.878 0.85 0.86364
ViT-b16 0.94 0.93 0.06 0.08 0.895 0.92 0.90741
ViT-l16 0.97 0.94 0.08 0.14 0.88 0.818 0.84783
ViT-l32 0.98 0.94 0.04 0.11 0.9184 0.935 0.92647
ViT-b32 0.95 0.93 0.07 0.1 0.837 0.91 0.87234
Columbia R50-ViT-l16 0.94 0.91 0.03 0.05 0.852 0.833 0.84247
ViT-b16 0.9 0.88 0.01 0.06 0.885 0.922 0.90385
ViT-l16 0.93 0.92 0.06 0.12 0.86 0.793 0.825
ViT-l32 0.95 0.92 0.02 0.08 0.893 0.91 0.90196
ViT-b32 0.92 0.91 0.04 0.07 0.825 0.883 0.85319

Fig. 6 Forged image from CASIAv1.0 Dataset, with Predicted Mask Fig. 7 Forged image from CASIAv2.0 Dataset, with Predicted Mask
and Probability Map and Probability Map

123
International Journal of Multimedia Information Retrieval (2025) 14:8 Page 9 of 11 8

Fig. 8 Forged image from MICC-F2000 Dataset, with Predicted Mask

and Probability Map Fig. 10 Forged image from Columbia Dataset, with Predicted Mask
and Probability Map

Table 4 Localization with different Loss

Loss Function Dataset Epoch Loss IOU

Dice Focal CASIA v1.0 20 1.048 0.9052

CASIA v2.0 20 1.0956 0.92844
MICC-F2000 25 1.141 0.8638
MICC-F600 29 1.288 0.8500
Columbia 12 0.8798 0.8896
Focal CASIA v1.0 10 0.6290 0.9089
CASIA v2.0 10 0.6843 0.9371
MICC-F2000 13 0.8312 0.8938
MICC-F600 15 1.057 0.8507
Columbia 8 0.4268 0.9179

and manipulations requires robust testing frameworks like

Data Augmentation & Transformation-Based Testing that
Fig. 9 Forged image from MICC-F600 Dataset, with Predicted Mask include Geometric Transformations, Color & Texture Varia-
and Probability Map tions, and Real-World deployment testing.

For instance, on CASIA v1.0, after 20 epochs, the mean 6.4 Result comparison
IOU reached 0.9052. Subsequently, employing Focal Loss
yielded promising results, surpassing Dice Focal Loss in In our experiments, the ViT-l32 model outperformed other
certain scenarios. For instance, on CASIA v1.0, after 10 Vision Transformer configurations like R50-ViT-l16, ViT-
epochs, the mean IOU was 0.9089. These results collec- b16, ViT-l16, and ViT-b32 due to its superior feature repre-
tively underscore the efficacy of our chosen model and sentation and deeper architecture. With a larger patch size
methodology in both classification and localization tasks, and enhanced capability to capture global context, it can
contributing valuable insights to the broader field of deep detect subtle image forgeries more comprehensively than
learning for image analysis. Ensuring an algorithm performs ever before, achieving training accuracy of 97.5% and vali-
reliably on complex datasets with intricate transformations dation accuracy of 94.5% on the CASIA v2.0 dataset, which

123
 8 Page 10 of 11 International Journal of Multimedia Information Retrieval (2025) 14:8

Table 5 Comparison of existing models with the proposed model on 7 Conclusion

various datasets
Dataset Model Precision Recall F1-score This study delves into image classification and object local-
ization, employing advanced models and loss functions. We
CASIAv1.0 Zhang et al. [29] – – 0.2764
opted for the Vision Transformer model, specifically ViT-
Tyagi et al. [25] 0.906 0.948 0.926
l32, for image classification, achieving impressive accuracy
Kadam et al. [13] 0.61 0.68 0.64
across diverse datasets. Our proposed model outperformed
Proposed model 0.9118 0.9002 0.9455
existing methods on CASIA v1.0 and v2.0 datasets and
CASIAv2.0 Chen et al. [5] – – 0.758
demonstrated competitive performance on MICC-F2000,
Deng et al. [7] – – 0.634 MICC-F600, and the Columbia dataset. For object localiza-
Tyagi et al. [25] 0.64 0.76 0.72 tion, we employed the Segment Anything Model (SAM) with
Zhang et al. [29] 0.7071 0.5758 0.6348 Focal Loss, achieving exceptional Intersection over Union
Proposed model 0.9108 0.9010 0.9042 (IOU) scores that surpassed existing methods. The results
MICC-F2000 Tyagi et al. [25] 0.66 0.50 0.77 highlight the effectiveness of our model in both tasks, posi-
Proposed model 0.900 0.920 0.9094 tioning it as a robust and versatile solution for forensic image
MICC-F600 Tyagi et al. [25] 0.70 0.72 0.65 analysis. Future research avenues may explore ensemble
Amiri et al. [1] 0.83 0.83 0.83 methods or alternative architectures for further enhance-
Proposed model 0.9184 0.935 0.9264 ments. Overall, our research contributes significantly to the
Columbia Zhang et al. [29] 0.9189 0.7096 0.8004 field of digital forensics, providing a potent tool for accurate
Tyagi et al. [25] 0.65 0.62 0.63 image classification and object localization with applications
Proposed model 0.893 0.910 0.901 in image forensics investigation.
Acknowledgements We would like to express our gratitude to the Uni-
versity of Hyderabad for providing us with the research environment
under IoE project [grant number:UoH-IoE-RC5-22-012].
Table 6 Comparison of existing models with the proposed model for
localization Author Contributions All the authors contributed to the study’s con-
Loss Function Dataset IOU ception and design.

Xu et al. [27] CASIA v1.0 0.816 Funding No funding is available for the Article Processing Charge (or
Tan et al. [23] CASIA v2.0 0.0.8394 publication fee).
Ganapathi et al. [8] CASIA v2.0 0.8816 Data Availability The authors confirm that the data supporting the
– MICC-F2000 – findings of this study are available within the article [and/or] its supple-
– MICC-F600 – mentary materials.
Ganapathi et al. [8] Columbia 0.9056 Code availability Available with the authors and provided to the
Proposed CASIA v1.0 0.9089 research community based on request and mutual understanding.
CASIA v2.0 0.9371
MICC-F2000 0.8938 Declarations
MICC-F600 0.8507
Columbia 0.9179 Conflict of interest There are no Conflict of interest.

References
served as the basis for future experiments. Table 5 shows 1. Amiri E, Mosallanejad A, Sheikhahmadi A et al Copy-move
the results of image classification on the proposed model forgery detection using eoa, dwt and dct. Pamukkale Univ J Eng
compared rigorously against state-of-the-art methods. Mov- Sci, 1000(1000):0–0
ing towards object localization, we employed the Segment 2. Asiri AA, Shaf A, Ali T, Pasha MA, Aamir M, Irfan M, Alqahtani S,
Alghamdi AJ, Alghamdi AH, Alshamrani AFA (2023) Advancing
Anything Model (SAM) with Focal Loss for varying epochs. brain tumor classification through fine-tuned vision transformers:
In Table 6, the proposed model is shown to be superior to a comparative study of pre-trained models. Sensors 23(18):7913
existing methods in terms of IOU. 3. Bayram S, Sencar HT, Memon N (2009) An efficient and robust
In summary, our proposed model consistently outper- method for detecting copy-move forgery. In: 2009 IEEE Interna-
tional Conference on Acoustics, Speech and Signal Processing, pp.
formed existing methods in image classification and show-
1053–1056. IEEE
cased superior object localization capabilities across diverse 4. Bi Y, Xue B, Mesejo P, Cagnoni S, Zhang M (2022) A survey on
datasets, affirming its efficacy in forensic image analysis. evolutionary computation for computer vision and image analysis:

123
International Journal of Multimedia Information Retrieval (2025) 14:8 Page 11 of 11 8

past, present, and future trends. IEEE Trans Evolutionary Comput 19. Milletari F, Navab N, Ahmadi SA (2016) V-net: Fully convolutional
27(1):5–25 neural networks for volumetric medical image segmentation. In:
5. Chen B, Qi X, Zhou Y, Yang G, Zheng Y, Xiao B (2020) Image 2016 fourth international conference on 3D vision (3DV), pp. 565–
splicing localization using residual image and residual-based fully 571. IEEE
convolutional network. J Vis Commun Image Represent 73:102967 20. Patil G, Palaiahnakote S, Gornale SS, Lopresti DP (2024) Altered
6. Chen CF (Richard), Fan Q, Panda R (2021) Crossvit: Cross- handwritten text detection in document images using deep learning.
attention multi-scale vision transformer for image classification. In: Int J Pattern Recognit Artific Intell 38(03):2452006
Proceedings of the IEEE/CVF International Conference on Com- 21. Patil G, Shivakumara P, Gornale SS, Pal U, Blumenstein M (2023)
puter Vision (ICCV), pp. 357–366 A new robust approach for altered handwritten text detection. Mul-
7. Deng L, Peng J, Deng W, Liu K, Cao Z, Wang W (2022) A dual- timed Tools Appl 82(14):20925–20949
stream input faster-cnn model for image forgery detection. In: 22. Shi YQ, Chen C, Chen W (2007) A natural image model approach
International Conference on Mobile Networks and Management, to splicing detection. In: Proceedings of the 9th workshop on Mul-
pp. 105–115. Springer timedia & security, pp 51–62
8. Ganapathi II, Javed S, Ali SS, Mahmood A, Vu N-S, Werghi N 23. Tan Y, Li Y, Zeng L, Ye J, Li X, et al (2023) Multi-scale target-aware
(2022) Learning to localize image forgery using end-to-end atten- framework for constrained image splicing detection and localiza-
tion network. Neurocomputing 512:25–39 tion. arXiv preprint arXiv:2308.09357
9. Guo M-H, Xu T-X, Liu J-J, Liu Z-N, Jiang P-T, Mu T-J, Zhang S-H, 24. Taneja N, Bramhe VS, Bhardwaj D, Taneja A (2023) Understand-
Martin RR, Cheng M-M, Hu S-M (2022) Attention mechanisms in ing digital image anti-forensics: an analytical review. Multimedia
computer vision: a survey. Comput Vis Media 8(3):331–368 Tools and Applications, pp. 1–22
10. Han D, Liu Q, Fan W (2018) A new image classification method 25. Tyagi S, Yadav D (2023) Forensicnet: modern convolutional neural
using cnn transfer learning and web data augmentation. Expert Syst network-based image forgery detection network. J Forensic Sci
Appl 95:43–56 68(2):461–469
11. Han K, Wang Y, Chen H, Chen X, Guo J, Liu Z, Tang Y, Xiao A, 26. Xiao B, Wei Y, Bi X, Li W, Ma J (2020) Image splicing forgery
Xu C, Xu Y, Yang Z, Zhang Y, Tao D (2023) A survey on vision detection combining coarse to refined convolutional neural network
transformer. IEEE Trans Pattern Anal Mach Intell 45(1):87–110 and adaptive clustering. Inf Sci 511:172–191
12. He Y, Li Y, Chen C, Li X (2023) Image copy-move forgery detec- 27. Xu Y, Muhammad I, Aiqing F, Jiangbin Z (2023) Multi-scale
tion via deep cross-scale patchmatch. In: 2023 IEEE International attention network for detection and localization of image splicing
Conference on Multimedia and Expo (ICME), pages 2327–2332. forgery. IEEE Trans Instrum Measur
IEEE 28. Yang B, Sun X, Guo H, Xia Z, Chen X (2018) A copy-move
13. Kadam KD, Ahirrao S, Kotecha K et al (2022) Efficient approach forgery detection method based on cmfd-sift. Multimed Tools Appl
towards detection and identification of copy move and image splic- 77:837–855
ing forgeries using mask r-cnn with mobilenet v1. Comput Intell 29. Zhang J, Wang H, He P (2023) Dual-branch multi-scale densely
Neurosci connected network for image splicing detection and localization.
14. Kingma DP, Ba J (2014) Adam: A method for stochastic optimiza- Signal Process Image Commun 119:117045
tion. arXiv preprint arXiv:1412.6980
15. Kirillov A, Mintun E, Ravi N, Mao H, Rolland C, Gustafson L, Xiao
T, Whitehead S, Berg Alexander C, Lo WY et al (2023) Segment
Publisher’s Note Springer Nature remains neutral with regard to juris-
anything. arXiv preprint arXiv:2304.02643
dictional claims in published maps and institutional affiliations.
16. Lai Y, Luo Z, Yu Z (2023) Detect any deepfakes: Segment any-
thing meets face forgery detection and localization. In: Chinese
Springer Nature or its licensor (e.g. a society or other partner) holds
Conference on Biometric Recognition, pp. 180–190. Springer
exclusive rights to this article under a publishing agreement with the
17. Lin TY, Goyal P, Girshick R, He K, Dollár P (2017) Focal loss for
author(s) or other rightsholder(s); author self-archiving of the accepted
dense object detection. In Proceedings of the IEEE international
manuscript version of this article is solely governed by the terms of such
conference on computer vision, pp. 2980–2988
publishing agreement and applicable law.
18. Mehrjardi FZ, Latif AM, Zarchi MS, Sheikhpour R (2023) A survey
on deep learning-based image forgery detection. Pattern Recogni-
tion, pp. 109778

123