CHAPTER 1

INTRODUCTION

The integration of near-infrared (NIR) and visible spectral imaging in the broad field of imaging
and computer vision is emerging as an exciting frontier Rabhate, where special attention has
been paid to bridging the near-infrared and visible spectrum data between will be used.
At its core, image fusion represents the amalgamation of multiple images captured from diverse
sources or modalities to generate a singular, comprehensive image that encapsulates the
strengths of each constituent image. This process holds immense potential in enhancing visual
perception, augmenting the quality of information extracted from imagery, and facilitating
more accurate and insightful analysis. While various modalities contribute to image fusion, the
integration of NIR and visible spectrum imagery emerges as a particularly compelling avenue
due to the inherent characteristics and complementary nature of these spectral bands.
The Near-Infrared spectrum, often referred to as the "near-invisible" spectrum, extends beyond
the visible spectrum, with wavelengths ranging from approximately 700 to 1400 nanometres.
Despite being imperceptible to the human eye, NIR radiation interacts uniquely with objects
and materials, providing valuable insights into their composition, structure, and characteristics.
On the other hand, the visible spectrum encompasses wavelengths that are perceptible to the
human eye, spanning from approximately 400 to 700 nanometres. This spectrum is
fundamental to human vision and serves as the primary source of visual information in our
environment.

1.1. Overview of Image Fusion

Image fusion occupies an important place in image fusion, which aims to combine information
from multiple sources to create composite images that provide a comprehensive understanding
of the scene. Here is an overview of the image fusion process.
Image acquisition: Multiple images of the same scene are captured using different sensors,
techniques or imaging techniques, often varying in resolution, perspective, and detail
Preprocessing: Before fusion, input images undergo preprocessing steps such as noise
reduction, geometric correction, and radiometric calibration to ensure accuracy and image
quality
Image registration: Image registration is used to properly align input images, ensuring that the
corresponding pixels in images represent the same viewing area, of which necessary for perfect
fusion.
Feature extraction: Relevant features are extracted from each input image, such as edges,
textures, or colors, needed for an intended application.

1
Fusion Techniques: Different fusion techniques are used depending on application
requirements and input image characteristics. The most common methods are:
- Approved search results
- Solid-color-saturation fusion
- Analysis of the main features
- Wavelet transform based fusion
Post-processing: post-processing techniques such as contrast enhancement, sharpening, and
noise filtering can be used to further enhance image quality after fusion
Evaluation: Finally, the quality of the fused image is evaluated using metrics such as visual
inspection, a quantitative measure of image quality. (e.g., signal-to-noise ratio, entropy, etc.)

1.2. Objective of Image Fusion

The main goal of image fusion is to produce composite images that preserve thermal radiation
information from infrared images and combine texture and texture quality information from
visible images together This convergence of both components enables hybrid images to provide
enhanced definition and fine detail.

The objectives of image fusion can be summarized as follows:

Enhancing Information Content: The primary goal of image fusion is to fuse complementary
data from multiple images to create a composite image with enhanced content. By integrating
information from different sources or modalities, image fusion aims to provide a more
comprehensive representation of the underlying scene than any single input image can offer.
Improving Image Quality: Another objective is to improve the overall quality of the image by
reducing noise, enhancing contrast, and increasing spatial or spectral resolution. Fusion
techniques aim to mitigate artifacts and distortions present in individual input images, resulting
in a fused image that is visually appealing and more informative.
Preserving Relevant Features: Image fusion seeks to preserve important features and details
present in the input images while discarding redundant or irrelevant information. This ensures
that the fused image retains essential characteristics of the scene, making it suitable for various
analysis and interpretation tasks.
Facilitating Decision Making: Image fusion aims to facilitate decision making by providing a
consolidated view of the scene that highlights critical information. By integrating data from
multiple sources, fusion techniques enable more informed decision making in applications such
as surveillance, target recognition, and medical diagnosis.
Optimizing Computational Efficiency: While achieving high-quality fusion results is
important, it is also essential to consider computational efficiency, especially in real-time or

2
resource-constrained applications. Fusion algorithms aim to strike a balance between accuracy
and computational complexity to ensure practical feasibility.
By addressing these objectives, image fusion techniques contribute to a wide range of fields,
including remote sensing, medical imaging, computer vision, and surveillance, enabling more
effective analysis and interpretation of visual data.

1.3. Evolution of Fusion Algorithms

Figure 1.1. shows the workflow of the proposed method it denotes over time, various
fusion algorithms have been developed, each with the goal of improving the quality and
information richness of fused images. These algorithms employ different techniques, such as
two-scale decomposition, to break down source images into components emphasizing specific
qualities before merging them to create the final composite image.

Fig 1.1. Workflow of the proposed method

Traditional Methods:
Early fusion algorithms relied on traditional signal processing techniques and mathematical
models to merge information from different image sources. These methods often employed
simple averaging or weighted averaging to combine pixel values from input images. While
effective in some cases, traditional fusion techniques lacked the ability to capture complex
relationships and subtle details present in the images.
Two-Scale Decomposition:
One significant advancement in fusion algorithms is the introduction of the two-scale
decomposition method. This technique revolutionized image fusion by breaking down source
images into multiple components, each emphasizing specific qualities or aspects. By
decomposing images into base and detail layers, two-scale decomposition allows for more
precise fusion of information, resulting in enhanced image quality and interpretability.

3
Integration of Optimization Models:
As fusion algorithms evolved, researchers began integrating optimization models into the
fusion process. Optimization-based approaches utilize mathematical optimization techniques
to iteratively refine the fusion process, ensuring optimal preservation of important image
features. These models often incorporate regularization terms to balance fidelity to input
images with the creation of a coherent composite image.
Deep Learning Revolution:
The rise of deep learning has changed the landscape of image fusion in CNN have demonstrated
sophisticated feature extraction and image recognition. By leveraging largescale datasets and
sophisticated network architectures, deep learning-based fusion algorithms can learn complex
mappings between input images and generate high-quality fused outputs.
Hybrid Approaches:
The latest trend in fusion algorithm development involves hybrid approaches that combine the
strengths of traditional methods with the power of deep learning. These hybrid models often
incorporate two-scale decomposition techniques as a preprocessing step before feeding the
decomposed images into deep neural networks for further processing. By integrating deep
learning with traditional fusion techniques, hybrid approaches aim to achieve superior fusion
results with enhanced efficiency and interpretability.

1.4. Evolution of Fusion Algorithms

Recent advancements in deep learning have sparked interest in integrating these methodologies
with traditional image fusion techniques like two-scale decomposition. This hybrid approach
seeks to leverage deep learning's powerful feature extraction capabilities while benefiting from
the interpretability and efficacy of traditional methods.
CNNs, are adept at obtaining hierarchical representations of image features. By training on a
wider range of data sets, CNNs automatically acquire suitable features for the fusion task,
including edge, texture, and semantic information. These acquisitions serve as guidelines for
the fusion process, facilitating the efficient integration of information from multiple images.

Finally, the internal fusion models are enabled by deep learning, allowing the mapping to be
learned directly from input images to fusion output. Unlike traditional methods based on
manual fusion rules or algorithms, these models collect fusion techniques directly from the
data, and can produce optimal and context-sensitive fusion results. By leveraging the
capabilities of deep learning, image fusion techniques can achieve state-ofthe-art results across
various applications, offering improved quality, adaptability, and efficiency in integrating
information from multiple images.

4
1.5. Role of Autoencoders
Autoencoder architectures play a crucial role in the deep learning stage of the fusion process.
These architectures are adept at feature extraction and image reconstruction, enabling the
model to capture fine details and enhance the overall quality of the fusion process.

Learning Representative Features:

Autoencoders are trained to learn representative features from input data by encoding It maps
the data to a lower hidden location. In the context of image fusion, autoencoders are capable of
capturing essential characteristics of both visible and infrared images, including texture, edges,
and structural patterns. By extracting these features, autoencoders facilitate the fusion process
by preserving relevant information while discarding noise and irrelevant details.
Non-linear Feature Extraction:
Unlike traditional linear methods, autoencoders employ non-linear activation functions and
multiple layers to extract complex features from input images. This non-linearity allows
autoencoders to capture intricate relationships between pixels and uncover high-level
representations that may not be discernible through linear transformations alone. As a result,
autoencoder-based fusion models can effectively capture the nuanced differences between
visible and infrared images, leading to more accurate and informative fusion outcomes.
Dimensionality Reduction:
Autoencoders provide important advantages by compressing input data to a lower hidden value,
thus reducing dimensionality.. This compression of input image information by autoencoders
enables fusion patterns to form a compact and manageable representation, thus reducing
computational complexity and memory requirements Such dimensionality reduction not only
speeds up the fusion process but also over -. It also protects against fittings, increasing the
generalizability of the model

Furthermore, autoencoders are adept at learning complex images. Trained with unsupervised
learning methods, they can access intuitive representations of the input without explicit
labeling. During training, autoencoders again adjust parameters to reduce reconstruction errors
between input and output images. This iterative process leads to the learning of materials that
are sensitive to changes in lighting, mood, and image conditions, thus ensuring that the learned
images are robust and transferable in fusion tasks and dataset in various fieldsAutoencoders
offer the important advantage of reducing dimensionality by encoding input data into a low-
dimensional latent space. This compression of input image information by autoencoders
enables fusion patterns to form a compact and manageable representation, thereby reducing
computational complexity and memory requirements Such dimensionality reduction not only
speeds up the fusion process but also over -. It also protects against fittings, increasing the
generalizability of the model

5
Furthermore, autoencoders are adept at learning complex images. Trained with unsupervised
learning methods, they can access intuitive representations of the input without explicit
labeling. During training, autoencoders again adjust parameters to reduce reconstruction errors
between input and output images. This iterative process leads to the learning of materials that
are sensitive to changes in lighting, mood, and image conditions, thus ensuring that the learned
images are robust and transferable in fusion tasks and dataset in various fields.
Adaptability to Complex Data Distributions:
Autoencoders are highly adaptable to complex data distributions and can effectively capture
the underlying structure of diverse image datasets. This adaptability is particularly
advantageous in image fusion, where the input images may exhibit varying degrees of
complexity and variability. By learning to encode and reconstruct images from different
modalities, autoencoders can adapt to the inherent characteristics of visible and infrared
imagery, ensuring robust performance across a wide range of fusion scenarios.

Interpretability and Visualization:

Another important aspect of autoencoders is their interpretability, which refers to the model's
ability to provide insights into the learned representations and fusion process. Researchers can
visualize the encoded features and reconstructed images produced by autoencoders to gain a
better understanding of how the model combines information from different modalities. This
interpretability allows researchers to debug and fine-tune the fusion algorithm more effectively,
leading to improved performance and reliability.
Performance Evaluation Metrics:
Several quality metrics are used to evaluate the efficiency of fusion algorithms, including
standard deviation, mean gradient, spatial frequency, entropy, and sum of correlations of
differences These metrics provide a quantitative evaluation of the fusion process. method's
ability to preserve important image information while enhancing overall image quality.

1.6. Proposed Approach: Algorithm Unraveling

Inspired by the principle of algorithm unraveling, the proposed method extends
optimizationbased decomposition algorithms into model-driven deep neural networks. This
novel approach combines traditional two-scale decomposition with deep learning techniques
to achieve superior image fusion results. The Algorithm Unraveling is a method designed to
improve how we combine multiple images into one, ensuring we get the best possible result.
Here's a breakdown in simple terms:
Getting the Right Details: By picking out the important parts from each image. This could be
edges, textures, or colures that help us understand what we're seeing.

6
Looking at Different Scales: Imagine looking at a picture up close versus far away. Algorithm
Unraveling considers both views. It breaks down the images into different scales or levels of
detail to make sure we don't miss anything.
Mixing Everything Together: Algorithm Unraveling doesn't just combine images at the same
scale. It mixes information from different scales to create a complete picture. This ensures we
get a clear view of both small details and the big picture.
Being Smart About It: Instead of using the same method every time, adapts. It looks at the
images and decides the best way to combine them based on what's in the pictures and what we
want to see in the final result.
Understanding What's Important: Algorithm Unraveling doesn't treat everything in the images
equally. It focuses on the important stuff, like objects or features we're interested in. This helps
ensure those things stand out in the final combined image.
Learning as It Goes: Algorithm Unraveling gets better over time. It learns from examples of
how images should be combined, so it can make smarter decisions in the future and create
better results.
Making Sure It's Good: After combining the images, Algorithm Unraveling checks to see if the
final result is good enough. It looks at things like how sharp the image is and how clear the
details are. If needed, it adjusts its methods to make the image even better.

1.7. Workflow of Algorithm Unraveling:

The workflow of the algorithm unraveling model has several steps, including input source
handling, decomposition, feature merging, and image fusion. Throughout the training phase,
deep networks based on auto-encoders are trained using data acquired from TNO and FLIR.
This training process enables the model to skillfully combine base-width features from infrared
and visible images.
1. Input Sources: The workflow begins with the acquisition of input images, typically
consisting of visible and infrared images captured by different sensors or modalities. These
input images serve as the basis for the fusion process and contain valuable information that
needs to be preserved and enhanced through fusion.
2. Decomposition Stage: In the decomposition stage, the input images undergo a two-scale
decomposition process, where they are divided into base and detail components. The applied
decomposition method removes high-frequency detail and low-basis information from the
input images. This approach allows the fusion model to prioritize the conservation of critical
resources while removing noise and redundant information.
3. Feature Merging: After the decomposition process, base detail feature maps extracted
from visible and infrared images are merged using a fusion layer. This merging process
combines the distinct information preserved in each decomposed image, creating a richer and
more detailed representation of the original scenes. The fusion layer effectively integrates the

7
complementary information from both modalities, leading to enhanced image quality and
information richness.
4. Image Fusion: After the completion of function merging, the fused image is produced
thru the decoder component of the Algorithm Unraveling model. This decoder is accountable
for reconstructing the fused photo from the merged function maps, with the intention of keeping
important info from both the seen and infrared pictures whilst mitigating artifacts and
inconsistencies. The resulting fused photo serves because the final output of the fusion method,
prominent by means of greater first-rate, clarity, and informativeness in contrast to the unique
enter images.
5. The evaluation and validation process is an important part of the fusion process, which
ensures the quality and efficiency of the algorithm unraveling model. Various quantitative
parameters are used to analyze the fused images, providing an objective measure of
performance and facilitating comparison with other fusion techniques. The standard deviation
acts as the main metric, indicating the variation of pixel values in the fused images. A higher
standard deviation usually indicates increased variability and can improve image quality.
Averaged, the gradient measures the variation in pixel intensity across the image, providing
insight into the overall image sharpness and information preserved after fusion. Spatial-
frequency analysis helps to analyze the frequency content of the fused images, and helps to
understand which high-frequency and low-frequency components are important for image
clarity and shape

6. Entropy is a metric used to quantify the amount of randomness or uncertainty in an

image, and it provides information about the complexity and abundance of the details behind
fusion. Higher entropy values tend to indicate greater content and visual complexity.

7. In addition, the integrated correlation between the differences is a comprehensive

metric, which assesses the similarity between the fused image and the original embedded image
This metric takes into account local and global differences, and provides general analysis of
mixture properties. Additionally, validation experiments are conducted using benchmark
datasets such as TNO, FLIR, and NIR sources to ensure the robustness and generalizability of
the proposed fusion methodology across different imaging scenarios.
8. Iterative Refinement and Optimization: The Algorithm Unraveling workflow is
designed to be iterative, allowing researchers to refine and optimize the fusion algorithm based
on evaluation results and feedback. By analyzing the performance of the model on various
datasets and scenarios, researchers can identify areas for improvement and fine-tune the model
parameters, architectures, and training strategies to enhance fusion quality, efficiency, and
robustness.

Performance Evaluation:

8
Performance evaluation of the Algorithm Unraveling method is conducted using benchmark
datasets, including TNO and FLIR, demonstrating superior performance and efficiency
compared to state-of-the-art fusion techniques. The results highlight the efficacy of the hybrid
approach in preserving important image information while enhancing overall image quality.
Performance evaluation plays a critical role in assessing the effectiveness and reliability of
image fusion techniques. Here are some key points regarding performance evaluation in the
context of the Algorithm Unraveling model

Quantitative Metrics:
The unraveling algorithm uses various quantitative metrics to unbiasedly assess the quality of
the fused image. These metrics include standard deviation (SD), average gradient (AG), spatial
frequency (SF), and entropy (EN). Each of these metrics provides unique insights into different
aspects of fused image quality, including contrast, sharpness, texture fidelity, and overall
information richness and sum of difference correlations (SCD) Each of these metrics
contributes to a different view of different aspects of a it’s about fused image quality, including
contrast, sharpness, texture fidelity, and overall richness of information

Benchmark Datasets:
To ensure the robustness and generalizability of the proposed fusion methodology, Algorithm
Unraveling conducts performance evaluation on benchmark datasets such as TNO, FLIR, and
NIR sources. These datasets contain diverse images captured under various imaging conditions
and scenarios, allowing researchers to assess the model's performance across different
modalities, resolutions, and environmental factors.

Compared to state-of-the-art methods, it allows to evaluate the performance and efficiency of

the algorithm unraveling model:
Algorithm Unraveling benchmarks its performance against state-of-the-art fusion methods to
demonstrate its superiority and effectiveness. By comparing the quality metrics of fused images
generated by Algorithm Unraveling with those produced by other fusion techniques,
researchers can assess the relative performance and identify areas where Algorithm Unraveling
outperforms or falls short compared to existing methods.

Efficiency and Computational Complexity:

In addition to image quality metrics, Algorithm Unraveling evaluates the efficiency and
computational complexity of the fusion process. This includes assessing factors such as
processing time, memory usage, and scalability to large datasets. By quantifying the
computational resources required for fusion, researchers can determine the practical feasibility
and scalability of the Algorithm Unraveling model for real-world applications.

9
 CHAPTER 2
Literature Survey
Image fusion, particularly combining visible and infrared images, has garnered significant
attention in image processing research due to its potential for enhancing image quality and
information richness. Various methodologies have been proposed, aiming to achieve superior
performance, interpretability, and reproducibility. Leveraging insights from existing literature,
recent advancements, and diverse fusion techniques, we present a comprehensive literature
survey highlighting key contributions, methodologies, and future directions in the field of
image fusion.
Hybrid Approaches:
Many recent studies have adopted hybrid approaches combining traditional methods like
twoscale decomposition with deep learning techniques. These approaches aim to leverage the
strengths of both methodologies, such as potent feature extraction capabilities of deep learning
and interpretability of traditional methods [1, 2].
Autoencoders in Deep Learning Stage:
The utilization of autoencoder architectures in the deep learning stage has shown promise in
capturing fine details and enhancing image quality. Training these autoencoders on datasets
like TNO and FLIR aids in effective feature extraction and reconstruction [3].
Algorithm Unraveling:
Algorithm unraveling introduces a model-driven deep neural network approach, integrating
optimization models into the neural network architecture. This approach enhances efficiency
and performance, as demonstrated by the Algorithm Unraveling Image Fusion Model [4].
Evaluation Metrics:
A variety of quantitative metrics, together with trendy deviation, average gradient, spatial
frequency, entropy, and sum of the correlations of differences, are hired to evaluate the pleasant
of fused images. These metrics provide valuable insights into the effectiveness of fusion
techniques. [5].
Framework and Workflow:
The proposed framework involves a systematic workflow encompassing input sources,
decomposition, feature merging, and image fusion. This structured approach ensures robustness
and generalizability of the fusion methodology [6].
Efficiency Model:
Optimization-based techniques, such as gradient descent, are employed to enhance the
efficiency of image fusion. These models focus on resolving specific challenges, such as
acquiring low-frequency background information [7].

10
Evaluation and Comparison:
Comparative analyses of fusion techniques against state-of-the-art methods using benchmark
datasets like TNO and FLIR provide insights into the effectiveness and applicability of fusion
algorithms [8].
Results and Conclusion:
Superior performance and efficiency of the proposed fusion model are demonstrated through
quantitative metrics and qualitative comparisons with existing methods. The conclusion
emphasizes the significance of marrying classical optimization with deep learning for
promising real-world applications [9].
Future Directions:
Future research directions include extending the proposed approach to other multi-modal fusion
applications, exploring fusion techniques combining conventional and deep learningbased
methods, and addressing specific challenges in infrared and visible image fusion [10].
In summary, the literature survey highlights the diverse methodologies, evaluation metrics,
frameworks, and future prospects in the field of image fusion, providing a comprehensive
overview for further research and development.

11
 CHAPTER 3
ARCHITECTURE AND DESIGN

3.1. Network framework in the training phase

Fig 3.1. the architecture consists of three main components: Encoder Base, Encoder Detail, and
Decoder.
Encoder Base and Encoder Detail are responsible for extracting base and detail features from
input images, respectively.

The Decoder combines the base and detail features to reconstruct the fused image.
During training, the network is optimized to minimize a loss function that includes both
pixelwise reconstruction error and structural similarity.

Fig 3.1. Network framework in the training phase

comprises three primary components: Encoder Base, Detail Encoder, and Decoder. Below is an
elaboration of each component:

Encoder Base

The Encoder Base module is responsible for extracting base features from the input images.
It takes the input images, both infrared and visible, and processes them to capture the essential
information that forms the base structure of the scene.
The base features typically represent low-frequency components and global structural
information.
12
The extracted base features are then passed to subsequent layers for further processing.

Encoder Detail:
The Encoder Detail module complements the Encoder Base by extracting detail features from
the input images.
It focuses on capturing high-frequency components and local details that are crucial for
enhancing the image's texture and finer features.
Similar to Encoder Base, Encoder Detail processes both infrared and visible images
independently to extract their respective detail features.
The detail features extracted by Encoder Detail are also forwarded to subsequent layers for
further processing.

Decoder
The Decoder module takes the base and detail features extracted by Encoder Base and Encoder
Detail, respectively, and combines them to reconstruct the fused image.
It performs the inverse operation of the encoding process, synthesizing the base and detail
features into a single image that represents the fusion of the input images.
The fused image generated by the Decoder aims to preserve the essential information from both
the infrared and visible spectra while enhancing the overall quality and informativeness.
During the training phase, the network parameters are optimized to minimize a loss function
that encompasses both pixel-wise reconstruction error and structural similarity. This loss
function ensures that the reconstructed fused image closely resembles the ground truth image,
both in terms of pixel values and structural characteristics. By iteratively adjusting the network
parameters based on the calculated loss, the network learns to effectively fuse infrared and
visible images while minimizing information loss and preserving important visual features.
In the context of the provided text, BCL and DCL likely refer to Base Convolutional Layer
(BCL) and Detail Convolutional Layer (DCL), respectively. These layers are components of
the Encoder Base and Encoder Detail modules in the proposed network architecture for image
fusion. Let's delve deeper into each:

Base Convolutional Layer (BCL)

BCL is a convolutional layer within the Encoder Base module. Its primary function is to
perform convolutions on the input image to extract base features. In the network architecture
described, BCL likely applies convolutional operations with specific parameters to capture
these base features from the input images. The output of BCL contributes to the base feature
representation that is further processed in subsequent layers for fusion.

13
Detail Convolutional Layer (DCL)
The DCL in the Encoder Detail module acts as a convolutional layer designed to perform
convolutions on the input image, extracting detail features. Detail features typically represent
high-frequency components and local details crucial for enhancing texture and finer features in
the image. Similar to BCL, DCL applies convolutional operations with specific parameters
tailored to capture detail features from the input images. The output of DCL contributes to the
detail feature representation that complements the base features extracted by BCL, enabling a
comprehensive fusion of information.
In summary, BCL and DCL are essential components of the Encoder Base and Encoder Detail
modules, respectively, in the image fusion network. They play a crucial role in extracting base
and detail features from input images, which are subsequently utilized in the fusion process to
generate a high-quality fused image.

3.2. Single BCL Layer

Fig 3.2. Single BCL Layer

Fig 3.2 illustrates the network architecture during the training phase and depicts a single BCL,
while a DCL has a similar structure but with distinct parameters. The number of input and output
channels for the first convolution units and is (1, H). The second convolutional
units, and are set as (H, 1). The value of H is set to 64. BCL and DCL do not
have any parameters that are shared between them. Applying the blur and Laplacian filters to X,
respectively, initializes the base encoder and detail encoder . The sigmoid function, a
batch regularization layer, and a 3x3 convolution unit make up the decoder. The convolution
unit has one input channel and one output channel. The restored image's pixel values are
normalized to a 0–1 range by means of the sigmoid function.

14
3.3. Network framework in the test phase
Fig 3.3 shows the network framework in the test phase. The trained model is utilized to fuse
input images from the Near-Infrared (NIR) and Visible spectrum. Let's delve into the testing
phase network and its working, elaborating on each step.

Fig 3.3. Network framework in the test phase

Input Preparation:
The testing phase begins by providing pairs of images from the Near-Infrared (NIR) and Visible
spectrum as input to the trained network.

Feature Extraction:
The input NIR and Visible images are separately passed through the Encoder Base and Encoder
Detail networks.
Encoder Base extracts base features representing essential characteristics of the input images,
while Encoder Detail captures detailed features that highlight finer details.

Feature Fusion:
The base and detail features obtained from Encoder Base and Encoder Detail are combined to
reconstruct the fused image.
These features are concatenated or merged in some way to preserve important information from
both input images while enhancing the overall quality of the fused image.

Fusion Process:

The combined features are passed through the Decoder network.

The Decoder network decodes the fused features to generate the final fused image.

15
During this process, the network optimally combines the base and detail features to produce
an image that effectively represents information from both the NIR and Visible spectrum
inputs.

Output Generation
The output of the Decoder network is the final fused image, which represents a combination of
information from the NIR and Visible spectrum inputs.

16
 CHAPTER 4
PROPOSED METHODOLOGY

4.1. Problem Statement

The ultimate goal is to combine visible and infrared images to create composite images that
support both thermal radiation data from infrared images and texture or detail from visible
images.
4.2. Hybrid Approach
The methodology combines traditional methods, such as the two-scale decomposition
technique, with deep learning, particularly through an autoencoder architecture.
Two-scale decomposition involves extracting high-frequency detail information and
lowfrequency base information from source images.

4.3. Autoencoder Architecture

An autoencoder architecture is employed in the deep learning stage of the methodology.

The autoencoder helps in feature extraction and image reconstruction, capturing fine details in
the images and improving overall image quality.

4.4. Training Phase

During the training phase, an autoencoder model is trained on paired infrared and visible image
datasets.

The model learns to capture features from both modalities and optimize the fusion process.

4.5. Testing Phase

In the testing phase, the trained model is applied to new pairs of infrared and visible images.

The input images are passed through the model, and the fused image is generated as the output.

4.6. Feature Fusion and Reconstruction

Base and detail features extracted from input images are merged in the fusion process.
The reconstructed fused image aims to preserve important information from both modalities
while enhancing overall image quality.

4.7. Evaluation
Metrics such as standard deviation, mean gradient, spatial frequency, entropy, sum of
correlations of differences play an important role in evaluating the quality of fused images This
quantitative analysis provides valuable insights into the effectiveness of the fusion process.

17
4.8. Performance Analysis
The performance of the proposed methodology is evaluated using benchmark datasets from
TNO, FLIR, and NIR sources. Comparative analysis with state-of-the-art methods
demonstrates the superior performance and efficiency of the proposed approach.

4.9. Applications
The fused images generated using this methodology can be applied in various fields such as
remote sensing, surveillance, medical imaging, and military applications.

4.10. ALGORITHM
Step 1: Import Necessary Libraries # import torchvision
from torchvision import transforms
import [Link] as vutils
import numpy as np
import torch
import time
from torch import nn
import [Link] as optim from [Link] import Variable
import [Link] as F
from PIL import Image
import [Link] as plt
import os
import [Link] as scio
import kornia from AU Net
import Encoder_Base, Encoder_Detail, Decoder from args
import train_data_path, train_path, device, batch_size, channel, lr, is_cuda, log_interval,
img_size, layer_numb, epochs

Step 2: Define Constants

Paths
train_data_path = "path_to_training_data_directory"
train_path = "path_to_save_trained_models_directory"

Model Parameters device = "cuda" # or "cpu" batch_size = 16 channel = 3 lr = 0.001

is_cuda = True # Set to True if using GPU, False if using CPU log_interval = 100 img_size = 256
layer_numb = 10
epochs = 100

Step 3: Load the Input Image

Define the path to the input image

input_image_path = "path_to_input_image.jpg"

18
Load the input image using PIL
input_image = [Link](input_image_path)

Display the input image input_image.show()

Step 4: Load Pretrained Models

Define paths to pretrained model files

encoder_base_model_path = "path_to_encoder_base_model.pth"
encoder_detail_model_path = "path_to_encoder_detail_model.pth" decoder_model_path =
"path_to_decoder_model.pth"

Instantiate model classes encoder_base = Encoder_Base() encoder_detail = Encoder_Detail()

decoder = Decoder()

Load pretrained weights

encoder_base.load_state_dict([Link](encoder_base_model_path))
encoder_detail.load_state_dict([Link](encoder_detail_model_path))
decoder.load_state_dict([Link](decoder_model_path))

If using GPU, move models to GPU if is_cuda:

encoder_base = encoder_base.cuda() encoder_detail = encoder_detail.cuda() decoder
= [Link]()

Step 5: Set the Models to Evaluation Mode if

Set models to evaluation mode encoder_base.eval() encoder_detail.eval() [Link]()

Step 6: Instance Segmentation

model =
[Link].maskrcnn_resnet50_fpn(pretrained=True) [Link]()
Preprocess input image def preprocess_image(image): transform = [Link]([
[Link](), [Link](mean=[0.485, 0.456, 0.406], std=[0.229,
0.224, 0.225]), ]) return transform(image).unsqueeze(0)
Load and preprocess input image image_path = 'input_image.jpg' image =
[Link](image_path).convert("RGB") input_image = preprocess_image(image)
Perform inference with torch.no_grad():
prediction = model(input_image)
Extract masks for each detected instance masks =
prediction['masks'].squeeze().detach().cpu().numpy()

Step 7: Visualize Instance Segmentation image_path = 'input_image.jpg' image =

[Link](image_path).convert("RGB") input_image = preprocess_image(image)
Perform inference with torch.no_grad():
prediction = model(input_image) # Extract masks for each detected instance
masks =

19
prediction['masks'].squeeze().detach().cpu().numpy()
Get list of bounding boxes boxes = prediction['boxes'].detach().cpu().numpy()
Get list of labels labels = prediction['labels'].detach().cpu().numpy()
Create figure and axes fig, ax = [Link](1) # Plot input image [Link](image) # Plot
instance masks for mask, box, label in zip(masks, boxes, labels):
Threshold mask mask = mask > 0.5 # Create mask polygon mask_poly =
np.zeros_like(mask, dtype=np.uint8) mask_poly[mask] = 1
Get bounding box coordinates
ymin, xmin, ymax, xmax = box # Create mask polygon poly_verts = [Link](mask_poly)
Create a rectangle patch rect = [Link]((xmin, ymin), xmax - xmin, ymax - ymin,
linewidth=1, edgecolor='r', facecolor='none')
Add the patch to the Axes ax.add_patch(rect) # Show plot [Link]()

20
 CHAPTER 5
CODING AND TESTING

5.1. Data Preprocessing

In this stage, the input data, typically consisting of paired infrared and visible images,
undergoes necessary transformations to make it suitable for model training. This involves
loading the images into memory, resizing them to a standard size (e.g., 256x256 pixels), and
converting them into tensor format, which is the standard data structure for deep learning
frameworks like PyTorch or TensorFlow. Additionally, if the pixel values of the images vary
significantly, normalization is applied to scale them to a common range (usually between 0 and
1).
5.2. Model Training
Model training is a crucial step where the Algorithm Unraveling architecture is initialized and
optimized to learn the mapping between input infrared-visible image pairs and their
corresponding fused images. The training data set is divided into training validation sets to
verify model performance and allow for overfitting reduction. A loss function, typically two-
word reconstruction and normalization, instructs the model to minimize the difference between
the predicted fusion images and the ground truth Throughout the training process. optimization
algorithms Adam Optimizer is used.
5.3. Model Testing
After training the model, it is tested on a separate dataset containing unseen image pairs. The
trained model is loaded, and the test images are fed through it to obtain the fused output images.
These fused images are then saved for further analysis and visualization. Testing the model on
unseen data helps assess its generalization ability and ensures that it can produce accurate
fusion results for new inputs.
5.4. Evaluation
Evaluation metrics is used quantitatively assessing the performance of the Algorithm
Unraveling model. Metrics like SD, AG, SF and EN are computed for both the training and test
datasets. These metrics provide insights into the quality, sharpness, texture preservation, and
overall effectiveness of the fused images. Comparing the Algorithm Unraveling model's
performance with other state-of-the-art fusion methods using these metrics helps validate its
efficacy and superiority.

21
5.5. Code Implementation
Implementing the Algorithm Unraveling model involves writing organized and modular code
that encapsulates various functionalities such as data loading, preprocessing, model definition,
training, testing, and evaluation. It's essential to use appropriate deep learning frameworks like
PyTorch or TensorFlow for building and training the model efficiently. It is important to
communicate the rules well with feedback, clarifying the purpose of each activity or lesson.
This process increases readability and makes it easier for new users to convert. Moreover, it is
important to rigorously test the code on sample data to ensure its validity and robustness.
5.6. Optimization and Deployment
After the model is trained and tested successfully, optimization for inference is performed to
ensure efficient deployment on target hardware platforms, such as CPUs, GPUs, or specialized
accelerators. This may involve techniques like model quantization, pruning, or model
compression to reduce its size and improve inference speed. Once optimized, the model and
associated code are packaged into a standalone application or library for deployment. Before
deployment, the model's performance is validated on real-world data to ensure consistency with
the evaluation results and suitability for practical applications.
By elaborating on each step, developers can gain a comprehensive understanding of the process
involved in implementing, testing, and deploying the Algorithm Unraveling Image Fusion
model for visible and infrared image fusion.

22
 CHAPTER 6
RESULTS AND DISCUSSION

The performance of the proposed Algorithm Unraveling model was evaluated extensively using
both qualitative and quantitative metrics on benchmark datasets, including the TNO and FLIR
datasets. The results demonstrate the efficacy and superiority of the Algorithm Unraveling
model over SOTA fusion methods in terms of image quality, target region preservation, and
computational efficiency.
6.1. Quantitative Evaluation
Quantitative evaluation metrics like SD, AG, SF and EN were computed for the fused images
produced by the Algorithm Unraveling model and compared against other fusion methods. The
results indicate that the Algorithm Unraveling model consistently outperformed existing
methods across all metrics on both TNO and FLIR datasets. Specifically, the Algorithm
Unraveling model achieved higher values for SD, AG, SF, and EN, indicating better image
quality, sharper edges, preserved texture details, and enhanced contrast compared to alternative
methods.
6.2. Qualitative Evaluation
In visual inspection, fusion images produced by algorithm unraveling models outperformed
those produced by other fusion techniques and showed remarkable improvements in image
clarity, contrast, and information retention Capacity and what's more, important target areas and
systems generated by algorithm unraveling models He demonstrated holding changes, making
them better suited for many real-world applications including exploration, remotely research
and medical imaging.
6.3. Computational Efficiency
In addition to superior fusion quality, the Algorithm Unraveling model exhibited efficient
computational performance, demonstrating rapid convergence during training and low
inference time during testing. The model's architecture, which incorporates algorithm
unraveling and deep learning techniques, allowed for parallel processing and optimized
memory utilization, enabling seamless integration into resource-constrained environments such
as embedded systems and edge devices.
6.4. Discussion
The results obtained from the evaluation of the Algorithm Unraveling model validate its
effectiveness and robustness in fusing infrared and visible images for various applications. By
leveraging the synergies between traditional optimization models and deep neural networks,
the Algorithm Unraveling model offers a versatile and scalable solution for image fusion tasks.
The hybrid approach, which combines the interpretability of algorithm unraveling with the
feature extraction capabilities of deep learning, addresses the limitations of existing fusion
methods and opens new avenues for research and development in the field of image processing.

23
Furthermore, the comprehensive evaluation conducted on benchmark datasets demonstrates the
generalizability and applicability of the Algorithm Unraveling model across different domains
and scenarios. Future work may focus on further refining the model architecture, exploring
alternative fusion strategies, and extending the application of the Algorithm Unraveling model
to other multi-modal fusion tasks such as hyperspectral imaging and medical diagnostics.
Overall, the Algorithm Unraveling model represents a significant advancement in image fusion
technology with promising implications for various real-world applications. By presenting and
discussing the results in this manner, researchers and practitioners can gain insights into the
performance and potential of the Algorithm Unraveling Image Fusion model for addressing
image fusion challenges in diverse domains.
6.5. Evaluation Metrics
The following metrics will be used to evaluate the quality of fused images: standard deviation
(SD), average gradient (AG), spatial frequency (SF), entropy (EN), and these metrics will allow
us to quantitatively characterize the efficacy of fusion. Higher values correspond to higher-
quality fused outcomes.
Datasets:
The FLIR and TNO dataset [16] was selected as the test subjects to thoroughly assess the
efficacy of the proposed method. The TNO dataset consists of numerous pre-aligned pairs of
near-infrared (NIR) and visible images. Similarly, the FLIR dataset consists of thermal infrared
and visible images. In order to conduct an experiment, a collection of thirty pairs of images
was selected from the FLIR dataset. All of the images were then resized to a consistent size of
256 × 256 pixels. The resizing was performed to simplify the assessment of multi-scale
transformations and it is using on a laptop equipped with 16 GB of Random-Access Memory
and an Intel Core i7 processor.
Quality Metrics:
The impact of the proposed algorithm's fusion has been assessed using four objective evaluation
measures. They are
Entropy (EN): EN is the evaluation of image quality to quantify the level of information
contained within the fused image.
Average Gradient (AG): The AG quality metric evaluates the perceptual clarity of an image by
analyzing its texture and contrast characteristics. The evaluation assesses how much the fused
image effectively maintains the details and edges in the source images. A higher AG value is
indicative of superior perceptual image quality.
Spectral Fidelity (SF): Spatial frequency acts as a metric of how much spectral information of
the input images is faithfully preserved in the fusion image It plays an important role in
ensuring that the fused image retains faithfulness to the original spectral characteristics of the
input photographs.
Spatial Distortion (SD): SD is a measure of the extent that spatial distortion or misalignment
occurs during the fusion process. It evaluates the degree to those spatial details are maintained
in the fused image.

24
Evaluation against rival algorithms using FLIR and TNO Dataset:
The validity of the proposed method is confirmed through the utilization of the FLIR dataset.
Fig 6.1 displays the fusion results of the source image, which portrays two individuals standing
next to their bicycle near the side of the apartment road. The visible image clearly displays the
specific features of the house and cycle. However, the individuals were not discernible in the
visible image, whereas they are detectable in the NIR image because of its high sensitivity to
thermal radiation, which captures data related to a person. The fusion performance of the
proposed method has been evaluated using both qualitative and quantitative measures.

Fig. 6.1. Fusion output

From a subjective standpoint, it can be observed that the fused image acquired through
FusionGAN exhibits subpar visual quality and is deficient in various features present in the
visible image. The fused images obtained through the utilization of DeepFuse exhibit a general
blurriness, characterized by unclear boundaries and a significant amount of noise. This
degradation in edge data hinders the visibility of the scene within the apartment, thereby posing
challenges in its observation. The DenseFuse algorithm exhibits a lack of consistency in
maintaining uniform brightness levels, as observed in the case of the road surface. Additionally,
it can be observed that the output generated by TVADMM exhibits a certain level of haziness.
At the same time, the results produced by TSIFVS appear to be dim. The proposed methodology
exhibits superior contrast in detecting salient targets compared to all other methodologies. It is
evident that the fused image produced by the proposed methodology effectively showcases rich
textures of two person with their cycles within the designated orange box.

25
 Table 6.1. Average results obtained by applying multiple techniques to the FLIR dataset.

Methods EN AG SF SD

DeepFuse 7.21 4.80 15.47 37.35

FusionGan 7.02 3.20 11.51 34.38

DenseFuse 7.21 4.82 15.50 37.32

TSIFVS 7.15 5.57 18.79 35.89

ImageFuse 6.99 4.15 14.52 32.58

TV-admm 6.80 3.52 14.04 28.07

Proposed 7.45 5.91 14.46 38.18

Table 6.1 presents the objective evaluation indices. The proposed method demonstrates
superior performance across various evaluation metrics, like EN, AG, SF and SD. This finding
shows that the proposed method has a higher ability to preserve the inherent feature data in the
input images compared to other approaches. As a result, the image obtained using the proposed
method displays improved levels of detail and superior visual effects. The proposed method
demonstrates exceptional performance in both subjective and objective evaluation.
Table 6.2. Average results obtained by applying multiple techniques to the TNO datasets

Methods EN AG SF SD

DeepFuse 6.86 3.60 11.13 32.25

FusionGan 6.58 2.42 8.76 29.04

DenseFuse 6.84 3.60 11.09 31.82

TSIFVS 6.67 3.98 12.60 28.04

ImageFuse 6.38 2.72 9.80 22.94

TV-admm 6.40 2.52 9.03 23.01

Proposed 6.90 3.33 9.86 33.5

Using a TNO dataset, Table 2 displays the average values of six objective assessment criteria,
with the red values emphasizing the highest values. The proposed algorithm outperformed all

26
alternatives in terms of fusion performance, with the exception of AG and SF, and received the
maximum achievable score

27
 CHAPTER 7
Conclusion and Future Enhancements

Proposed work introduces a novel approach for fusing visible and infrared images by
combining traditional two-scale decomposition methods with the efficiency and transparency
of deep learning, particularly through an autoencoder architecture. Our Algorithm Unraveling
model integrates algorithm unraveling techniques to establish a logical connection between
deep neural networks and traditional optimization algorithms, resulting in a hybrid fusion
model capable of preserving thermal radiation information from infrared images while
retaining texture and quality details from visible images.
We presented a comprehensive workflow of the Algorithm Unraveling model, detailing the
process of decomposition, feature merging, and image fusion. By leveraging autoencoder
architecture in the deep learning stage, our model learns to capture fine details from input
images, enhancing the quality of the fusion process. Furthermore, we introduced an efficient
optimization algorithm based on gradient descent to update base and detail feature maps during
training, ensuring convergence and robust performance.
Evaluation of the Algorithm Unraveling Evaluation of the model on benchmark datasets such
as TNO and FLIR revealed superiority over modern fusion methods in terms of image quality,
target area preservation and computational efficiency All these metrics though several metrics
including standard deviation, mean slope, spatial frequency, entropy and sum used of
correlations of differences Their unraveling model consistently improved compared to existing
methods performance, reinforcing its effectiveness and capability various functions.

Future Enhancements
Moving forward, several avenues for improvement and future research directions can be
explored to further enhance the effectiveness and applicability of the Algorithm Unraveling
model:
Extension to other modalities
Investigate the applicability of the Algorithm Unraveling model to other multi-modal fusion
tasks beyond infrared and visible images, such as hyperspectral imaging or medical diagnostics.
Optimization and efficiency
Explore strategies to optimize the computational efficiency of the Algorithm Unraveling
model, particularly for real-time applications and resource-constrained environments.
Fine-tuning and adaptation
Develop techniques for fine-tuning the Algorithm Unraveling model to specific fusion
scenarios or domains, allowing for adaptation to diverse imaging conditions and requirements.

28
Integration of domain knowledge
Incorporate domain-specific knowledge or constraints into the fusion process to improve the
interpretability and relevance of fused images for practical applications.
Exploration of loss functions
Investigate alternative loss functions or regularization techniques to further enhance the quality
and realism of fused images generated by the Algorithm Unraveling model.
Scale-up experimentation
Conduct experiments with larger and more diverse datasets to validate the generalizability and
robustness of the Algorithm Unraveling model across various imaging conditions and
applications.
User studies and feedback
Engage end-users and domain experts to gather feedback on the usability, interpretability, and
utility of the fused images generated by the Algorithm Unraveling model, informing further
refinements and enhancements.
Exploration of transfer learning
Investigate the potential benefits of transfer learning techniques for fine-tuning pre-trained
Algorithm Unraveling models on specific fusion tasks or datasets, accelerating convergence
and improving performance.
Integration of uncertainty estimation
Develop methods for estimating uncertainty or confidence intervals associated with fused
images generated by the Algorithm Unraveling model, providing users with valuable insights
into the reliability and trustworthiness of the fusion results.
Development of real-world applications
Collaborate with industry partners and stakeholders to explore real-world applications of the
Algorithm Unraveling model in areas such as surveillance, remote sensing, medical imaging,
and industrial inspection, facilitating technology transfer and commercialization efforts.
By addressing these areas of improvement and continuing to innovate in the field of image
fusion, we aim to advance the state-of-the-art and unlock new possibilities for applications in
surveillance, remote sensing, medical imaging, and beyond.

29