Medical Image

Processing in Machine
and Deep Learning

Presented by
[Link]
Research Scholar of Computer Science (AI)

13-03-2024 [Link], Research Scholar in CS 1

 What is Medical Image?

• Visual representations of the internal structures and

functions of the human body or other living organisms,
captured through various imaging modalities.
Source: Google image
• To diagnose, monitor, or treat medical conditions.
• Medical imaging, also known as radiology
• Non-invasive tests that allow doctors to diagnose injuries
and diseases without being intrusive.
• Procedures help determine whether surgery would be an
effective treatment option; locate tumors for treatment
and removal; find blood clots or other blockages
• Greatly reducing the amount of guesswork done by
doctors, allowing them to more effectively deal with
patients' injuries and diseases.

13-03-2024 [Link], Research Scholar in CS 2

Source: Google image
 A historical timeline of the key discoveries in medical
imaging

Source: Najjar R. Redefining Radiology: A Review of Artificial Intelligence Integration in Medical Imaging. Diagnostics. 2023; 13(17):2760. [Link]

3/13/2024 [Link], Research Scholar in CS 3

 Role of AI in Medical Image

Najjar, R. Redefining Radiology: A Review of Artificial Intelligence Integration in Medical Imaging. Diagnostics 2023, 13, 2760.
13-03-2024 [Link], Research Scholar in CS 4
 Type of Medical Images
X-ray imaging
• Create images of bones and some soft tissues.
• Used for detecting fractures, joint dislocations, and certain lung
conditions.
Computed Tomography (CT) Scan
Source: Google image

• Combines X-rays and computer processing to generate detailed

cross-sectional images (slices) of the body.
• visualizing internal structures like organs, blood vessels, and
bones.
Magnetic Resonance Imaging (MRI)
• Utilizes strong magnetic fields and radio waves to create
detailed images of soft tissues, organs, and joints.
• Often preferred for imaging the brain, spinal cord, joints, and
muscles.

13-03-2024 [Link], Research Scholar in CS 5

Type of Medical Images (Cont.)
Ultrasound Imaging
• High-frequency sound waves to create images of internal
structures.
• Examining the abdomen, pelvis, and fetal development during
pregnancy.
Nuclear Medicine Imaging
• Involves the administration of a radioactive substance
(radiopharmaceutical) to visualize the distribution and function of
organs and tissues.
• Techniques include Single Photon Emission Computed
Tomography (SPECT) and Positron Emission Tomography (PET).
Mammography
• A specialized X-ray technique used for breast imaging to detect
and diagnose breast diseases, including breast cancer.

13-03-2024 [Link], Research Scholar in CS 6

Source: Google image
 Fluoroscopy Source: Google image

Type of Medical Fluoroscopy

• Real-time X-ray imaging is used to observe the movement of
Images (Cont.) internal structures, such as the gastrointestinal tract or blood vessels.
• Used for procedures such as angiography and gastrointestinal
studies.
Endoscopy
• Involves the use of a flexible tube with a light and camera to
visualize the interior of organs or body cavities.

13-03-2024 [Link], Research Scholar in CS 7

 Type of Medical Images
(Cont.)

Histopathology Images
• Microscopic images of tissues obtained
through staining techniques, such as
Hematoxylin and Eosin (H&E), are used for
pathological examination to diagnose Source: Google image

diseases.
Dermatological Images:
• Images of skin conditions and lesions, often
captured using dermoscopy or clinical
photography.

Source: Google image

13-03-2024 [Link], Research Scholar in CS 8

Type of Medical Images (Cont.)
Hematoxylin & Eosin Histology Images
• Staining is a widely used technique in histology to colorize
different cellular structures, making them visible under a
microscope.
• Hematoxylin stains nuclei blue-purple, while Eosin stains Source: Google image
cytoplasm and extracellular structures pink.
• Staining method provides a broad overview of tissue architecture
and helps in the identification of various cellular components.
• Histology images stained with H&E are commonly used in
pathology and medical research to study tissues, diagnose
diseases, and understand the microscopic details of organs and
structures.
• These images are often captured using a microscope and can be
found in pathology reports, medical textbooks, and scientific
publications.

13-03-2024 [Link], Research Scholar in CS 9

Want to be a master in this area?
Prerequisites - Type of Medical images

Basic Image Processing - Filtering, segmentation, feature extraction etc.,

Introduction to ML and DL - Supervised, unsupervised, Self-supervised

Specific ML / DL Models for Medical Imaging - CNN-image, U-Net-segment, RNN-sequence of image

Data Preparation and Augmentation - Cleaning and augmentation (New images – GAN)

Framework and Libraries - TensorFlow, PyTorch, Scikit-learn

Case Studies and Projects - Publicly available dataset, small projects

Stay Updated - Read recent ML/DL papers, Community Involvement

Course and Tutorials - Coursera, edX, or Udacity

Networking - Attend conferences, webinars, and meetups to network

13-03-2024 [Link], Research Scholar in CS 10

 Image Enhancement
Medical Image Processing
Image Enhancement
• Contrast Enhancement: Adjusting the pixel
intensity to improve the visibility of structures
within an image.
• Histogram Equalization: Enhancing the contrast
by redistributing pixel intensities.
Image Segmentation
• Thresholding: Dividing an image into regions
based on intensity thresholds.
Source: Google image
• Region Growing: Grouping pixels based on
similarity criteria.
• Watershed Segmentation: Separating overlapping
objects using a topographical analogy.
• DL-based Techniques: U-Net, Mask-RCNN etc.,

13-03-2024 [Link], Research Scholar in CS 11

 Medical Image Processing
Image Registration – fitting one images onto another
• Spatial Registration: Aligning images from different
modalities or time points in space.
• Intensity-Based Registration: Aligning images based
on pixel intensities.
Image Registration

Source: Google image

Filtering Techniques
• Spatial Filtering: Applying filters (e.g., Gaussian,
median) to enhance or suppress certain features.
• Frequency Domain Filtering: Using Fourier
transforms for image filtering.
Image Reconstruction
• Computed Tomography (CT) Reconstruction:
Creating cross-sectional images from X-ray
projections.
• Magnetic Resonance Imaging (MRI)
Reconstruction: Assembling images from raw MRI
data.
13-03-2024 [Link], Research Scholar in CS 12
 Medical Image Processing

Noise Reduction
• Median Filtering: Replacing each pixel with
the median value in its neighborhood.
• Wavelet Denoising: Decomposing images
into frequency components for noise
reduction.
Feature Extraction
• Texture Analysis: Extracting information
related to the spatial arrangement of pixels.
• Shape Analysis: Quantifying geometric
properties of structures within an image.
Source: Google image

13-03-2024 [Link], Research Scholar in CS 13

 Medical Image Processing
Image Fusion
• Multimodal Fusion: Combining information from
different imaging modalities.
• Temporal Fusion: Integrating information from images
captured at different time points.
Machine Learning in Medical Imaging
• Segmentation with Convolutional Neural Networks
(CNNs): Training deep learning models for accurate
segmentation.
• Classification with Support Vector Machines (SVM)
Source: Google image or Random Forests: Automated diagnosis based on
image features.
3D Imaging Techniques
• Volumetric Rendering: Creating 3D representations of
medical images.
• 3D Reconstruction from 2D Slices: Reconstructing a
3D model from a series of 2D images.

13-03-2024 [Link], Research Scholar in CS 14

Why computer vision and machine learning?

Automation
Accuracy and Quantitative Early Detection Personalized
& Time Consistency Analysis and Diagnosis Medicine
Efficiency

Research and
Integration with Multi-model data Challenges in Scale
Innovation

Use of Computer Vision and Machine Learning in Bio-medical

13-03-2024 [Link], Research Scholar in CS 15

Cheng, P.M.; Montagnon, E.; Yamashita, R.; Pan, I.; Cadrin-Chênevert, A.; Perdigón Romero, F.; Chartrand, G.; Kadoury, S.; Tang, A. Deep Learning: An Update for Radiologists. Radiographics 2021, 41,
1427–1445.
 Machine Learning Process

Source: Google image

13-03-2024 [Link], Research Scholar in CS 17
 Deep Learning Process

13-03-2024 [Link], Research Scholar in CS Source: Google image 18

 Basic Learning Foundations: Algorithms and Techniques
Learning Style Common Algorithms Methods Applied Ares of Medical Image

Supervised Learning Linear or logistic regression, Decision trees and random Cancer diagnosis, Organ segmentation, Radiotherapy
forests, Support vector machines , Convolutional neural dose denoising, Radiotherapy dose prediction,
networks, Recurrent neural networks Conversion between image modalities

Unsupervised Learning Auto encoders, Dimensionality reduction (e.g., Principal Domain adaptation tasks, Classification of patient
component analysis), Clustering groups, Image reconstruction

Reinforcement Learning Q-learning , Markov Decision Processes Tumor segmentation, Image reconstruction, Treatment
planning
Semi-supervised Learning Generative Adversarial Networks Tumor classification, Organ segmentation, Synthetic
image generation
Self-Supervised Learning Pretext task: distortion (e.g. rotation), color- or intensity- Image classification or segmentation
based, patch extraction

Transfer Learning Inductive, Transductive, Unsupervised Radiotherapy toxicity prediction, Adaptation to

different clinical practices, Improving model
generalization[
Ensemble Learning Bagging - Bootstrap AGGregatING - (e.g. random forests), Radiotherapy dose prediction, Estimation of
Boosting (e.g. AdaBoost, gradient boosting) uncertainty , Stratification of patients

Barragán-Montero, Ana, et al. "Artificial intelligence and machine learning for medical imaging: A technology review." Physica Medica 83 (2021): 242-256.

13-03-2024 [Link], Research Scholar in CS 19

 Types of Learning in ML and Commonly Applied in Medical Image
Supervised Learning Uses labeled datasets to train algorithms to predict outcomes and recognize
patterns. (Classification, Regression)

Learns from data without human supervision, Unsupervised models are given
Primary Type Unsupervised Learning unlabeled data and allowed to discover patterns and insights without any
explicit guidance or instruction.

Machine learning training methods based on rewarding desired behaviors and

Reinforcement Learning
punishing undesired ones (Hit and trial)

Semi-Supervised Combines supervised and unsupervised learning by using both labeled and
Learning unlabeled data to train models for classification and regression tasks.
Hybrid Type
Self-supervised Model is pre-trained using unlabelled data and the data labels are generated
Learning automatically, which are further used in subsequent iterations as ground truths.

Reuse of a pre-trained model on a new task. This can be useful when the
Transfer
second task is similar to the first task, or when there is limited data available for
Learning Style the second task.

Ensample Enhances accuracy and resilience in forecasting by merging predictions from

multiple models.
13-03-2024 [Link], Research Scholar in CS 20
 Tasks we need to know in DL process in Medical Image analysis
• Classification is sometimes also known as Computer-Aided Diagnosis (CADx). From the clinician's
Classification
perspective, classification ascertains if a disease state is present or not, i.e., is blood present on this MRI brain
scan signifying a hemorrhagic stroke?
• Localization implies the identification of normal anatomy, For example, where is the kidney in this ultrasound
Localization image?
• Detection, sometimes known as Computer-Aided Detection (CADe), This is in contrast to detection, which
Detection implies an abnormal, pathological state, for example, where are all the lung tumors in this CT scan of the
lung?

Segmentation • Segmenting the outline of a lung tumor helps the clinician determine its distance from major anatomical
structures, and helps to answer a question such as, should this patient be operated on, and if so, what should be
the extent of resection?

Registration • Image registration is employed in neurosurgery or spinal surgery, to localize a tumor or spinal bony landmark,
in order to facilitate surgical tumor removal or spinal screw implant placement.
• A reference image is aligned to a second image, called a sense image and various similarity measures and
reference points are calculated to align the images, which can be 2 or 3-dimensional.
• The reference image may be a pre-operative MRI brain scan and the sense image may be an intraoperative MRI
brain scan done after a first-pass resection, to determine if there is remnant tumor and if further resection is
required.

13-03-2024 [Link], Research Scholar in CS 21

 Summary of Existing Research works in Medical Image analysis
Task Organ Modality Performance Metrics Network Dataset Organ Modality / Task

Detection Brain, MRI, ABUS, AUC, True Positive VAE, 3D-CNN, CNN, CT (Classification /
IDL, LIDC-IDRI, Lung
Breast, OCT, X-ray, Rate, VOI, False GAN, VGG-16, sCAE, Detection
Eyes, CMR, WGD, positive, Sensitivity, AEE ADNI, BRATS,
Chest US Area under ROC, F1- MRI, PET, Multiple
ACDC 2017,
Abdomen score, Shadow (Classification,
OASIS-3, ISLES Brain
confidence Segmentation,
2018, Open Neuro,
Diagnosis)
ABIDE
Segmentation Brain, MRI, DCE- Dice score, Sensitivity, AE, AD, FCNN +CRF,
Breast, MRI, OCT, CT, Specificity, PPV, FCN, cGAN, GANs, Mammography, WSI
DDSM,
Eyes, Mammograms Runtime, TS, Intensity CNN (Segmentation,
MIAS,CAMELYO Breast
Chest normalization, Classification/
N17, INbreast
Abdomen Accuracy, ASSD, SP Detection)
MESSIDOR-2, SLO, OCT
Registration Brain, MRI, OCT, Dice Score, CNN, DFCN, GANs, 3D- DRIVE, STARE, Eyes (Classification,
Breast, MR-CT, CT- Computation time, CNN, U-Net CHASEDB1 Segmentation)
Eyes, MRI Reliability, Runtime,
Musculos X-ray (Detection)
Chest Deformation field, DIR, MURA
keletal
Abdomen HD, ASD run time,
NRMSE ISIC 2018 Skin JPEG (Detection)

Classification Brain, fMRI, MRI, ROIs, Classify ASD, DNN, 3D CNN, HVSMR 2018 Heart CMR (Segmentation)
Breast, Histopathology, Accuracy, VOIs, AlexNET, VGG-19, AE,
Eyes, Mammography, Sensitivity, Specificity, AD, GANs, MIL ChestX-ray 14 Chest X-ray (Multiple)
Chest OCT, CT, MRI DR severity accuracy OAI Knee X-ray, MRI (Multiple)
Abdomen
Altaf, F., Islam, S. M., Akhtar, N., & Janjua, N. K. (2019). Going deep in medical image analysis: concepts, methods, challenges, and future directions. IEEE
TCIA Multiple Multiple
Access, 7, 99540-99572.
 Source: Google image
Source: Google image
• XAI aims to explain the information behind the black-box
model of deep learning that reveals how the decisions are
XAI: Explainable AI made.
• Providing insights into how the predictions are made to
Techniques achieve trustworthiness, causality, transferability, confidence,
fairness, accessibility, and interactivity.
13-03-2024 [Link], Research Scholar in CS 23
Criteria used to classify XAI methods
Explanation Type Techniques
Feature • Backpropagation (BP), Guided-BP,
(Highlights the feature • Deconv Network,
XAI contribution for making • Layer-wise relevance propagation
Methods decisions, original image (LRP),
with an overlay of
saliency map)
• Class activation map (CAM),
Explanation Interpretation Model Explanation • Grad-CAM, Local Interpretable model-
forms Types Specificity Scopes agnostic explanations (LIME),
• GraphLIME,
Model- • Shapley additive explanations (SHAP),
Feature Map Intrinsic Global • Attention
specific
Example-based • ProtoPNet,
Model- (Provide explanations by • Triplet Network
Textual Post-hoc Local presenting similar • Explainable deep neural network
agnostic
instances or examples (xDNN)
from the dataset that
Example- influenced the model's
based decision.)
Textual • Testing with concepts activation vectors
(provide written (TCAV),
descriptions of the factors • Image Captioning
influencing a model's
decision)
13-03-2024 [Link], Research Scholar in CS 24
Criteria used to classify XAI methods (cont.)
Interpretation Type
• Intrinsic Explanation - Inherent to the model itself and built into its architecture or training process. (linear
model, decision tree, etc.,)
• Post Hoc Explanation- Generated after the model has made predictions and is not inherently part of the
model's architecture or training process (Deep learning techniques - CAM, Grad-CAM, LIME, SHAP)
Model Specificity
• Model-Specific Explanation - designed with the inner workings and assumptions of a particular model in
mind. (Visualizing Feature Map or attention mechanism)
• Model-Agnostic Explanation - aim to provide insights into the model's behavior without relying on
specific model characteristics. (LIME and SHAP)
Explanation Scopes
• Local Explanation - Providing insights into the decision-making process of a machine learning model at an
individual prediction level (LIME and SHAP)
• Global Explanation - To provide insights into the overall behavior and functioning of a machine learning
model across the entire dataset (SHAP)

13-03-2024 [Link], Research Scholar in CS 25

 XAI-Methods Used in Medical Images
XAI Organ Modality CAM (Class Activation Map)
• Highlights the most important regions of an image, and makes
Bone X-ray
a specific prediction.
CAM Lung Ultrasound, X-ray, CT Grad-CAM (Gradient-weighted Class Activation Mapping)
Breast X-ray • Provides more detailed and precise visualizations of which
parts of an image are influencing the network's decision and
Lung CT, X-ray output concerning its convolutional feature maps.
Grad-CAM Chest Ultrasound Saliency
• Highlights which parts of the input data are most crucial for a
Colon Colonoscopy
model's decision-making process.
Breast X-ray • Saliency techniques may highlight important features across
Attention Bone CT the entire input space, Grad-CAM specifically localizes the
importance of features within the input image, offering more
Colon Colonoscopy
precise and detailed visualizations.
Saliency Lung, Skin CT, X-ray LIME and SHAP
Lung HER, Radiomics, CT • LIME focuses on providing local explanations by
SHAP approximating the model's behavior around individual
Chest HER instances, while SHAP offers both local and global
LIME, SHAP Chest X-ray explanations by assigning feature contributions to model
predictions using Shapley values.
SHAP, LIME, Scoped Lung HER
Attention
Rules
• Pay more attention to important details when processing
Image Caption Chest CT information
13-03-2024 [Link], Research Scholar in CS 26
 Challenges of AI Integration into Clinical Practice

Workflow Data Quality Disease Clinical Utility Ethical XAI:

Integration and Diversity • Check speed Consideration Explainability
• PACS and Generalization • Data-driven • Accuracy • Informed consent • ‘Block box’
dictation software • Bias mitigation in • Domain-experts • Efficiency • Data privacy problem
Require data collection • Prevention of • Explain how it
• Automation, Require Misdiagnoses was predicted
• Standardization • Privacy • Confidentially
• Quality Control • Annotations • Guidelines

13-03-2024 27
 Existing Challenges in Medical Image analysis using ML and DL
Annotation
Costs and
Data Expertise Clinical
Imbalance and Validation and
Rarity Integration

High quality Interpretability

labelled data is and
limited Explainability

Limited Ethical and

publicly Legal
available Challenges
datasets Consideration

13-03-2024 [Link], Research Scholar in CS 28

 Future Direction in Medical Image Analysis
Wrapping deep
features for
medical image
analysis
Dealing with Training
small data size partially
– use transfer Frozen deep
learning networks

Future Direction

Miscellaneous Ensample
data deep models
augmentation and Multi-task
techniques learning
Using GAN
for synthetic
data
generation

Source: Google image

13-03-2024 [Link], Research Scholar in CS 29
13-03-2024

Is it possible to
achieve [this] with
one expert in a
particular field?
[Link], Research Scholar in CS

• Barragán-Montero, A., Javaid, U., Valdés, G., Nguyen, D., Desbordes, P., Macq,
B., ... & Lee, J. A. (2021). Artificial intelligence and machine learning for
medical imaging: A technology review. Physica Medica, 83, 242-256.
• Suganyadevi, S., Seethalakshmi, V., & Balasamy, K. (2022). A review on deep
learning in medical image analysis. International Journal of Multimedia

Notable Articles Information Retrieval, 11(1), 19-38.

• Puttagunta, M., & Ravi, S. (2021). Medical image analysis based on deep
for Further learning approach. Multimedia tools and applications, 80(16), 24365-24398.
• Anwar, S. M., Majid, M., Qayyum, A., Awais, M., Alnowami, M., & Khan, M.
Insights and K. (2018). Medical image analysis using convolutional neural networks: a
review. Journal of medical systems, 42, 1-13.
Learning that I • Singha, A., Thakur, R. S., & Patel, T. (2021). Deep learning applications in

have used in my
medical image analysis. Biomedical Data Mining for Information Retrieval:
Methodologies, Techniques and Applications, 293-350.

studies • Altaf, F., Islam, S. M., Akhtar, N., & Janjua, N. K. (2019). Going deep in
medical image analysis: concepts, methods, challenges, and future
directions. IEEE Access, 7, 99540-99572.
• Najjar, R. (2023). Redefining radiology: a review of artificial intelligence
integration in medical imaging. Diagnostics, 13(17), 2760.
• Chaddad, A., Peng, J., Xu, J., & Bouridane, A. (2023). Survey of explainable AI
techniques in healthcare. Sensors, 23(2), 634.

13-03-2024 31
[Link], Research Scholar in CS

Thank You..!

Keep Learning and

Upgrading Yourself
until you reach
where you want to
be..!

3/13/2024 32