Deep Learning 2024-25

Table of Contents
1. Image Recognition Based on Lightweight Convolutional Neural Network: Recent
Advances............................................................................................................................................ 2-9
2. Attention Is All You Need .................................................................................................... 10-17
3. Rethinking of BERT Sentence Embedding for Text Classification ................................... 18-26
4. A Unified Understanding of Deep NLP Models for Text Classification .......................... 27-34
5. Multi-step Transfer Learning in Natural Language Processing for the Health
Domain ............................................................................................................................ 35-41

Dept of CSE (Data Science), MIT Mysore 1

Deep Learning 2024-25

1. Image Recognition Based on Lightweight Convolutional Neural

Network: Recent Advances
Authors: Ying Liu, Jiahao Xue, Daxiang Li, Weidong Zhang, Tuan Kiang Chiew, Zhijie Xu
Published: Image and Vision Computing, 2024 (Elsevier, Open Access)

This paper reviews the evolution of lightweight CNNs and how they enable high‐performance
image recognition on devices with limited computational resources.

Abstract

This review paper presents a detailed examination of how lightweight Convolutional Neural
Networks (CNNs) have transformed the field of image recognition, especially in situations
where computational resources are limited. Traditional CNNs such as VGG, ResNet, and
AlexNet provide strong feature extraction capabilities but require large numbers of parameters
and high floating-point operations (FLOPs), making them unsuitable for mobile or embedded
systems. The paper begins by introducing classical lightweight models—including
SqueezeNet, MobileNet, and ShuffleNet—and explains the architectural innovations that allow
these models to drastically reduce parameter count while maintaining accuracy.

Next, the paper categorizes recent research advancements into three main strategies: model
compression (pruning, quantization, knowledge distillation, and low-rank factorization),
optimization of CNN architectures, and hybrid lightweight CNN–Transformer networks.
Experimental comparisons on datasets such as ImageNet, CIFAR-10, and CIFAR-100
demonstrate that these methods significantly reduce computational complexity while achieving
competitive or even superior accuracy to traditional models. The authors also highlight existing
technical challenges, such as data dependency, model interpretability, and computational
overhead during training. Finally, the paper discusses future research opportunities,
emphasizing the potential of neural architecture search (NAS) and hybrid CNN–Transformer
models.

Introduction

Image recognition is fundamental to modern computer vision applications, including

healthcare diagnostics, agricultural monitoring, autonomous driving, industrial inspection, and
surveillance systems. Historically, early image recognition relied on manual feature extraction
techniques such as texture descriptors, edge detectors, and handcrafted filters. However, such

Dept of CSE (Data Science), MIT Mysore 2

Deep Learning 2024-25

techniques required significant domain expertise and lacked the ability to capture high-level
semantic information.

The introduction of deep learning—particularly Convolutional Neural Networks—

revolutionized image recognition by enabling automatic feature extraction. CNNs learn
hierarchical representations that capture both local and global patterns within images. Although
highly effective, deeper and wider CNNs have grown increasingly computationally expensive,
sometimes containing tens of millions of parameters and requiring billions of operations per
image.

This computational burden limits their deployment on resource-constrained edge devices such
as smartphones, IoT nodes, drones, and embedded boards. As a result, lightweight CNNs have
emerged as a critical research direction. These architectures aim to strike a balance between
accuracy, computational cost, memory usage, and energy consumption. The paper provides a
structured survey of the latest lightweight models and optimization techniques, offering
valuable insights into their strengths and limitations.

Problem Statement

Traditional CNN architectures achieve excellent accuracy but suffer from several limitations:

Key Issues Addressed by the Paper

• High Computational Complexity: Large CNNs require GPUs or high-end hardware,

making them difficult to deploy on embedded or mobile devices.

• Large Memory Footprint: Deep networks often contain millions of parameters that
exceed memory constraints of edge devices.

• Inefficient Power Consumption: Heavy computation leads to increased energy usage,

which is unsuitable for real-time, battery-powered systems.

• Latency Constraints: Applications such as real-time surveillance, robotics, and

autonomous vehicles require fast inference speeds.

• Risk of Overfitting: Larger models also tend to overfit when trained on limited
datasets.

Therefore, a systematic exploration of lightweight CNN methodologies is essential to make

image recognition feasible in practical, real-time environments with limited resources

Dept of CSE (Data Science), MIT Mysore 3

Deep Learning 2024-25

Objectives

The primary goals of the paper are:

• To introduce classical lightweight CNN models and explain how they reduce
parameters and computational cost.

• To review recent research advancements in lightweight image recognition methods.

• To classify optimization strategies into model compression, architectural

optimization, and CNN-Transformer hybrids.

• To summarize the theoretical foundations of techniques such as pruning,

quantization, distillation, and low-rank factorization.

• To compare experimental performance across widely used datasets like ImageNet,

CIFAR-10, and CIFAR-100.

• To identify gaps, challenges, and future directions for the development of

lightweight models.

• To provide a structured reference framework for researchers working on efficient

deep learning for image recognition.

Methodology / Approach

The authors follow a structured survey methodology:

1. Review of Classical Lightweight Networks

They begin by describing influential lightweight models such as:

• SqueezeNet – Uses 1×1 filters and fire modules for compression.

• MobileNet (V1–V3) – Introduces depthwise separable convolution and inverted

residuals.

• ShuffleNet – Utilizes channel shuffle and grouped convolutions.

2. Categorization into Three Optimization Strategies

A major contribution of the paper is the classification of methods into:

Dept of CSE (Data Science), MIT Mysore 4

Deep Learning 2024-25

1. Model Compression – Techniques applied to existing networks to remove redundancy.

2. Optimization of Lightweight Network – Creation of new architectures specifically

designed for efficiency.

3. Combining Transformers with Lightweight CNNs – Hybrid models using both

global and local representations.

3. Comparative Evaluation

The paper synthesizes results from multiple original research works by comparing:

• Accuracy

• Parameter count

• FLOPs

• Speed

• Efficiency trade-offs

4. Analytical Framework

The paper includes a conceptual diagram (Fig. 3) that visually organizes the categories of
lightweight image recognition methods.

This structured approach ensures consistency, clarity, and comprehensive coverage of the field.

Experimental Setup

Although this is a survey paper, it compiles and analyzes experimental results from numerous
studies.

Datasets Used for Evaluation

• ImageNet – 1.2M training images, diverse, high complexity.

• CIFAR-10 – Small images (32×32), 10 classes.

• CIFAR-100 – Similar to CIFAR-10 but with 100 classes.

Evaluation Metrics

Dept of CSE (Data Science), MIT Mysore 5

Deep Learning 2024-25

• Top-1 Accuracy – Measures exact class prediction match.

• Top-5 Accuracy – Used for large datasets like ImageNet.

• Parameters (in Millions) – Indicates model size.

• FLOPs – Indicates computational cost per image.

Training Details Across Referenced Works

Common elements across studies include:

• Use of SGD optimizers

• Standard learning rate schedules

• Varying bit-width configurations for quantization

• Different pruning ratios

• Hyperparameter tuning for NAS-based networks

The paper consolidates these diverse experiment results into unified comparison tables (Tables
6–11).

Results and Discussion

1. Pruning Results

Pruning eliminates redundant neurons or filters.

• CHEX improves accuracy on ResNet-34 while reducing FLOPs by 45%.

• DMCP outperforms other pruning approaches on MobileNetV2.

• Progressive pruning strategies avoid accuracy loss.

2. Quantization Results

Quantization reduces numerical precision of weights/activations.

• IR-Net and DIR-Net retain high accuracy even at extremely low bit widths.

• 1-bit binarized networks perform well on smaller datasets, but ImageNet requires
higher bit widths.

Dept of CSE (Data Science), MIT Mysore 6

Deep Learning 2024-25

3. Knowledge Distillation Results

Distillation transfers knowledge from a large teacher model to a smaller student model.

• ReviewKD, DKD, and KCD show large accuracy improvements for lightweight
models.

• Student networks benefit more when teachers have much higher capacity.

4. Low-Rank Factorization Results

Factorization reduces convolutional kernels into lower-rank matrices.

• BATUDE achieves accuracy improvements while reducing FLOPs by up to 2.97×.

• HRank identifies low-rank feature maps to decide pruning efficiently.

5. Lightweight Network Design

Modern lightweight architectures such as EfficientNet, GhostNet, FBNetV3, and CCNNet

offer outstanding trade-offs.

• FBNetV3 achieves the highest accuracy among lightweight CNNs evaluated.

• EfficientNet uses compound scaling to balance depth, width, and resolution.

6. CNN–Transformer Hybrids

Models like MobileViT, CMT, HybridNet successfully integrate global attention with local
convolutions.

• Mobile-Former achieves higher accuracy than Swin-2G with far fewer FLOPs.

• CMT-Ti reduces computation by two-thirds while maintaining transformer-level

accuracy.

These results illustrate that lightweight CNNs can achieve state-of-the-art performance while
remaining efficient and deployable on constrained hardware.

Dept of CSE (Data Science), MIT Mysore 7

Deep Learning 2024-25

Limitations

1. Dependence on Large, High-Quality Datasets

Lightweight networks still require large datasets for training, and performance drops in low-
data scenarios.

2. Low Interpretability

Lightweight CNNs compress or alter intermediate representations, making it difficult to

understand how the model behaves internally.

3. Computational Cost During Design

NAS-based and hybrid models require high compute during training/search, even though
inference is lightweight.

4. Accuracy vs Efficiency Trade-off

Excessive compression may lead to information loss and degrade accuracy.

5. Hardware Constraints

Some compression techniques rely on specialized hardware (e.g., INT8 support), limiting
universal deployment.

These limitations highlight the balance required between accuracy, efficiency, and practical
usability.

Key Insights

Lightweight CNNs are becoming essential as AI moves from cloud servers to edge devices.

• Model compression (pruning, quantization, distillation) significantly enhances

efficiency but must be applied with caution to avoid accuracy loss.

• Architecturally optimized lightweight networks (GhostNet, ShuffleNet, EfficientNet)

often outperform compressed large models because they are designed from the ground
up for efficiency.

• Hybrid CNN–Transformer models represent the future of image recognition because

they combine local feature extraction with global awareness.

Dept of CSE (Data Science), MIT Mysore 8

Deep Learning 2024-25

• Neural Architecture Search (NAS) will likely automate and accelerate lightweight
model development.

• Interpretability and dataset diversity remain open research challenges that must be
addressed for real-world adoption.

Conclusion

Lightweight CNN research has grown rapidly due to the increasing demand for deploying
image recognition models on mobile and embedded devices. This survey provides a detailed
overview of classical lightweight models and recent optimization strategies across three major
categories. Experimental comparisons show that lightweight models can maintain strong
accuracy while dramatically reducing computational cost. The paper concludes by identifying
future research opportunities, particularly in improving dataset quality, enhancing model
interpretability, and reducing design-time computation through methods like neural
architecture search. Overall, lightweight CNNs hold significant promise for making deep
learning more accessible, deployable, and efficient in real-world environments.

References

[1] L. Li, X. Mu, S. Li, H. Peng, A review of face recognition technology, IEEE Access 8 (2020)
139110–139120.

[2] S. Jia, J. Liao, M. Xu, Y. Li, J. Zhu, W. Sun, X. Jia, Q. Li, 3-d gabor convolutional neural
network for hyperspectral image classification, IEEE Trans. Geosci. Remote Sens. 60 (2022)
1–16.

[3] I. Arel, D.C. Rose, T.P. Karnowski, Deep machine learning-a new frontier in artificial
intelligence research [research frontier], IEEE Comput. Intell. Mag. 5 (4) (2010) 13–18.

[4] Y. Tian, Artificial intelligence image recognition method based on convolutional neural
network algorithm, IEEE Access 8 (2020) 125731–125744.

[5] Y. LeCun, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard, L. D. Jackel,
Backpropagation applied to handwritten zip code recognition, Neural Comput. 1 (4) (1989)
541–551.

Dept of CSE (Data Science), MIT Mysore 9

Deep Learning 2024-25

2. Attention Is All You Need

Authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan
Gomez, Łukasz Kaiser, Illia Polosukhin
Published: NeurIPS 2017

This is the breakthrough paper that introduced the Transformer architecture, now the
foundation of modern LLMs (GPT, BERT, T5, etc.).

Abstract

The paper proposes the Transformer, a novel sequence-to-sequence architecture that

eliminates both recurrence and convolution, relying solely on attention mechanisms to
model global dependencies in sequence data. Prior state-of-the-art models, such as RNNs,
LSTMs, GRUs, and CNN-based approaches, require sequential computation or large
receptive fields, making them computationally expensive and limiting parallelization. The
Transformer overcomes these challenges by using self-attention to relate all input and output
positions regardless of distance, allowing complete parallelization during training and
significantly reducing training time. Experimental results on large-scale machine translation
tasks—specifically WMT 2014 English-to-German and English-to-French—demonstrate
that the Transformer achieves state-of-the-art BLEU scores, outperforming previous models
and ensembles while using only a fraction of their training computation. The model achieves
28.4 BLEU for EN→DE and 41.0 BLEU for EN→FR, establishing a new benchmark in
translation quality. The paper concludes that attention mechanisms alone are sufficient for
high-quality sequence transduction and that the Transformer represents a major step forward
in sequence modeling.

Introduction

The authors begin by explaining the dominance of RNN-based architectures such as

LSTMs and GRUs in sequence modeling tasks like machine translation. These networks
process sequences step-by-step, which restricts parallelism and leads to slow training,
especially on long sequences. Convolutional models attempt to increase parallelization but
require deep stacks of layers to capture long-range dependencies, increasing computational
cost.

Dept of CSE (Data Science), MIT Mysore 10

Deep Learning 2024-25

The key limitation across all prior models is sequential computation, which slows down
processing and prevents efficient GPU usage. At the same time, attention mechanisms had
already shown strong potential for learning relationships across distant positions, but they
were always paired with RNNs.

The Transformer challenges this paradigm by showing that attention alone—without any
recurrence—can produce high-quality sequence representations. This allows the model to
process all input positions simultaneously, drastically speeding up training. The authors
highlight that the Transformer is the first architecture to rely entirely on self-attention and
achieves state-of-the-art translation performance with significantly reduced computation
time.

Problem Statement

Existing sequence transduction models face several challenges:

1. Lack of Parallelization

RNNs process tokens one at a time, preventing parallel computation and causing slow
training.
2. Difficulty Modeling Long-Range Dependencies

Both RNNs and CNNs require many sequential steps or deep layers to relate distant tokens,
increasing complexity.

3. High Computational Cost

State-of-the-art CNN and RNN models demand huge training resources, making them costly
and inefficient. Examples include GNMT and ConvS2S, which consume significant GPU
time.

4. Need for Faster, More Scalable Models

As datasets grow larger, the limitations of sequential architectures become more pronounced.

The paper addresses these issues by introducing a model that:

• removes recurrence entirely

• uses self-attention as the core mechanism

• supports full parallelization

Dept of CSE (Data Science), MIT Mysore 11

Deep Learning 2024-25

• provides shorter gradient paths for long-range dependencies

Objectives

The main goals of the authors are:

• To design a fully attention-based architecture for sequence modeling

• To eliminate recurrence and convolution, achieving maximum parallelization

• To prove that self-attention is sufficient for high-quality sequence transduction

• To compare computational complexity of attention vs. RNN vs. CNN models

• To evaluate the model on large machine translation benchmarks

• To reduce training time and improve scalability on GPUs

• To demonstrate state-of-the-art BLEU scores with far lower computational cost

Methodology / Approach

The Transformer follows a standard encoder–decoder architecture, but with revolutionary

internal components:

1. Encoder Stack (6 layers)

Each layer contains:

• Multi-head self-attention

• Position-wise feed-forward network

• Residual connections + LayerNorm

2. Decoder Stack (6 layers)

Each layer adds a third sub-layer:

• Masked self-attention (to prevent looking at future tokens)

Dept of CSE (Data Science), MIT Mysore 12

Deep Learning 2024-25

• Encoder–decoder attention

3. Scaled Dot-Product Attention

Attention(Q, K, V) = softmax(QKᵀ / √dk) V

Scaling prevents gradient problems for large dk.

4. Multi-Head Attention

Eight attention heads capture relationships in different representation subspaces.

5. Positional Encoding

Since the model has no recurrence or convolution, positional information is injected using
sine/cosine encodings based on frequency.

6. Feed-Forward Networks

Two linear layers with ReLU, identical for all positions.

This methodology results in a fully parallelizable architecture with extremely short path
lengths for dependencies.

Experimental Setup

Datasets

• WMT 2014 English–German (4.5M sentence pairs)

• WMT 2014 English–French (36M sentence pairs)

Data Encoding

• Byte Pair Encoding (BPE) with 37k tokens (EN–DE)

• Word-piece vocabulary of 32k tokens (EN–FR)

Dept of CSE (Data Science), MIT Mysore 13

Deep Learning 2024-25

Batching

Sentences grouped by similar length, each batch containing ~25k source & 25k target tokens.

Hardware

• Trained on 8× NVIDIA P100 GPUs

• Base model training time: 12 hours

• Big model training time: 3.5 days

Optimizer

Adam with learning rate schedule:

lrate = dₘₒdₑₗ⁻⁰·⁵ × min(step⁻⁰·⁵, step × warmup_steps⁻¹·⁵)
with warmup_steps = 4000.

Regularization

• Dropout = 0.1 (or 0.3 for big model EN–FR)

• Label smoothing = 0.1

Results and Discussion

1. Machine Translation Performance

On WMT14 English→German:

• Transformer (big): 28.4 BLEU → New SOTA

• Transformer (base): 27.3 BLEU

On WMT14 English→French:

• Transformer (big): 41.0 BLEU → Best single-model performance

Dept of CSE (Data Science), MIT Mysore 14

Deep Learning 2024-25

2. Training Cost Comparison

Transformers achieve better accuracy with much lower FLOPs than RNN or CNN models.
For example:

• ConvS2S ensemble EN–FR: 1.2×10²¹ FLOPs

• Transformer big EN–FR: 2.3×10¹⁹ FLOPs

3. Model Ablations

• Too few heads (1–4) reduces accuracy

• Very large feed-forward dimension improves performance

• Sinusoidal vs. learned positional embeddings → similar performance

• Dropout significantly prevents overfitting

4. Interpretability

Attention heads learn syntactic and semantic patterns without explicit guidance.

5. Key Insight

Attention is efficient, parallelizable, and highly expressive—making recurrence unnecessary.

Limitations

Although revolutionary, the Transformer has some weaknesses:

1. Quadratic Complexity with Sequence Length

Self-attention requires O(n²) operations due to the QKᵀ matrix.

2. Requires Positional Encoding

Because the model has no recurrence, positional information must be manually injected.

3. Limited at Extremely Long Sequences

For very long sequences, computation becomes expensive; restricted attention may be needed
(as noted by authors).

4. Data and Compute Hungry During Training

Dept of CSE (Data Science), MIT Mysore 15

Deep Learning 2024-25

While inference is fast, training large Transformer models still requires significant GPU
resources.

5. Potential Overfitting Without Regularization

Dropout and label smoothing are crucial—models overfit easily without them.

Key Insights

• The Transformer introduced the idea that attention alone can outperform
architectures using recurrence or convolution.

• Multi-head attention allows the model to focus on different linguistic patterns

simultaneously.

• The architecture is fully parallel, enabling massive speed-ups on GPUs.

• Positional encoding is a clever way to handle sequence order without recurrence.

• This paper laid the groundwork for modern LLMs, proving that scaling attention-
based models leads to dramatic improvements.

• The design’s simplicity, modularity, and efficiency make it one of the most influential
architectures in AI history.

• The Transformer’s ability to learn long-range dependencies with short computational

paths was a major breakthrough.

Conclusion

The paper introduces the Transformer, the first architecture to remove recurrence entirely and
operate solely using attention. Through experiments on large machine translation tasks, the
model demonstrates superior performance, faster training, and much lower computational
cost compared to RNN and CNN models. By combining multi-head self-attention, feed-
forward layers, positional encoding, and full parallelization, the Transformer sets a new
standard for sequence modeling. Its success has inspired a new generation of models across
NLP, vision, speech, and multimodal domains.

Dept of CSE (Data Science), MIT Mysore 16

Deep Learning 2024-25

References

[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv
preprint arXiv:1607.06450, 2016.

[2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by
jointly learning to align and translate. CoRR, abs/1409.0473, 2014.

[3] Denny Britz, Anna Goldie, Minh-Thang Luong, and Quoc V. Le. Massive exploration of
neural machine translation architectures. CoRR, abs/1703.03906, 2017.

[4] Jianpeng Cheng, Li Dong, and Mirella Lapata. Long short-term memory-networks for
machine reading. arXiv preprint arXiv:1601.06733, 2016.

[5] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk,
and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical
machine translation. CoRR, abs/1406.1078, 2014.

Dept of CSE (Data Science), MIT Mysore 17

Deep Learning 2024-25

[Link] of BERT Sentence Embedding for Text Classification

Authors: Omar Galal, Ahmed H. Abdel-Gawad, Mona Farouk

Published: Neural Computing and Applications, 2024

This paper investigates how BERT embeddings—final layer, hidden layers, and aggregated
representations—can be better leveraged for text classification tasks such as Arabic
sentiment analysis and sarcasm detection, proposing improved aggregation architectures
that outperform traditional fine-tuning strategies.

Abstract

This work revisits the standard practice of using BERT for text classification, where the
[CLS] token embedding is commonly used as the sentence representation. The authors argue
that this default approach underutilizes the richness of BERT’s contextual token embeddings
and hidden layers. To address this, they propose multiple aggregation architectures that
learn better ways to combine BERT embeddings for classification tasks. Experiments on
Arabic sentiment analysis and sarcasm detection demonstrate that these aggregated
representations deliver superior performance. Surprisingly, the study shows that freezing
BERT—treating it as a feature extractor—can outperform full fine-tuning, resulting in better
accuracy and drastically reduced training time. Further, the paper introduces multi-task
learning architectures where sentiment, sarcasm, and dialect classification are jointly trained.
These models achieve new state-of-the-art results on the ArSarcasm-v2 dataset, including
an F1-Sarcastic score of 64.41 and an FPN (sentiment metric) of 75.26. The findings
emphasize that BERT’s hidden layers contain valuable information, and learning customized
aggregation strategies can outperform traditional approaches without requiring expensive
model retraining.

Introduction

Text classification plays a central role in various NLP applications including sentiment
analysis, sarcasm detection, abusive language detection, depression detection, and topic
categorization. Historically, classification models used linguistic features, TF–IDF vectors, or

Dept of CSE (Data Science), MIT Mysore 18

Deep Learning 2024-25

neural architectures such as CNNs and LSTMs. The emergence of pre-trained language
models like BERT has significantly advanced classification performance.

However, the standard BERT usage pattern—extracting the [CLS] token embedding and
fine-tuning the entire model—does not fully utilize the rich token-level contextual
representations that BERT produces. The authors highlight that BERT generates contextual
embeddings for every token across 12 hidden layers, but the default method only uses a
single vector.

Building on prior work such as Sentence-BERT and attention-based pooling methods, this
research rethinks how sentence embeddings should be constructed for classification tasks.
The paper argues that:

• The [CLS] embedding may not be the optimal sentence representation.

• BERT hidden layers hold additional semantic information.

• Freezing BERT and training only external aggregation layers can reduce cost and
improve performance.

• Multi-task learning can enhance performance for related tasks like sentiment +
sarcasm detection.

Thus, the paper proposes new aggregation architectures and evaluates them over Arabic
sentiment and sarcasm datasets.

Problem Statement

Although BERT significantly improves text classification performance, its default usage for
classification has limitations:

✔ Using only the [CLS] embedding may not capture the complete semantic meaning of the
sentence.
✔ Final-layer embeddings alone might not be sufficient; earlier hidden layers also carry
meaningful representations.
✔ Fine-tuning BERT is computationally expensive, slow, and requires high memory—often
unnecessary.
✔ It remains unclear whether problem type (e.g., sentiment vs sarcasm) affects the optimal

Dept of CSE (Data Science), MIT Mysore 19

Deep Learning 2024-25

aggregation strategy.
✔ Multi-task learning is underexplored: Can sentiment and sarcasm jointly improve
classification?
✔ Is it possible that freezing BERT (no fine-tuning) could actually surpass fine-tuning by
preventing overfitting?

This paper attempts to systematically answer these questions by designing and evaluating
new aggregation architectures using BERT token and layer embeddings.

Objectives

The authors set the following goals:

• To evaluate whether the [CLS] embedding is the best sentence representation for
classification.

• To design and propose aggregation architectures that learn how to combine BERT’s
output embeddings more effectively.

• To study whether final-layer embeddings are enough, or if hidden layers should

also be aggregated.

• To determine how task type (sentiment vs sarcasm) influences the optimal strategy.

• To compare performance when BERT is frozen vs fine-tuned.

• To explore multi-task learning architectures (sentiment + sarcasm + dialect) and

measure how task interaction influences performance.

• To achieve new state-of-the-art results on the ArSarcasm-v2 benchmark.

Methodology / Approach

The methodology focuses on designing new architectures that aggregate BERT’s contextual
embeddings more effectively. According to the diagrams and descriptions (pages 3–6), the
authors introduce:

Single-Task Architectures

Dept of CSE (Data Science), MIT Mysore 20

Deep Learning 2024-25

1. Average Aggregator

o Averages all token embeddings to create a sentence vector.

o Hypothesis: pooling all token information may outperform using [CLS].

2. Attention Aggregator

o Adds a trainable self-attention layer on top of BERT outputs.

o Learns how to weigh token embeddings based on importance.

3. Residual Attention Aggregator

o Adds a skip connection from [CLS] to the attention output.

o Helps the model decide between [CLS] and aggregated embeddings.

4. Transformer Aggregator

o Places a Transformer encoder layer above BERT.

o More expressive and powerful than single attention.

5. Parallel Aggregator (P-SUM)

o Uses the last four hidden layers of BERT.

o Each layer feeds into its own classifier; outputs are averaged.

o Motivated by research indicating different layers encode different linguistic

features.

6. Hierarchical Aggregator (H-SUM)

o Stacks the outputs of BERT layers hierarchically.

o More complex but allows deeper contextual fusion.

Dept of CSE (Data Science), MIT Mysore 21

Deep Learning 2024-25

Multi-Task Learning Architectures

The authors extend P-SUM and H-SUM to jointly learn:

• Sentiment

• Sarcasm

• Dialect

Two kinds of MTL strategies are tested:

1. MTL Without Informing

o Tasks share BERT layers but do not explicitly pass predictions to each other.

o Sharing leads to consistent improvements in sarcasm detection.

2. MTL With Explicit Informing

o Outputs of one task feed into another (e.g., sarcasm informed by sentiment).

o Multiple strategies tested: All-informing-all, T1-informed-others, etc.

o Performance depends heavily on label distribution interactions.

Training Setup

• Implemented in PyTorch.

• Pretrained models from HuggingFace.

• BERT frozen vs fine-tuned evaluated separately.

• MAX length = 128 tokens (tweets).

• Optimizer: Adam.

• 5 epochs, learning rate between 1e-6 and 1e-3 depending on freezing.

Dept of CSE (Data Science), MIT Mysore 22

Deep Learning 2024-25

Experimental Setup

Dataset

The experiments use ArSarcasm-v2, which contains:

• ~12k training tweets

• ~3k test tweets

• Labels:

o Sentiment (POS, NEG, NEU)

o Sarcasm (True/False)

o Dialect (MSA, EGY, LEV, Gulf, NOR)

Shown in Table 1 of the paper.

Models Evaluated

6 transformer models: MARBERT, ARBERT, Qarib, Arabic-BERT, GigaBERT, mBERT.

MARBERT performs best due to training on Arabic tweets.

Tasks Measured

• Sentiment metric: FPN (average F1 of POS & NEG).

• Sarcasm metric: F1-Sar (F1 score for sarcastic class).

Experiment Dimensions

The authors systematically test:

• Frozen vs fine-tuned models

• Final-layer vs hidden-layer aggregation

• Single-task vs multi-task

• With vs without informing

Dept of CSE (Data Science), MIT Mysore 23

Deep Learning 2024-25

Results and Discussion

Sentiment Analysis – Single Task

• Attention and Transformer aggregators outperform the default [CLS] approach for
most models.

• MARBERT + AttentionAggregator = Highest performance among simpler models.

• Using hidden layers (P-SUM and H-SUM) significantly improves results:

o Best FPN = 75.26, new state-of-the-art.

o Freezing MARBERT performs better than fine-tuning.

Sarcasm Detection – Single Task

• Unlike sentiment, OrdinaryAggregator performs best for most models.

• Sarcasm is less sensitive to complex embedding structures, possibly due to smaller

dataset.

• H-SUM with MARBERT (frozen) achieves the best single-task F1-Sar.

Multi-Task Learning Results

• MTL without informing improves sarcasm performance dramatically:

o P-SUM-MTL = 63.6 F1-Sar

o H-SUM-MTL = 64.41 F1-Sar, best in the paper

• MTL with informing is tricky:

o Informing sarcasm using sentiment sometimes hurts performance due to label

imbalance.

o All-informing-all performs well for sentiment but can reduce sarcasm

accuracy.

Dept of CSE (Data Science), MIT Mysore 24

Deep Learning 2024-25

Final Achievements

• New State of the Art Results:

o Sentiment: 75.26 FPN

o Sarcasm: 64.41 F1

• Counter-intuitive but important: Frozen BERT outperforms fine-tuned BERT in

many architectures.

Limitations

Informing strategies can confuse the classifiers due to class distribution skewness (negative
tweets ≠ sarcastic tweets).

• Some architectures (H-SUM SeparateBERTs) are too computationally heavy and

prone to overfitting.

• Sarcasm detection remains challenging due to subtle linguistic cues, limited data, and
inconsistent annotation.

• Multi-task architectures require careful balancing to avoid performance degradation in

individual tasks.

Key Insights

• BERT’s hidden layers contain valuable complementary information; using only the
final layer is suboptimal.

• Simple pooling (AverageAggregator) works surprisingly well when BERT is frozen.

• Freezing BERT avoids overfitting and reduces training cost dramatically, while still
achieving SOTA results.

• Multi-task learning enhances performance for difficult tasks like sarcasm detection.

• Explicit informing between tasks must be applied carefully depending on dataset

properties.

Dept of CSE (Data Science), MIT Mysore 25

Deep Learning 2024-25

• Aggregation architectures can extract far more from BERT than the standard [CLS]
embedding.

Conclusion

The study demonstrates that rethinking how BERT embeddings are aggregated leads to
substantial performance improvements in text classification tasks. By experimenting with
alternative aggregation techniques—attention layers, Transformer encoders, and multi-layer
fusion—this research shows that BERT’s representational power extends far beyond the
[CLS] token. The authors achieve new state-of-the-art results in both Arabic sentiment
analysis and sarcasm detection, while also revealing the surprising advantage of freezing
BERT rather than fine-tuning it. The work encourages future researchers to explore pre-
trained models as feature extractors and to design smarter aggregation architectures rather
than relying on default BERT configurations.

References

1. Abbes I, Zaghouani W, El-Hardlo O, et al (2020) Daict: a dialectal arabic irony corpus

extracted from twitter. In: proceedings of the twelfth language resources and evaluation
conference, pp 6265–6271

2. Abdelali A, Hassan S, Mubarak H, et al (2021) Pre-training bert on arabic tweets: practical

considerations. arXiv preprint arXiv: 2102.10684 [[Link]]

3. Abdul-Mageed M, Elmadany A, Nagoudi EMB (2020) Arbert & marbert: deep

bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785

4. Abu Farha I, Magdy W (2020) From Arabic sentiment analysis to sarcasm detection: The
ArSarcasm dataset. In: Proceedings of the 4th workshop on open-source arabic corpora and
processing tools, with a shared task on offensive language detection. European language
resource association, Marseille, France, pp 32–39, [Link]

5. Abu Farha I, Magdy W (2021) Benchmarking transformer-based language models for

Arabic sentiment and sarcasm detection. In: proceedings of the sixth Arabic natural language
processing workshop. Association for computational linguistics, Kyiv, Ukraine (Virtual), pp
21–31, [Link] wanlp-1.3

Dept of CSE (Data Science), MIT Mysore 26

Deep Learning 2024-25

A Unified Understanding of Deep NLP Models for Text

Classification

Authors: Zhen Li, Xiting Wang, Weikai Yang, Jing Wu, Zhengyan Zhang, Zhiyuan Liu,
Maosong Sun, Hui Zhang, Shixia Liu
Published: 2022 (arXiv), supported by Tsinghua University and Microsoft Research.

The paper introduces DeepNLPVis, a unified visual analytics system for understanding,
debugging, and comparing deep NLP models for text classification through an information-
theoretic measure and multilevel visualization.

Abstract

The paper addresses the increasing complexity of deep NLP models—CNN-based, LSTM-
based, Transformer-based—and the difficulties developers face in understanding and
debugging them. Although many explainability tools exist, most are model-specific and
cannot provide a unified understanding of how different NLP architectures learn from text. To
overcome this gap, the authors propose DeepNLPVis, a visual analysis system built on a
mutual-information-based measure that quantitatively explains how each layer of an NLP
model preserves and transforms information from input words. The system models both
intra-word information (how individual words contribute to predictions) and inter-word
information (relationships between words such as phrase formation and non-adjacent
dependencies). DeepNLPVis provides a three-level coordinated visualization—corpus-level,
sample-level, and word-level—to support exploration from dataset-wide patterns to specific
prediction behaviors. Case studies on sentiment classification and model comparison show
that this tool enables developers to detect mislabeled samples, identify spurious correlations,
understand phrase learning behaviors, and compare architectures such as BERT and BiLSTM
under a unified interpretability framework.

Dept of CSE (Data Science), MIT Mysore 27

Deep Learning 2024-25

Introduction

Deep NLP models have rapidly evolved, resulting in improved text classification
performance but significantly increased architectural complexity. Models such as LSTMs,
CNNs, and Transformers often contain millions of parameters and complicated training
pipelines. This complexity makes understanding model behavior, diagnosing prediction
errors, and comparing architectures extremely difficult for developers.

The paper identifies key challenges:

• Developers often fine-tune large pretrained models (e.g., BERT), but debugging them
requires deep insights into layer-wise behavior.

• Current interpretability tools are fragmented—they focus on either RNNs or attention

models, lacking a unified way to analyze different architectures.

• NLP practitioners struggle with issues such as incorrect labels, biased samples, and
unexpected word interpretations, and existing tools provide little help in tracking
these issues across the model’s layers.

• Simpler models like CNNs or LSTMs could be viable alternatives in some cases, but
there is no unified evaluation mechanism to compare them with Transformers.

To address these problems, the authors propose DeepNLPVis, an interactive, unified

visualization tool that helps developers understand how any NLP model processes text across
layers, how words influence predictions, and how inter-word relationships evolve.

Problem Statement

Despite major advances in NLP, the field lacks a unified framework for understanding the
internal mechanisms of different deep text classification models. Key problems include:

• Existing explainability tools analyze only a specific model type (e.g., attention
visualizers for Transformers).

• Developers cannot systematically track how information flows across layers for
words, phrases, or dependencies.

• Debugging is difficult because errors may arise from data issues (e.g., mislabeled
samples), word-level misunderstandings, or architectural limitations.

Dept of CSE (Data Science), MIT Mysore 28

Deep Learning 2024-25

• There is no method to jointly understand intra-word information (word

contributions) and inter-word information (relationships, phrase formation).

• There is no unified method for comparing models like BERT, LSTM, and CNN using
consistent interpretability metrics.

These gaps create barriers to improving model performance, debugging, and selecting
suitable architectures for specific classification tasks.

Objectives

The paper aims to:

• Develop a unified interpretability measure that works across all NLP architectures.

• Reveal both word-level contributions and word relationships across layers.

• Build a multi-level visualization (corpus → sample → word) to assist developers in

identifying model weaknesses.

• Help discover issues such as mislabeled data, shortcut learning, and model
misunderstandings.

• Provide tools to compare multiple NLP models fairly and transparently.

• Support debugging of deep models and guide data augmentation or architectural

refinement.

Methodology / Approach

The approach combines two main innovations:

1. Mutual-Information-Based Interpretability Measure

This measure quantifies:

Intra-word information

How much each word contributes to the model’s final prediction.

Calculated using:

• Word embeddings at each layer

Dept of CSE (Data Science), MIT Mysore 29

Deep Learning 2024-25

• Perturbation-based mutual information (e.g., change in prediction when a word

embedding is perturbed)

The sign of contribution indicates polarity—toward which class the word pushes the
prediction.

Inter-word information

Captures relationships between words, including:

• Learned phrases (e.g., “a good movie”)

• Non-adjacent dependencies (e.g., “not” modifying a distant positive term)

This is extracted using:

• Context vectors

• Clustering at each layer

• Mutual information flow between word embeddings across layers

2. Three-Level Visual Analytics System (DeepNLPVis)

Corpus-Level Visualization

Provides overall model performance using:

• Confusion matrix

• Hexagonal heatmap with prediction score distribution

• Keyword importance distribution

Helps identify misclassified classes, biased samples, and important words.

Sample-Level Visualization

Displays:

• Complete text sample

• Word contributions

Dept of CSE (Data Science), MIT Mysore 30

Deep Learning 2024-25

• Layer-wise information flow visualization

This reveals phrase formation, sentiment shifting, mistake propagation, and turning structures
such as "though" or "if."

Word-Level Visualization

Shows:

• Contribution of a word across layers

• Polarity changes

• Word context clusters

This helps interpret ambiguous words like “care” or “like,” showing how context influences
meaning.

Experimental Setup

Models Evaluated

• BERT

• CNN-based classifier

• LSTM and BiLSTM

• ELMo + LSTM model

All models were analyzed under the same interpretability framework.

Datasets Used

• Stanford Sentiment Treebank (SST-2) for sentiment analysis

• Multiple datasets used in questionnaires and user studies

Participants

• Surveys conducted with 46 NLP developers

• Follow-up interviews with 5 experts (1–10 years of NLP experience)

Dept of CSE (Data Science), MIT Mysore 31

Deep Learning 2024-25

Analysis Workflow

• Models trained normally

• Word and layer-wise information extracted offline

• Visualizations generated for interactive debugging

Results and Discussion

Case Study 1: Debugging BERT for Sentiment Classification

DeepNLPVis revealed:

• BERT learns linguistic structure in early layers and task-specific features in deeper
layers.

• Words like “laughs,” “care,” and “absurd” gradually become strong sentiment
indicators.

• Ambiguous words such as “like” have multiple meaning clusters; BERT correctly
distinguishes them by context.

• Turning structures in sentences (“if…”, “though…”) are understood across layers,

improving prediction accuracy.

Misclassified samples showed issues like:

• Conflicting labels

• Shortcut learning (spurious correlations, e.g., “DVD” always tagged negative)

Case Study 2: Improving Model Performance

Developers used insights from visualization to:

• Fix mislabeled samples

• Remove biased or noisy words (e.g., “DVD,” “subconscious”)

Dept of CSE (Data Science), MIT Mysore 32

Deep Learning 2024-25

• Add adapter layers to BERT to improve handling of class-relevant words

This raised accuracy from 93.23% → 93.92%.

Case Study 3: Comparing BERT vs. ELMo+LSTM

DeepNLPVis enabled fair comparison:

• BERT had better overall accuracy and more stable keyword identification.

• ELMo+LSTM recognized rare sentiment words better due to its pretrained

embeddings.

• The unified measure made it possible to directly compare layer behaviors across very
different architectures.

Limitations

• The mutual-information computation is expensive, especially for large corpora.

• Visualization of extremely long texts becomes complex.

• Important curves (inter-word relations) must be filtered to avoid clutter—risk of

missing subtle dependencies.

• Tool currently focuses on text classification; extension to other NLP tasks requires
more research.

Key Insights

• Deep NLP models learn both word-level and phrase-level patterns across layers.

• Context strongly influences meaning; ambiguous words evolve differently based on

surrounding text.

• Misclassifications often stem from data issues, not model limitations.

• Simpler models (LSTM/CNN) can be competitive when analyzed and debugged

effectively.

Dept of CSE (Data Science), MIT Mysore 33

Deep Learning 2024-25

• A unified interpretability system greatly improves debugging and model comparison.

Conclusion

The paper presents a unified interpretability framework and visualization system for
understanding deep NLP models for text classification. By combining a mutual-information-
based measure with multi-level visualization, DeepNLPVis enables developers to analyze
model behavior from the dataset level down to individual words. The tool supports
debugging, data cleaning, model improvement, and architecture comparison. Case studies
demonstrate its effectiveness in improving sentiment classification accuracy, diagnosing data
issues, understanding phrase formation, and comparing BERT with LSTM models. The work
emphasizes that unified explainability is crucial for advancing NLP model development and
deployment.

REFERENCES

[1] S. Minaee, N. Kalchbrenner, E. Cambria, N. Nikzad, M. Chenaghlu, and J. Gao, “Deep

learning–based text classification: A comprehensive review,” ACM Computing Surveys, vol.
54, no. 3, pp. 1–40, 2021.

[2] Y. Kim, “Convolutional neural networks for sentence classification,” in the Conference on
Empirical Methods in Natural Language Processing, 2014, pp. 1746–1751. 13

[3] H. Strobelt, S. Gehrmann, H. Pfister, and A. M. Rush, “LSTMVis: A tool for visual
analysis of hidden state dynamics in recurrent neural networks,” IEEE Transactions on
Visualization and Computer Graphics, vol. 24, no. 1, pp. 667–676, 2018.

[4] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I.

Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing
Systems, 2017, pp. 5998–6008.

[5] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep
bidirectional transformers for language understanding,” in the Conference of the North
American Chapter of the Association for Computational Linguistics, 2019, pp. 4171–4186.

Dept of CSE (Data Science), MIT Mysore 34

Deep Learning 2024-25

5. Multi-Step Transfer Learning in Natural Language Processing

for the Health Domain

Authors: Thokozile Manaka, Terence Van Zyl, Deepak Kar, Alisha Wade (2024)

Abstract

The paper addresses the challenge of limited high-quality, annotated clinical text data, which
slows NLP progress in the health domain. Verbal Autopsy (VA) narratives, written by non-
clinicians, are noisy, contain spelling errors, and lack medical terminology, making text
classification for cause-of-death (COD) difficult. To overcome this, the authors propose a
Multi-Step Transfer Learning framework that leverages knowledge from multiple
domains—English, biomedical, and VA text—through feature extraction and fine-tuning.
The approach first uses ELMo, trained in the English domain, to handle linguistic
irregularities and character-level variations in VA texts. Then BERT, trained in the
biomedical domain, is initialized with these embeddings for improved domain alignment.
This hybrid approach enhances representation learning and leads to improved COD
classification accuracy. Experiments show that initializing BERT with ELMo embeddings
yields better performance than using each model alone, and the method generalizes well
across VA datasets, IMDb reviews, and medical transcription datasets.

Introduction

Many developing countries lack medical certification of deaths, so VA reports—narratives

describing symptoms and events before death—are used for COD estimation. These
narratives contain spelling mistakes, grammar inconsistencies, multiple languages, and
layperson terminology. Traditional NLP models struggle due to limited annotated data,
irregular linguistic patterns, and vocabulary mismatch with pretrained models.

Access to clinical data is further restricted by privacy policies, leading to fewer domain-
specific annotated datasets and slow adoption of NLP in healthcare. Existing pretrained
medical models like BioBERT or BioELMo perform well on biomedical text but still face
challenges with VA narratives because VA texts lack standardized terminology and contain
multilingual variations.

Dept of CSE (Data Science), MIT Mysore 35

Deep Learning 2024-25

The paper proposes Multi-Step Transfer Learning, a framework designed to:

• Capture generic linguistic features using ELMo trained on English text.

• Reduce domain divergence by initializing BERT (trained on PubMed biomedical text)

with ELMo-learned VA embeddings.

• Improve overall VA narrative classification, specifically for COD due to uncontrolled

hyperglycemia.

This approach exploits both character-level and word-level information, offering a more
robust solution than single-domain models.

Problem Statement

VA narratives are difficult to classify because:

• They contain spelling errors, inconsistent grammar, and local dialects.

• Domain-specific pretrained models do not match VA vocabulary.

• Label imbalance exists, especially for rare CODs like uncontrolled hyperglycemia.

• Training data is limited and often noisy.

• Existing transfer learning methods generally use a single source domain.

The key problem is: How to effectively adapt knowledge from multiple domains to
improve VA text classification when high-quality annotated medical data is scarce?

Objectives

• To develop a multi-step transfer learning framework combining English-domain

and biomedical-domain pretrained models.

• To improve VA text representation through character-level and word-level embedding

integration.

• To evaluate the benefit of initializing BERT with ELMo embeddings across multiple
datasets.

• To reduce the distribution gap between VA language and pretrained model domains.

Dept of CSE (Data Science), MIT Mysore 36

Deep Learning 2024-25

• To address class imbalance using cost-sensitive classification and specialized loss

functions.

Methodology / Approach

1. Multi-Step Transfer Learning Framework

The proposed framework consists of three stages:

Stage 1: Pretraining with ELMo (English Domain)

• ELMo learns character-level and contextual representations from English Wikipedia

and news data.

• Helps handle VA spelling variations, noisy text, and out-of-vocabulary words.

• ELMo embeddings are extracted and used to initialize BERT’s embedding layer.

Stage 2: Intermediate Training with BERT (Biomedical Domain)

• BERT pretrained on PubMed abstracts learns domain-specific medical terminology.

• ELMo-initialized BERT reduces domain mismatch between VA data and biomedical

text.

• Supports better understanding of symptoms and medical patterns relevant for COD
classification.

Stage 3: Fine-Tuning on VA Classification

• Fine-tuning is applied for the final task: predicting COD due to uncontrolled
hyperglycemia.

• A fully connected classification layer is added on top of BERT outputs.

• Cost-sensitive learning is used to address class imbalance.

Dept of CSE (Data Science), MIT Mysore 37

Deep Learning 2024-25

Experimental Setup

Datasets Used

• English Corpus: Wikipedia + News Crawl for ELMo.

• Biomedical Corpus: PubMed abstracts for BERT.

• Agincourt VA Dataset: 8,698 narratives; 77 cases due to uncontrolled

hyperglycemia.

• PHMRC VA Dataset: 11,979 records across multiple countries.

• IMDb Reviews: 50,000 balanced sentiment samples for generalization testing.

• Medical Transcriptions: 2,324 samples across 21 disease categories.

Models Used

• ELMo, BioELMo

• BERT Experts (WikiBooks, PubMed)

• Fine-tuned BERT variants with ELMo initialization

Hyperparameter Tuning

Various combinations of epochs (2–4), learning rates (2e−5 to 5e−5), and loss functions were
tested.
Results show Dice Loss performs better than Weighted Cross-Entropy for highly imbalanced
VA data.

Results and Discussion

1. ELMo Performance

• ELMo trained on English Wikipedia exhibited the lowest perplexity, meaning it best
captured language structure relevant to VA text.

• ELMo trained on small VA vocabulary performed poorly due to limited and noisy
text.

Dept of CSE (Data Science), MIT Mysore 38

Deep Learning 2024-25

2. Benefits of Multi-Step Transfer Learning

• BERT initialized with ELMo consistently outperformed standalone BERT.

• Best results occurred when ELMo was trained on English Wikipedia, not VA text.

• This demonstrates the advantage of using a large, clean linguistic corpus before
applying biomedical knowledge.

3. Comparison Across Datasets

• For both Agincourt and PHMRC VA datasets, Multi-Step Transfer Learning achieved
higher recall and F1-scores than individual models.

• Framework generalizes well to IMDb and Medical Transcription datasets.

• Fine-tuning adaptation consistently outperformed feature extraction.

4. Observations on Class Imbalance

• Weighted cross-entropy struggled due to extreme imbalance between positive and

negative COD cases.

• Dice loss was more stable and improved F1-score and AUC-ROC.

Limitations

• The study focuses solely on text classification; does not explore NER or relation
extraction.

• Computational constraints required splitting data into smaller batches, possibly

impacting embedding quality.

• VA datasets vary significantly across countries, limiting universal generalization.

• VA data still contain noisy language that even hybrid models may struggle with.

Dept of CSE (Data Science), MIT Mysore 39

Deep Learning 2024-25

Key Insights

• Combining character-level (ELMo) and word-level (BERT) knowledge improves

VA text representation.

• Domain mismatch can be effectively reduced through multi-step transfer learning.

• Fine-tuning offers better results than feature extraction for medical NLP tasks.

• VA text requires models capable of handling spelling variation, multilingual context,

and colloquial descriptions.

• The framework generalizes well across datasets of different sizes and domains.

Conclusion

The paper demonstrates that multi-step transfer learning significantly enhances VA text
classification for COD prediction, specifically in detecting deaths due to uncontrolled
hyperglycemia. By first learning linguistic structure using ELMo in the English domain and
then transferring biomedical knowledge with BERT, the model captures both general
language patterns and medical context effectively. The approach outperforms standalone
models, handles noisy data better, and adapts well across English, biomedical, and VA
domains. Future work includes extending this framework to NER, relation extraction, CNN-
enhanced architectures, and exploring COD prediction using ChatGPT.

References

1. United Nations (2013) Department of economic and social affairs, population division,
united nations. World Population Prospects: The 2012 revision

2. World Health Organisation (2007) Verbal autopsy standards: ascertaining and attributing
cause of death, Geneva. Switzerland, World Health Organisation

3. Hirschman L, Chapman WW, D’Avolio LW, Savova GK, Uzuner O (2011) Overcoming
barriers to NLP for clinical text: the role of shared tasks and the need for additional creative
solutions. J Am Med Inform Assoc 18(5):450–453 ++

Dept of CSE (Data Science), MIT Mysore 40

Deep Learning 2024-25

4. Ohno-Machado L, Nadkarni P, Chapman W (2011) Natural language processing: an

introduction. J Am Med Inform Assoc 18:544–51

5. Pan SJ, Yang Q (2010) A survey on transfer learning. IEEE Trans Knowl Data Eng
22(10):1345–1359

Dept of CSE (Data Science), MIT Mysore 41