A Comparative Analysis of Multimodal Misogyny

Memes using Deep Learning with Semi Supervised

Learning Algorithms
S. Karishma V. Akila
Research Scholar Associate Professor
Computer Science and Engineering Computer Science and Engineering
Puducherry Technological University Puducherry Technological University
Puducherry, India Puducherry, India
16karish@[Link] akila@[Link]
Abstract— The word "misogyny" is derived from the ancient One challenging issue that society faces is misogyny.
Greek word "mīsoguníā," which denotes a strong dislike for Throughout history, misogyny has taken many forms,
women. Social media services are increasingly being abused by including ideas such as male privilege, inflexible systems,
others to disseminate unpleasant and abusive information, in
discrimination based on gender, sexual harassment,
addition to providing people with a forum to express their
ideas, emotions and opinions. Detecting such content is
dehumanization of women, violence against women and
particularly complex, as the meaning conveyed by images and objectifying women [5] [6]. The United Nations' Sustainable
overlaid text often lacks a direct correlation. This ambiguity Development Goals recognized women as being at risk
increases the difficulty of automated detection, especially when category and they face a number of societal issues [7] [8].
memes propagate derogatory or hateful messages targeting Even in public settings, harassment and misconduct against
women. A route to more effective and efficient model training women are commonplace. Women frequently use social
is provided by Semi Supervised Learning (SSL) methods, media to share their stories and look for encouragement or
particularly in situations when labeled data is limited. To camaraderie from others going through similar struggles.
facilitate the automatic detection of misogynistic sentiment
Regrettably, certain members of society single out women
within these comments, SSL algorithms such as self training,
co-training and mean student-teacher models are explored who express their views online and condemn them for their
with MAMI dataset contains 10000 memes and results are hardships. Severe mental health issues are exacerbated by
compared. This work contributes to the ongoing efforts in this online harassment and abuse [9] [10]. Hence, it's crucial
automatic hate speech detection and presents a valuable to create computational models and methods that not only
dataset for further advancements in multimodal meme recognize and flag critical and offensive remarks directed at
detecting research. women but also aid in fostering a supportive and upbeat
Keywords— Cyberbullying, Social Media, Misogyny, Deep culture.
Learning, Gated Recurrent Unit, Semi Supervised Learning
User-generated content has increased due to social
media's recent expansion, making these sites vital
I. INTRODUCTION knowledge sources for a variety of users. Comments on
Cyberbullying refers to the act of harassing, threatening social media, which are frequently emotional, can have a
or targeting individuals through harmful or aggressive significant impact on people's and communities' mental
behavior on digital media including social networking sites, health. Of these, hostile and abusive information is
messaging services and online discussion forums, serve as particularly concerning because it often targets particular
platforms for cyberbullying. Unlike conventional forms of people or underprivileged groups. Therefore, it is essential
bullying, cyberbullying can persist indefinitely, allowing for to quickly detect and eliminate such dangerous meme
continuous occurrences reach a wide audience within a short content [11].
period, and be difficult to control due to its online nature.
The widespread growth of social media usage had the When labeled data is limited, SSL methods provide a
advent of cyberbullying as a major societal issue, presenting means to train models more effectively and efficiently. SSL
substantial challenges to online security and individual well- is cost-effective because it lessens reliance on large labeled
being [1]. datasets by using unlabeled data. SSL models can capture a
more complete data distribution by including unlabeled data,
"Mīsoguníā," is an olden Greek word which indicates a which frequently results in increased accuracy and
deep animosity to women, is where the word "misogyny" resilience. SSL has proved essential for NLP activities like
comes from. In addition to serving as a platform for language translation and sentiment analysis, particularly
handlers to express their thoughts, feelings and sentiments, when dealing with code-mixed input [18]. This article
social media sites are becoming more and more misused by provides the comparison of various SSL models to detect
others to spread offensive and abusive content. Despite the the text+image misogynistic content and examines the
fact that social media sites have both positive and bad application of various deep learning models for this task.
effects, people's general well-being may suffer as a result of The most successful strategy was determined by comparing
unpleasant content [2]. Hate speech on the internet has been and contrasting different models.
shown to have detrimental impacts on society rather than
only remaining online [3]. As a result, identifying and The manuscript is organized as follows: A thorough
reporting offensive postings, comments and discussions analysis of related literature is provided in Section 2. The
becomes crucial. The fact that many individuals actively dataset and the chosen methods are described in Section 3.
search for information on social media through posts and The experimental work is described in Section 4, and the
comments makes it even more important [4]. Verbal outcomes of the implemented model are analyzed in Section
harassment on the Internet has become a global problem 5. Finally, the article is concluded in Section 6.
during the last ten years, spreading across social media sites
such as Facebook, Instagram, X (early known as Twitter)
and YouTube. Risky groups like women, children, senior II. LITERATURE SURVEY
citizens and LGBT people often become the targets of such Jindal et al. [12] leveraged the potential of large language
content. As a result, it is crucial to identify such information models is leveraged via learning of transfer by constructing
and take the necessary corrective action. But because social embeddings of fusion, which are generated by extracting
media contains so much data, this is not possible by hand; and integrating features from multi-modalities, including
automated techniques are needed instead. images, text and text captioned images for the purpose of
misogyny detection. Their framework incorporated
advanced models, including the Vision Transformer (ViT),
DistilBERT for textual processing, CLIP (Contrastive
Language–Image Pre-training), and BLIP (Bootstrapping
Language–Image Pre-training) using the SemEval-2022
Task 5 MAMI dataset.

Hua et al. [13] proposed a new framework based on

machine learning, namely the BERT-based back-translation
text and entire-image multimodal model with contrast
learning (TTEC). In order to help the framework
understand certain generic features about a given issue, the
news text is first back-translated. Second, to generate
multimodal characteristics, a BERT-based model is given
both textual and visual information. Thirdly, by using
comparable news that has already been released, contrastive
learning is used to generate more logical multimodal
representations.

Zhang et al. [14] examined how well the dual-stream

CLIP multimodal pretrained models and the single-stream
UNITER multimodal pretrained models handled image/text
pairings with and without substantial correlation. With
image characteristics taken from the CLIP model, the
XGBoost classifier performs best and is resilient to domain
change. In light of this, we suggest the PBR system, an
ensemble system combining rule-based adjustment,
pretraining models, and boosting techniques. Our late
sequential fusion methodology is employed to fuse text data
into the system.

Grasso et al. [15] proposed the PBR system, a

collaborative agenda integrating pretrained models with a
boosting mechanism and rule-based refinements. This
system employs a late sequential fusion strategy for
incorporating textual data. Additionally, they introduced Fig. 1. Architecture diagram of detecting misogynistic memes using RNN
KERMIT (Knowledge-Empowered Model in Harmful
Meme Detection) is an innovative outline developed for B. Data Preparation
knowledge-enriched classification of destructive memes.
Loading and Cleaning: The preprocessing pipeline begins
Cao et al. [16] introduced PromptHate, an advanced by loading the MAMI dataset, followed by the removal of
framework that utilizes implied and formless knowledge redundant columns and the cleaning of textual data. This
entrenched within Pretrained Language Models (PLMs) for step involves eliminating punctuation, converting text to
the categorization of hateful memes. This approach involves lowercase, and segmenting it into individual words.
a PLM that processes a concise textual representation, Preprocessing: Subsequently, essential text preprocessing
which is derived from both an image captioning model and a techniques are applied, including tokenization (dividing text
Natural Language Processing (NLP) model. By utilizing the into words or sub-words), the removal of stop words (such
extensive world knowledge embedded in the PLM, as "the," "a," and "is"), and lemmatization (reducing words
PromptHate determined whether a given meme propagates to their root forms).
hate or is non-hateful. Splitting Data: The dataset is subsequently divided into
training and testing ensuring a systematic evaluation of the
III. METHODOLOGY structure’s performance.
A. Dataset description Word Embeddings: The textual data is represented by
means of pre-trained GloVe word embeddings, which
Misogyny detection is assessed using the SemEval-2022 encode words as numerical vectors, effectively capturing
Task 5 dataset [17] comprising 10,000 training memes and their semantic relationships.
1,000 testing memes.
C. Model Building
TABLE I. EXAMPLE OF MISOGYNOUS AND NON-MISOGYNOUS
MEMES The core architecture of the proposed approach is a A
Recurrent Neural Network (RNN) utilizing Gated Recurrent
Units (GRU) is employed. This neural network architecture
is well-suited for handling sequential data, particularly 4. Learning rate:
textual information as it captures dependencies and
contextual information effectively. GRU, an optimized Description: The learning rate is a key hyperparameter that
variant of Long Short-Term Memory (LSTM) networks, dictates the size of steps taken by the optimizer when
reduces computational complexity by incorporating update updating weights, directly influencing the model’s
(z) and reset (r) gates [6]. convergence speed and performance.
Impact: More learning rate might lead to faster
z = σ (xt Uz + st-1 Wz) (1) convergence but could exceed the optimal solution. A little
r = σ (xt Ui + st-1 Wr) (2) learning rate might affect in slower convergence but
potentially a more stable and accurate model.
Search Space: The code explores learning rates between
1e-3 and 1e-2 on a logarithmic scale.

Semi-Supervised Learning

Between supervised learning, which uses labeled data,

and unsupervised learning, which only uses unlabeled data,
there is a subset of machine learning called semi-supervised
learning (SSL). A little amount of labeled data is mixed with
a large amount of unlabeled data to improve model
performance.
Fig. 2. Internal architecture of Gated Recurrent Unit (GRU)
Working of Semi-Supervised Learning
D. Hyperparameters Setting
1. Use the small, labeled dataset to train a model.
1. Units: 2. Forecast categories for the unlabeled data using the
trained model.
Description: This hyperparameter determines the quantity 3. Incorporate high-confidence predictions into the
of units (or neurons) in GRU layer of the model. training set.
Impact: The number of units influences the model's ability 4. Retrain the model iteratively to refine its predictions.
to learn complicated structure in the data. More complex
interactions may be captured with a larger number of units, Self-Training in Semi-Supervised Learning
but overfitting may also become more likely. On the other
hand, underfitting could result from using fewer units. Self-training is a simple but powerful semi-supervised
Search Space: In the code, units are explored within the learning method that creates pseudo-labels for unlabeled
range of 32 to 256 with a step of 32, meaning the tuner tries data by using a model that was first trained on a small
values like 32, 64, 96, and so on. labeled dataset. The model's performance is then iteratively
improved by retraining it on both labeled and confidently
2. Dropout: pseudo-labeled data.
Description: Dropout is a regularized method that helps to
avoid overfitting by casually ignoring a fraction of units
during training.
Impact: By introducing this randomness, dropout compels
the model to study more robust features and reduces its
reliance on specific units. This can lead to better overview
performance on unseen data.
Search Space: The code explores dropout rates between 0.2
and 0.5 with a step of 0.1.

3. Optimizer:

Description: The optimization process in neural networks Co training in Semi-Supervised Learning

involves adjusting model weights during training based on
the computed loss function. The optimizer plays a crucial A semi-supervised learning approach called co-training
role in this procedure, determining how model updates its makes the assumption that features can be divided into two
parameters. redundant and independent viewpoints. The model is
Impact: The advantages and disadvantages of optimizers iteratively improved by using a big amount of unlabelled
vary. Choosing the right optimizer can significantly affect data and a small amount of labelled data.
the model's performance.
Search Space: The code considers three optimizers: 'adam',
'rmsprop', and 'nadam'.
  Execution environment: The GRU models were
executed using TensorFlow as the backend on the Keras
2.0 API with Python 3.10.12 in a Windows 11 64-bit
operating system.

The easyocr package is employed to extract text from

images, which is subsequently stored as a CSV file with
binary labels. The CSV file undergoes preprocessing,
including text normalization (conversion to lowercase),
removal of numerical values and special characters,
stopwords removal and lemmatization. The processed text is
then transformed into numerical representations using
GloVe word embeddings for MAMI dataset.

Following preprocessing, the dataset is separated into

25% training data and 75% testing data. The classification
Mean Teacher Model in Semi-Supervised Learning
task is performed using the GRU-based structure,
distinguishing between misogynous and non-misogynous
The Mean Teacher Model is a semi-supervised learning
content. We computed metrics such as accuracy, precision,
method that improves model training by encouraging
recall and F1-Score by executing self training, cotraining
consistency between a student model (which is being
and mean teacher student SSL models. In this proposed
trained) and a teacher model (which is an exponential
approach, self-training approach using MAMI dataset
moving average of the student). This approach helps the
outperformed better than the other SSL algorithms,
model generalize better using unlabeled data by enforcing
achieving a training accuracy of 95.28% and a testing
consistency between predictions under small perturbations
accuracy of 79.64%. These optimal parameters contribute to
(such as noise or data augmentation).
enhanced model performance in classifying misogynistic
memes while ensuring robustness and generalizability.

V. PERFORMANCE EVALUATION METRICS

One essential technique for assessing a classification
model's performance is a confusion matrix. It offers a
comprehensive analysis of the model’s predictive results by
systematically categorizing its predictions into four distinct
outcomes:
 True Positive (tp): Comments that are actual and
forecasted labels classify the meme as non-
misogynistic.
 True Negative (tn): Comments that are actual and
forecasted labels classify the meme as misogynistic.
 False Positive (fp): Comments that are actual meme is
misogynistic but the model wrongly forecasted it as
E. Model Training and Evaluation non-misogynistic.
 False Negative (fn): Comments that are meme is non-
The framework is trained using the fit method, where it misogynistic but the model wrongly forecasted it as
learns to classify input text sequences as either misogynous misogynistic.
or non-misogynous.
The performance of the GRU architecture is assessed
IV. RESULTS AND DISCUSSION using multiple terms which provide perceptions of the
model’s categorization capabilities. These metrics are
Performance is evaluated using SSL to identifying briefed as:
misogynistic memes on social media. The experimental
setup is implemented on Google Colaboratory, leveraging Accuracy
GPU acceleration. The ratio of truly categorized records relative to the
Software and Libraries: entire number of records in the dataset. It serves as a general
measure of effectiveness of the model.
 Data processing and preparation: Conducted using
pandas (v2.1.4) and numpy (v1.26.4). Accuracy = (3)
 Deep learning framework: Implemented using
TensorFlow (v2.17.0).
Precision
 Model evaluation: Performed using scikit-learn
(v1.5.2).
 The ratio of correctly forecasted non-misogynistic VI. CONCLUSION
memes (true positives) to the entire number of memes the This study explored three SSL models by training Gated
model forecasted as non-misogynistic. It quantifies the Recurrent Unit (GRU) model with GloVe embedding for
reliability of the model’s positive predictions. MAMI dataset for misogyny detection. The model is
optimized using the RMSprop optimizer with the following
Precision = (4) hyperparameter configurations: input size of 100 units, GRU
size of 128 units, dropout rate of 0.3, ReLU and sigmoid
activation functions, learning rate of 0.001, and batch size of
64. In this proposed approach, self-training approach using
Recall
MAMI dataset outperformed better than the other SSL
The ratio of correctly forecasted non-misogynistic
algorithms, achieving a training accuracy of 95.28% and a
memes to the entire number of actual non-misogynistic
testing accuracy of 79.64%. Models are better able to
memes. This metric indicates the capability of the technique
understand the subtle aspects of language when both labeled
to recognize positive records accurately.
and unlabeled data are included. Some of the challenges of
SSL are:
Recall = (5)
Data Quality Assurance: It is crucial to guarantee the
F1-score caliber of unlabeled data. Data that is of poor quality or
A stable metric that takes both incorrect positives and irrelevant might cause model confusion and impair
incorrect negatives is the harmonic average of recall and performance. To preserve data integrity, routine audits and
precision. When there are class imbalances in the dataset, it data cleaning procedures are necessary.
is especially helpful.
Domain Knowledge Incorporation: By utilizing domain
F1 - score = 2 . (6) knowledge, filters and rules may be created to pre-process
and screen unlabeled input, lowering the possibility of
include inaccurate information.
TABLE II. PERFORMANCE EVALUATION METRICS FOR MAMI
DATASET Robust Model Design: It's critical to create models that can
F1- recognize and ignore aberrant data points or outliers in the
Model Accuracy Precision Recall unlabeled dataset. This capacity to withstand noise
Score
contributes to the learning process's continued
Self
79.64 82.13 77.21 79.59 dependability.
training
Co-training 77.6 82.46 71.69 76.7
The SSL models can also use for multimodal, non-
Mean
english and code-mixed datasets with necessary techniques
teacher
76.28 81.58 73.25 75.45 and embeddings in future.
student
model REFERENCES

In Table II, it depicts that the performance of three SSL [1] Sihab-Us-Sakib, S., Rahman, M. R., Forhad, M. S. A., & Aziz, M. A.
models using MAMI dataset in which self-training model Cyberbullying detection of resource constrained language from social
has outperformed with other two algorithms. media using transformer-based approach. Natural Language
Processing Journal, 100104, 2024.
[Link]
[2] W. Akram and R. Kumar, “A work on positive and negative effects of
social media on society,” International Journal of Computer Sciences
and Engineering, vol. 5, no. 10, pp. 351–354, Oct. 2017, doi:
10.26438/ijcse/v5i10.351354.
[3] D. Sharma, V. Gupta and V. K. Singh, “Detection of Homophobia &
Transphobia in Malayalam and Tamil: Exploring Deep Learning
Methods,” in Communications in computer and information science,
2023, pp. 217–226. doi: 10.1007/978-3-031-28183-9_15.
[4] H. Herrestad and S. Biong, “Relational hopes: A work of the lived
experience of hope in some patients hospitalized for intentional self-
harm,” International Journal of Qualitative Studies on Health and
Well-Being, vol. 5, no. 1, p. 4651, Jan. 2010, doi:
10.3402/qhw.v5i1.4651.
[5] E. Guest, B. Vidgen, A. Mittos, N. Sastry, G. Tyson and H. Margetts,
“An Expert Annotated Dataset for the Detection of Online
Misogyny,” A, pp. 1336–1350, Jan. 2021, doi:
10.18653/v1/[Link]-main.114.
[6] M. Paciello et al, “Online sexist meme and its effects on moral and
Fig. 3. Performance evaluation graph of SSL models emotional processes in social media,” Computers in Human Behavior,
vol. 116, p. 106655, Dec. 2020, doi: 10.1016/[Link].2020.106655.
[7] A. Singh et al., “Artificial intelligence for Sustainable Development
Goals: Bibliometric patterns and concept evolution trajectories,”
 Sustainable Development, vol. 32, no. 1, pp. 724–754, Jul. 2023, doi:
10.1002/sd.2706.
[8] A. Singh et al., “MIMIC: Misogyny identification in multimodal
internet content in Hindi-English Code-Mixed Language,” ACM
Transactions on Asian and Low-Resource Language Information
Processing, Apr. 2024, doi: 10.1145/3656169.
[9] M. De Choudhury, S. S. Sharma, T. Logar, W. Eekhout and R. C.
Nielsen, “Gender and Cross-Cultural Differences in Social Media
Disclosures of Mental Illness,” Proceedings of the 2017 ACM
conference on computer supported cooperative work and social
computing, pp. 353–369, Feb. 2017, doi: 10.1145/2998181.2998220.
[10] D. Sharma et al., “Detection of Homophobia & Transphobia in
Malayalam and Tamil: Exploring Deep Learning Methods,”
Communications in computer and information science, pp. 217–226,
2023. doi: 10.1007/978-3-031-28183-9_15.
[11] Singh A, Sharma D, Singh VK. Misogynistic attitude detection in
YouTube comments and replies: A high-quality dataset and
algorithmic models. Computer Speech & Language. 2024; 89:
101682. ttps://[Link]/10.1016/[Link].2024.101682
[12] N. Jindal, P. K. Kumaresan, R. Ponnusamy, S. Thavareesan, S.
Rajiakodi, and B. R. Chakravarthi, “MISTRA: Misogyny Detection
through Text-Image Fusion and Representation Analysis,” Natural
Language Processing Journal, p. 100073, Apr. 2024, doi:
10.1016/[Link].2024.100073.
[13] J. Hua, X. Cui, X. Li, K. Tang, and P. Zhu, “Multimodal fake news
detection through data augmentation-based contrastive learning,”
Applied Soft Computing, vol. 136, p. 110125, Feb. 2023, doi:
10.1016/[Link].2023.110125.
[14] J. Zhang and Y. Wang, “SRCB at SemEval-2022 Task 5: Pretraining
Based Image to Text Late Sequential Fusion System for Multimodal
Misogynous Meme Identification,” Proceedings of the 16th
International Workshop on Semantic Evaluation (SemEval-2022),
Jan. 2022, doi: 10.18653/v1/[Link]-1.81.
[15] B. Grasso, V. La Gatta, V. Moscato, and G. Sperlì, “KERMIT:
Knowledge-EmpoweRed model in harmful meme deTection,”
Information Fusion, vol. 106, p. 102269, Jan. 2024, doi:
10.1016/[Link].2024.102269.
[16] Cao, R., Lee, R.K.W., Chong, W.H., Jiang, J., 2023. Prompting for
multimodal hateful meme classification. arXiv preprint
arXiv:2302.04156.
[17] [Link]
Automatic-Misogyny-Identification-MAMI-/
[18] [Link]