A Comprehensive Framework for Semantic

Similarity Analysis of Human and AI-Generated

Text Using Transformer Architectures and
Ensemble Techniques
Lifu Gao Ziwei Liu Qi Zhang
Cornell University University of Illinois at Urbana-Champaign University of Chinese Academy of Sciences
Washington, USA Urbana, IL Beijing, China
cliffe0616@[Link] ziweil2@[Link] zhangqilike@[Link]
arXiv:2501.14288v2 [[Link]] 31 Jan 2025

Abstract—The rapid advancement of large language models capturing subtle semantic differences, as DeBERTa’s ability
(LLMs) has made detecting AI-generated text an increasingly to separately model content and position information allows
critical challenge. Traditional methods often fail to capture the for more precise analysis of semantic relationships. The model
nuanced semantic differences between human and machine-
generated content. We therefore propose a novel approach based is further enhanced with two layers of Bi-directional LSTM
on semantic similarity analysis, leveraging a multi-layered archi- to capture sequential dependencies and long-range semantic
tecture that combines a pre-trained DeBERTa-v3-large model, Bi- patterns, which are crucial for identifying the characteristic
directional LSTMs, and linear attention pooling to capture both evolution of ideas in human writing. A linear attention pooling
local and global semantic patterns. To enhance performance, we mechanism is then employed to focus on the most relevant
employ advanced input and output augmentation techniques such
as sector-level context integration and wide output configurations. semantic features, reducing noise and improving the model’s
These techniques enable the model to learn more discriminative ability to distinguish between human and AI-generated pat-
features and generalize across diverse domains. Experimental terns. The final output is produced through a fully connected
results show that this approach works better than traditional layer, ensuring robust classification.
methods, proving its usefulness for AI-generated text detection To further improve performance, we employed several ad-
and other text comparison tasks.
Kerwords—AI-generated text detection, semantic analysis, vanced input augmentation techniques such as Electra models
multi-layer neural network, data augmentation, pre-trained lan- pre-trained with Replaced Token Detection (RTD) objectives,
guage model sector-level context concatenation, adversarial weight pertur-
bation, and dynamic target shuffling to improve the model’s
I. I NTRODUCTION robustness. We also applied a wide output configuration to
The rise of AI-generated content, driven by language models allow our model to capture both local and global semantic
like ChatGPT, has created problems for content moderation patterns effectively, making it particularly adept at identifying
and text classification. Detecting machine-generated text is the subtle differences between human and AI-generated text.
important for many use cases, including combating misin- These enhancements help the model generalize better, mak-
formation and verifying academic work. Traditional detection ing it more accurate in distinguishing AI-generated text from
methods, which often rely on surface-level features such as those of human origin, and it is proven capable of achieving
syntax and word frequency, struggle to capture the nuanced state-of-the-art performances in our metrics of choice.
differences in how humans and machines construct meaning.
This study instead proposes a novel approach based on se- II. R ELATED W ORK
mantic similarity analysis, which focuses on the underlying Detecting AI-generated text has become an important re-
patterns of semantic relationships rather than surface-level search area due to the growth of large language models
features. Our hypothesis is that while human and AI-generated (LLMs) and generative adversarial networks (GANs). Yan et
text can convey similar meanings, they differ fundamentally al. [1] discuss generative LLMs, focusing on the challenges
in how semantic relationships are structured and maintained. they present to distinguish AI-generated text from human-
These differences, though subtle, can be detected through written text. These challenges are also present with GANs,
careful analysis of semantic patterns. which Gui et al. [2] discuss in terms of their applications for
Our approach leverages a pre-trained DeBERTa-v3-large content generation. GANs are particularly used for creating
model as the foundation, which provides robust semantic fake text, a problem addressed by Zellers et al. [3] in their
understanding through its disentangled attention mechanism study on defending against fake news generated by neural
and enhanced mask decoder. This is particularly effective for networks.
 To improve the detection of AI-generated text, Chakraborty B. Bidirectional LSTM Enhancement
et al. [4] review different methods for identifying machine- To capture sequential dependencies and enrich feature rep-
generated content, looking at how various AI models perform. resentation, a Bi-LSTM layer is appended to the transformer
A major challenge is the domain specificity of text, as models outputs:
trained on general datasets may not work well in specialized Xlstm = Bi-LSTM(Xbert ), (2)
areas. For example, SciBERT, a model for scientific text, has
been successful in detecting AI-generated academic papers by where Xlstm combines forward and backward dependencies.
recognizing domain-specific language patterns [5]. Dehaerne Adversarial Weight Perturbation (AWP) is introduced during
et al. [6] also explore machine learning to detect machine- the second epoch to enhance robustness by simulating ad-
generated code, highlighting the difficulties in identifying such versarial scenarios, ensuring that the Bi-LSTM learns more
content. generalizable features. The pipline of LSTM Enhancement is
Some studies focus on user interactions with AI-generated shown in Fig 2.
content. Lu [7] suggests the use of decision trees and TF-
IDF to improve the satisfaction of chatbot users, which can
also help detect AI-generated dialogue. Li [8] examines how
multimodal data can improve product recommendations, a
method that could also be used to detect AI-generated content
by combining different data sources.
Text summarization models, like those used by Liu and
Lapata [9], have shown promise in detecting AI-generated text
by analyzing the structure and coherence of the content. Schick
and Schütze [10] study few-shot learning, which could be used
to detect subtle linguistic patterns of AI-generated text.

III. M ETHODOLOGY
This section presents a comprehensive framework using
deep learning methods and ensemble techniques for semantic
similarity detection. By integrating transformer-based archi-
Fig. 2. The pipline of Bidirectional LSTM Enhancement.
tectures, bidirectional LSTM layers, and novel tricks such as
Adversarial Weight Perturbation (AWP) and linear attention
pooling, we achieve state-of-the-art performance. Additionally, C. Linear Attention Pooling
we introduce dynamic target grouping and fine-tuned ensemble For dimensionality reduction and improved focus on key
methods to boost diversity and robustness, ensuring superior features, linear attention pooling is applied:
generalization. The pipline of model is shown in Fig 1. T
X
Xpool = αt · Xlstm,t , (3)
t=1
where αt are learned attention weights, and T is the sequence
length. Dynamic target shuffling during each training step
augments this module by exposing the pooling layer to di-
verse target sequences, enhancing generalization. The linear
attention pooling is shown in Fig 3.
Fig. 1. The pipline of transformer-based architectures model. D. Fully Connected Layer
The final representation is passed through a fully connected
layer to compute the similarity score:
A. Transformer Backbone
We utilize DeBERTa-v3-large as the primary feature extrac- Ypred = FC(Xpool ). (4)
tor: Differentiated learning rates are applied, with a lower learning
Xbert = DeBERTa(Xinput ), (1) rate (2e−5 ) for the transformer and a higher rate (1e−3 ) for the
LSTM and fully connected layers. This strategy ensures effi-
where Xinput represents the tokenized input sequence. The cient optimization while preserving the pre-trained knowledge
DeBERTa backbone provides contextualized embeddings, of the transformer.
leveraging disentangled attention to capture fine-grained rela-
tionships. Additionally, freezing the embedding layers during E. Alternative Model Architectures
fine-tuning stabilizes training and reduces overfitting, as the To enhance ensemble diversity, we incorporated additional
semantic similarity task involves short text sequences. architectures, each tailored to leverage specific strengths:
 G. Data Preprocessing
Effective data preprocessing is critical to model perfor-
mance. The following steps were implemented:
1) Target Grouping and Stratification: Data was grouped
by anchor phrases and stratified based on semantic similarity
scores:
G = GroupBy(Anchor, Context)[Target], (11)
ensuring balanced data distribution across training folds. Tar-
gets sharing common words with anchors were allocated to the
same folds to maintain contextual consistency. The Semantic
Similarity Average Score Graphic and the Target Distribution
Bar graph in Fig 4 show the Average Semantic Similarity
Score of each combination of anchor and context, simulating
Fig. 3. The linear attention pooling. the results of the grouping and stratification of the target.

1) Electra-Based Models: Electra models, pre-trained with

a replaced token detection (RTD) objective, complement the
transformer backbone by capturing finer-grained semantic nu-
ances. The model is formulated as:
Xelectra = Electra(Xinput ), (5)
where the RTD mechanism provides robust token-level un-
derstanding. Expanding dimensions for weaker models like Fig. 4. Semantic similarity average score graph.
SimCSE improves compatibility during ensemble integration:
2) Dynamic Target Shuffling: During each training step,
Xwide = ExpandDims(Xelectra ). (6) target sequences were shuffled dynamically:
(i)
2) Wide Output Configurations: For models with lower Stargets = Shuffle(T (i) ), (12)
baseline performance, we expanded the output dimensions: (i)
where Stargets represents the shuffled target set at step i. This
wide reduces overfitting and exposes the model to diverse input
Xout = Concat(Xtransf ormer , Xcontext ), (7)
combinations.
where contextual information is explicitly integrated, enhanc- 3) Contextual Augmentation: Sector-level contexts were
ing representation diversity. extracted and added to the input:
3) Bi-LSTM and Sector Contexts: Bi-LSTM layers were aug
adapted to integrate grouped sector-level contexts: Xinput = Concat(Xanchor , Xtarget , Xsector ). (13)

Xsector = Bi-LSTM(Xcontext[0] ), (8) This augmentation enriches the input representation, aligning
it with hierarchical domain knowledge.
where context[0] represents sector-level information (e.g., 4) Tokenization and Padding: Inputs were tokenized using
F21 for ”F”). This hierarchical approach adds a structured a subword tokenizer and padded to a uniform sequence length:
representation for weakly supervised data.
Xtoken = Pad(Tokenizer(Xraw )). (14)
F. Loss Function Padding ensured compatibility with batch processing while
The primary loss function is the Pearson correlation loss: preserving contextual integrity.
Cov(Ypred , Ytrue ) IV. E VALUATION M ETRICS
Lpearson = − , (9)
σ(Ypred ) · σ(Ytrue ) The performance of the models was evaluated using the
where Cov represents covariance, and σ denotes standard following metrics:
deviation. Additionally, we employed a mean squared error 1) Pearson Correlation Coefficient: The primary metric is
(MSE) loss as a secondary measure: the Pearson correlation coefficient, which measures the linear
n
correlation between predicted and true scores:
1X Cov(Ypred , Ytrue )
Lmse = (Ypred,i − Ytrue,i )2 . (10) ρ= , (15)
n i=1 σ(Ypred ) · σ(Ytrue )
These loss functions, combined with AWP and dynamic where Cov represents covariance, and σ is the standard devi-
target shuffling, ensure robust optimization. ation.
 2) Mean Squared Error (MSE): To evaluate prediction TABLE I
accuracy, the mean squared error was computed: A BLATION S TUDY RESULTS

n Model Pearson (%) MSE F1-Score (%) AUC (%)

1X DeBERTa-v3-large 86.1 0.015 88.5 91.2
MSE = (Ypred,i − Ytrue,i )2 . (16) DeBERTa + LSTM 86.6 0.014 89.1 92.3
n i=1
+ Linear Attention Pooling 86.8 0.013 89.4 92.8
+ Target Shuffling 87.2 0.012 90.1 93.5
This metric captures the average squared difference between Ensemble Model 87.5 0.011 91.2 94.7
predictions and actual values.
3) F1-Score: F1-score was used to evaluate the balance
between precision and recall for binary classification tasks: VI. C ONCLUSION
This study demonstrates the effectiveness of leveraging
Precision · Recall
F1 = 2 · . (17) transformer-based architectures with Bi-LSTM enhancements,
Precision + Recall
adversarial weight perturbation, and dynamic preprocessing
This ensures a comprehensive evaluation of the model’s per- strategies for comparing the semantic similarity between hu-
formance on edge cases. man and AI-generated text. The integration of diverse models,
4) Area Under Curve (AUC): The AUC metric evaluates combined with linear attention pooling and target shuffling,
the ability of the model to distinguish between classes by cal- significantly improves robustness and accuracy. The ensemble
culating the area under the Receiver Operating Characteristic strategy achieves state-of-the-art performance across multiple
(ROC) curve: evaluation metrics, setting a robust foundation for practical ap-
Z 1 plications in patent search and examination processes. Future
work will explore domain-specific pretraining and other aug-
AUC = T P R(F P R)d(F P R), (18)
0 mentation techniques to further enhance model generalization.

where TPR is the true positive rate and FPR is the false R EFERENCES
positive rate. [1] X. Yan, Y. Xiao, and Y. Jin, “Generative large language models explained
[ai-explained],” IEEE Computational Intelligence Magazine, vol. 19,
no. 4, pp. 45–46, 2024.
V. E XPERIMENT R ESULTS [2] J. Gui, Z. Sun, Y. Wen, D. Tao, and J. Ye, “A review on generative
adversarial networks: Algorithms, theory, and applications,” IEEE trans-
Table I provides a detailed view of performance gains across actions on knowledge and data engineering, vol. 35, no. 4, pp. 3313–
the evaluation metrics in the ablation study. The losses and 3332, 2021.
[3] R. Zellers, A. Holtzman, H. Rashkin, Y. Bisk, A. Farhadi, F. Roesner,
performance indicator metrics in each training epochs of the and Y. Choi, “Defending against neural fake news,” Advances in neural
final model are shown in Figure 5. information processing systems, vol. 32, 2019.
[4] S. Chakraborty, A. S. Bedi, S. Zhu, B. An, D. Manocha, and F. Huang,
“On the possibilities of ai-generated text detection,” arXiv preprint
arXiv:2304.04736, 2023.
[5] I. Beltagy, K. Lo, and A. Cohan, “Scibert: A pretrained language model
for scientific text,” arXiv preprint arXiv:1903.10676, 2019.
[6] E. Dehaerne, B. Dey, S. Halder, S. De Gendt, and W. Meert, “Code
generation using machine learning: A systematic review,” Ieee Access,
vol. 10, pp. 82 434–82 455, 2022.
[7] J. Lu, “Enhancing chatbot user satisfaction: A machine
learning approach integrating decision tree, tf-idf, and
bertopic,” Preprints, November 2024. [Online]. Available:
[Link]
[8] S. Li, “Harnessing multimodal data and mult-recall strategies
for enhanced product recommendation in e-commerce,” Preprints,
September 2024. [Online]. Available: [Link]
preprints202409.2417.v1
[9] Y. Liu and M. Lapata, “Text summarization with pretrained encoders,”
arXiv preprint arXiv:1908.08345, 2019.
[10] T. Schick and H. Schütze, “Exploiting cloze questions for few shot
text classification and natural language inference,” arXiv preprint
arXiv:2001.07676, 2020.

Fig. 5. Performance indicator change chart