NATIONAL INSTITUTE OF TECHNOLOGY

PATNA

An Internship Report on
Detection of Machine Generated Text using Machine
Learning Techniques

Submitted by

Sudhir Nishad
202211087
Computer Science and Engineering
Indian Institute of Information Technology Vadodara International Campus
Diu

Under the supervision of

Dr. Akshay Deepak

Associate Professor, Department of Computer Science & Engineering
NIT Patna

nit_patna_logo.png

Department of Computer Science & Engineering

National Institute of Technology Patna
Patna-800005

15th May 2025 – 15th July 2025

Declaration

I hereby declare that the internship report titled “Detection of Machine Generated
Text using Machine Learning Techniques” is a genuine record of the work car-
ried out by me, Sudhir Nishad (Enroll. Number: 202211087), a student of [Link].
in Computer Science and Engineering, from Indian Institute of Information Technology
Vadodara International Campus Diu, during the academic tenure 2022–2026.

This internship was undertaken as part of the Major Project coursework of 6th
semester. I further declare that this report is an original contribution and has not been
submitted elsewhere for the award of any other degree or diploma.

Sudhir Nishad
202211087
Computer Science and Engineering
IIIT Vadodara International Campus Diu

1
Certificate

This is to certify that Sudhir Nishad, Enroll. Number: 202211087, pursuing [Link].
in Computer Science and Engineering from the Indian Institute of Information Technol-
ogy Vadodara International Campus Diu, Batch 2022-2026, has successfully completed
an internship under my supervision on the topic “Detection of Machine Generated
Text using Machine Learning Techniques”.

The duration of the internship was from 15th May 2025 to 15th July 2025. Out of
this, he was physically present at NIT Patna from 15th May 2025 to 25th June 2025.
The remaining internship was completed in online mode.

Dr. Akshay Deepak

Associate Professor, CSE Dept.
NIT Patna

2
Acknowledgment

I would like to acknowledge and express my deepest gratitude to my mentor Dr. Akshay
Deepak, Associate Professor, Department of Computer Science & Engineering, National
Institute of Technology Patna for the valuable guidance, encouragement, and cooperation
throughout the project.

I also wish to express my sincere gratitude to Sangeeta Mam, Ph.D. scholar, and
fellow researchers of the CSE Department for their support. I acknowledge the technical
staff of the department whose assistance was valuable.

Sudhir Nishad
Enroll. No.: 202211087
IIIT Vadodara International Campus Diu

3
Contents

Abstract 5

1 Introduction 6

2 Literature Survey 7

3 Problem Definition 8

4 Solution Strategy 9

5 Design 11

6 Dataset Description 12

7 Methodology and Implementation 13

8 Performance Analysis 16

9 Conclusion 19

References 20

4
Abstract

In this work, we explore and compare machine learning and deep learning techniques for
the binary classification task of detecting whether a given text is written by a human
or generated by an AI system. We use a large-scale dataset sourced from Hugging Face,
comprising over 1.3 million samples labeled as either human-written or AI-generated. For
the machine learning approach, we employ a TF-IDF vectorizer coupled with a Random
Forest classifier, while the deep learning pipeline utilizes a Bidirectional Long Short-
Term Memory (BiLSTM) network. The experimental results demonstrate that while
the traditional machine learning model performs well, the BiLSTM model significantly
outperforms it with an accuracy of over 95%.

5
Chapter 1 Introduction

With the rapid advancement of large language models such as GPT, BERT, and LLaMA,
AI-generated text has become increasingly fluent and difficult to distinguish from human-
written content. While these models have numerous beneficial applications, they also raise
concerns around plagiarism, misinformation, and content authenticity.
This research aims to address the challenge of identifying whether a given text is
generated by an AI or written by a human. We propose and compare two approaches:
a traditional machine learning model using TF-IDF and Random Forest, and a deep
learning model based on a Bidirectional LSTM architecture.
Both models are trained on a large dataset containing over 1.3 million labeled text
samples sourced from Hugging Face. Our results show that while the machine learning
model performs reasonably well, the deep learning model significantly outperforms it
in terms of classification accuracy, demonstrating the effectiveness of sequential neural
networks for this task.

6
Chapter 2 Literature Survey

The detection of AI-generated text has become an active area of research, driven by the
widespread use of large language models (LLMs) such as OpenAI’s GPT series, Google’s
BERT, and Meta’s LLaMA. Several works have proposed traditional and deep learning-
based solutions to distinguish human-authored content from AI-generated outputs.
In early approaches, machine learning algorithms such as Logistic Regression, Sup-
port Vector Machines (SVM), and Random Forests were used with feature engineering
techniques like TF-IDF and n-grams to classify text. These methods showed reasonable
performance on small and moderately sized datasets but lacked the ability to capture
contextual and semantic relationships within the text.
With the advent of deep learning, models such as Recurrent Neural Networks (RNNs),
Long Short-Term Memory (LSTM) networks, and Transformers have been employed to
learn temporal dependencies and semantic patterns. Bidirectional LSTM models, in
particular, have shown high performance in sentiment analysis, text classification, and
fake news detection.
Recent tools like OpenAI’s GPT Detector and projects such as GLTR (Giant Lan-
guage Model Test Room) aim to identify AI-generated content using statistical irregu-
larities or sampling probabilities. However, these tools often struggle with newer, more
sophisticated models or out-of-domain data.
In our work, we compare both traditional and deep learning techniques on a large-
scale dataset. We demonstrate that a Bidirectional LSTM network trained on tokenized
sequences provides better generalization and higher accuracy than traditional TF-IDF-
based models, offering a scalable solution to this growing challenge.

7
Chapter 3 Problem Definition

With the rapid evolution of AI-generated content, especially through advanced language
models such as GPT and LLaMA, distinguishing between human-written and machine-
generated text has become increasingly challenging. This indistinguishability raises con-
cerns related to misinformation, academic dishonesty, fake news, and automated spam
generation.
The core problem addressed in this study is:

To develop an effective classification system that can accurately

detect whether a given text is written by a human or generated by
an AI model.

To solve this, we explore two approaches:

• A traditional machine learning pipeline using TF-IDF vectorization and a Random

Forest classifier.

• A deep learning approach utilizing a Bidirectional LSTM neural network trained

on tokenized and padded text sequences.

The goal is to evaluate and compare both methods in terms of accuracy, generalization,
and practical deployment potential for real-world use cases.

8
Chapter 4 Solution Strategy

To solve the problem of distinguishing between human-written and AI-generated text,

we employ two primary approaches: a traditional machine learning-based pipeline and a
deep learning-based pipeline. Both models are trained on the same dataset and evaluated
using consistent metrics, allowing for a direct performance comparison.

Machine Learning Approach

The machine learning pipeline is designed to convert raw text into numerical feature
vectors using a bag-of-words strategy, followed by classification using an ensemble method.

• Vectorization: The text data is transformed into numerical form using the Term
Frequency–Inverse Document Frequency (TF-IDF) method. This technique helps to
down-weight commonly occurring words and emphasizes more informative terms.
We limit the vocabulary size to the top 10,000 most frequent words and remove
English stop words to eliminate noise.

• Label Encoding: The ‘source‘ field, which contains the text origin (”human” or
”ai”), is encoded using One Hot Encoding and then transformed into binary labels:
0 for human-written and 1 for AI-generated.

• Model Selection: A Random Forest Classifier is selected due to its robustness,

interpretability, and ability to handle high-dimensional sparse data efficiently. We
use 100 estimators (trees) in the forest, and a fixed random state is set for repro-
ducibility.

• Training: The dataset is split into an 80% training set and a 20% test set using
stratified sampling to preserve label distribution. The model is trained on the
TF-IDF vectors and binary labels.

• Evaluation: The trained model is evaluated using accuracy, precision, recall, and
F1-score to capture both overall and class-wise performance. A confusion matrix is
also generated to assess misclassification rates.

9
 10

• Model Persistence: The trained Random Forest model is saved using joblib for
reuse during deployment or future inference.

Deep Learning Approach

The deep learning pipeline leverages sequential modeling capabilities of Recurrent Neural
Networks to capture contextual dependencies within the text.

• Text Tokenization: Raw text is tokenized using Keras’ Tokenizer, which assigns
a unique integer to each word. We retain only the top 10,000 most frequent words
to manage vocabulary size and represent out-of-vocabulary terms using a special
token (<OOV>).

• Sequence Padding: All tokenized sequences are padded or truncated to a uniform

length of 200 tokens using Keras’ pad sequences() function. This ensures that
input shapes remain consistent during training and evaluation.

• Model Architecture: The deep learning model is built using Keras’ Sequential
API with the following layers:

– Embedding layer to learn dense vector representations of words.

– Bidirectional LSTM layer to capture long-range dependencies in both for-
ward and backward directions.
– Dense hidden layer with ReLU activation.
– Output layer with sigmoid activation for binary classification.

• Compilation: The model is compiled with binary cross-entropy as the loss function
and the Adam optimizer. Accuracy is used as the primary evaluation metric during
training.

• Training & Validation: The model is trained on 80% of the dataset and vali-
dated on the remaining 20% using mini-batch gradient descent. Early stopping is
employed to prevent overfitting.

• Evaluation: After training, the model’s performance is measured using accuracy,

precision, recall, and F1-score. Predictions are compared against true labels to
evaluate robustness and reliability.

• Model Saving: The trained model is saved as deep model.h5. For subsequent
runs, the model is reloaded to save computation time.
Chapter 5 Design

The system is designed to classify text as either human-written or AI-generated by im-

plementing two independent pipelines: one using traditional machine learning, and the
other using deep learning. The design consists of several key components shared by both
approaches, with architecture-specific variations in modeling and preprocessing.

1. Overall Architecture
The system follows these main stages:

1. Data Loading: Load .parquet files from a specified directory.

2. Data Preprocessing:

• Drop missing entries.

• Encode labels (human/ai).
• Process text into numerical form using TF-IDF or tokenization.

3. Model Building:

• ML: TF-IDF + Random Forest Classifier.

• DL: Embedding + Bidirectional LSTM + Dense Layers.

4. Training:

• Split dataset into 80% training and 20% testing.

• Fit model on training data.

5. Evaluation:

• Generate accuracy and classification report on test data.

6. Model Saving:

• ML models can be saved using joblib.

• DL model is saved as deep model.h5 and reused if already available.

11
Chapter 6 Dataset Description

The dataset used in this study was sourced from Hugging Face, consisting of over 1.3 mil-
lion text samples labeled as either human-written or AI-generated. Each sample contains
two main fields:

• text: The content of the text (ranging from short phrases to full paragraphs).

• source: The origin label — either “human” or “ai”.

Key Characteristics
• Total Records: Approximately 1,392,522 entries.

• Class Labels:

– “human” → Class 0
– “ai” → Class 1

• Format: .parquet files

• Language: English Dataset Structure Example:

Text Source
“. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ” human
“. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ” ai

Table 6.1: Example structure of dataset entries

12
Chapter 7 Methodology and Imple-
mentation

This research employs two different strategies for classifying text origin: a traditional
Machine Learning (ML) pipeline and a Deep Learning (DL) pipeline. The objective is
to explore, implement, and compare their performance in detecting whether a given text
sample is generated by a human or an AI system.

1. Machine Learning Methodology

1.1 Preprocessing
• The input dataset was read from multiple .parquet files and merged into a single
dataframe.

• Rows with missing values were dropped to maintain data quality.

• The source column, indicating whether the text is “human” or “ai”, was encoded
using One Hot Encoding via [Link]. The one-
hot labels were then converted into single class indices:

– “human” → 0
– “ai” → 1

• The text column was vectorized using TF-IDF (Term Frequency–Inverse Document
Frequency), a popular method to convert raw text into numerical feature vectors:

– max features = 10000 was used to keep only the top 10,000 frequent terms.
– stop words = ’english’ removed commonly used but uninformative words.

13
 14

1.2 Model Construction

• A Random Forest Classifier from [Link] was chosen for its ability to
handle high-dimensional data and its robustness to overfitting.

• Parameters:

– n estimators = 100: Uses 100 decision trees.

– random state = 42: Ensures reproducibility.

• The dataset was split into 80% training and 20% testing using train test split.

1.3 Model Training and Evaluation

• The model was trained on the TF-IDF features and the corresponding encoded
labels.

• Evaluation metrics included:

– Accuracy Score: Overall percentage of correct predictions.

– Classification Report: Precision, Recall, and F1-Score per class.

2. Deep Learning Methodology

2.1 Preprocessing
• The same dataset was reused and labels were again encoded using One Hot Encoding
and reduced to binary format (0/1).

• A Keras Tokenizer ([Link]) was used

to convert each text sample into a sequence of integers representing word indices:

– num words = 10000 kept the top 10k most frequent words.
– oov token = ’<OOV>’ handled out-of-vocabulary tokens.

• The resulting sequences were padded to ensure uniform length:

– max len = 200 was used with pad sequences() to truncate or pad all se-
quences to 200 tokens.
 15

2.2 Model Architecture

The deep learning model was built using the Keras Sequential API with the following
layers:

model = Sequential([
Embedding(input_dim=10000, output_dim=128, input_length=200),
Bidirectional(LSTM(64)),
Dense(64, activation=’relu’),
Dense(1, activation=’sigmoid’) # Output for binary classification
])

2.3 Model Compilation and Training

• Loss Function: binary crossentropy (suitable for binary classification)

• Optimizer: adam (adaptive learning)

• Metrics: accuracy

• The model was trained on the 80% training set and validated on the 20% test set
using the .fit() method.

2.4 Model Saving and Reuse

• The trained model was saved using:

[Link]("models/deep_model.h5")

• On future runs, the model is reloaded if already present to save time and compu-
tation.

2.5 Evaluation
• The model was evaluated using:

– Test accuracy via [Link]()

– Classification report via [Link] report() af-
ter predicting labels using [Link]()
Chapter 8 Performance Analysis

In this section, we compare the performance of the traditional Machine Learning model
and the Deep Learning model on the AI-human text classification task. The comparison
is based on various evaluation metrics including accuracy, precision, recall, and F1-score.

Metrics Overview
• Machine Learning Technique: Random Forest with TF-IDF features.

accuracy of model_final.PNG

• Deep Learning Technique: Bidirectional LSTM with embedding and padded

sequences.

16
 17

accuracy_deep_learning2_final.PNG

1. Accuracy Comparison

Approach Accuracy
Machine Learning 87.09%
Deep Learning 96.38%

Table 8.1: Accuracy comparison between Machine Learning and Deep Learning models

2. Class-wise Metrics

Observations
• The deep learning model achieves higher precision, recall, and F1-scores for both
classes, indicating more accurate and balanced performance.
 18

Metric Class Machine Learning Deep Learning

Human 0.80 0.94

Precision
AI 0.89 0.97

Human 0.67 0.93

Recall
AI 0.94 0.98

Human 0.73 0.93

F1-Score
AI 0.91 0.98

Table 8.2: Class-wise performance metrics for Machine Learning and Deep Learning
models

• Its high recall (0.93 for human, 0.98 for AI) shows strong capability in minimizing
false negatives, which is crucial for reliable classification.

• The machine learning model performs well overall but struggles with identifying
human-written text (recall of 0.67), leading to higher misclassification rates.

• These results suggest that the BiLSTM-based deep learning approach is better
suited for capturing contextual patterns and offers more robust detection in prac-
tical scenarios.
Chapter 9 Conclusion

In this research, we presented a comparative study on the detection of AI-generated and

human-written text using two distinct approaches: Machine Learning (ML) and Deep
Learning (DL). Both techniques were applied to a large, labeled dataset sourced from
Hugging Face, comprising over 1.3 million text samples.
The machine learning pipeline utilized TF-IDF vectorization and a Random Forest
Classifier, achieving a respectable accuracy of 87.09%. However, it demonstrated notice-
able imbalance in class-wise recall—particularly underperforming in identifying human-
written content.
On the other hand, the deep learning pipeline employed tokenization, embedding, and
a Bidirectional LSTM neural network, which significantly outperformed the ML model
with an accuracy of 96.38%. It also achieved higher precision, recall, and F1-scores across
both classes, indicating more robust and reliable performance.
These findings highlight the importance of sequence-aware architectures in modeling
natural language and underscore the value of deep learning techniques for distinguishing
between AI-generated and human-authored text, especially in contexts that demand high
accuracy and fairness.

19
References

[R1] Vaswani, A., et al. (2017). Attention is All You Need. NeurIPS. [Link]
org/abs/1706.03762

[R2] Hugging Face Datasets. (n.d.). AI vs Human Text Dataset. [Link]

co/datasets

[R3] Breiman, L. (2001). Random Forests. Machine Learning, 45(1), 5–32. https:
//[Link]/10.1023/A:1010933404324

[R4] Chollet, F. (2015). Keras: Deep Learning for Python. GitHub Repository. https:
//[Link]/keras-team/keras

20