GURU GOBIND SINGH COLLEGE OF ENGINEERING
& RESEARCH CENTRE, NASHIK
MINI PROJECT REPORT
Academic year: 2024-25
TITLE OF PROJECT
“Fine Tuning a Pre-Trained Transformer
(BERT)”
Bachelor of Engineering (Computer Engineering)
Course: Laboratory Practice-VI (NLP)
Course code: 410255
By
Name(s):
[Link]
Under the Guidance of
Mrs. Akshata Dighe
1
Guru Gobind Singh College of Engineering & Research Centre,
Nashik
Mini-Project Report
Name of Programme: Computer Engineering Academic Year: 2024-25
Semester: BECO-Sem 2 Course code: 410255
Name of Course: Laboratory Practice-VI (NLP)
-------------------------------------------------------------------------------------------------------------------------------
Title of Mini-Project: “Fine Tuning a Pre-Trained Transformer (BERT)”
1.0 Rationale:
A mini-project on fine-tuning a pre-trained transformer offers a relevant and impactful
learning experience by providing hands-on practice with state-of-the-art deep learning models,
fostering practical skills in NLP or other domains, and enabling students to leverage transfer learning
for efficient model adaptation; this project's manageable complexity, availability of resources, clear
evaluation metrics, and adaptability to various skill levels make it a feasible and valuable educational
endeavours, culminating in a deeper understanding of transformer architectures and improved
proficiency in deep learning frameworks.
2.0 Aim /Benefits of Mini-Project:
This mini-project offers a practical application of state-of-the-art deep learning techniques,
thereby enhancing the understanding of transformer architectures and transfer learning. It fosters
proficiency in relevant programming frameworks and libraries (e.g., TensorFlow, PyTorch, and
Hugging Face Transformers), as well as domain-specific skills (e.g., NLP). Students gain improved
ability to adapt pre-trained models for specific tasks, hands-on experience in hyperparameter tuning
and model evaluation, and critical thinking and problem-solving skills through experimentation and
analysis, ultimately leading to a more comprehensive understanding of deep learning's practical
applications.
3.0 Course Outcomes Achieved (COs):
a) Use tools and techniques in the area of software development to build mini projects (CO2)
b) Generate and manage deployment, administration & security (CO4)
4.0 Literature Review: -
This project's literature review covers the foundational transformer architecture, the evolution
of pre-trained language models (PLMs) and their pre-training objectives, various fine-tuning
techniques, including traditional methods, parameter-efficient fine-tuning (PEFT) like LoRA and
adapter layers, and prompt tuning, alongside knowledge distillation and transfer learning strategies.
It explores the diverse applications of fine-tuned transformers in NLP tasks and beyond, highlighting
state-of-the-art results and relevant datasets while emphasizing appropriate evaluation metrics and
the use of tools like Hugging Face Transformers, TensorFlow, and PyTorch, providing a
comprehensive overview of the field and establishing the project's context within current research.
2
5.0 Actual Methodology followed:
The methodology for this mini-project involves selecting a relevant pre-trained transformer
model from the Hugging Face Transformers library, followed by identifying and preparing a suitable
dataset for the chosen task, such as text classification or question answering. This includes data
preprocessing steps like tokenization, padding, and formatting the data into the model's expected
input format. The core of the project is fine-tuning the selected transformer model using a deep
learning framework like TensorFlow or PyTorch, experimenting with various hyperparameters,
including learning rates, batch sizes, and epochs, and applying parameter-efficient techniques like
LoRA if necessary. The project will then evaluate the model's performance using appropriate
metrics, such as accuracy, F1-score, or BLEU, and analyse the impact of different fine-tuning
strategies. Finally, the project will document the findings and potentially explore deployment
options, highlighting the model's performance and the insights gained from the fine-tuning process.
1. Algorithm:
1. Start
2. Choose a suitable pre-trained transformer model (BERT) from the Hugging Face
Transformers library based on the target task.
3. Identify and acquire a relevant dataset for the chosen task.
4. Preprocess the dataset:
5. Fine-tuning Configuration
6. Fine-tune the Model
7. Evaluate the Model
8. Refine and Tune the Hyperparameter
9. Stop
6.0 Actual Code of Program
CODE
import torch
from transformers import BertTokenizer, EncoderDecoderModel, Trainer, TrainingArguments
from [Link] import Dataset
import datasets
from sklearn.model_selection import train_test_split
def load_builtin_dataset(num_samples=5000, min_length=200, max_length=1000):
dataset = datasets.load_dataset("cnn_dailymail", "3.0.0")
train_data = dataset["train"].shuffle(seed=42)
filtered_data = train_data.filter(lambda example: min_length <= len(example['article'].split()) <=
max_length)
# Select up to num_samples after filtering
3
selected_data = filtered_data.select(range(min(num_samples, len(filtered_data))))
texts = [item["article"] for item in selected_data]
summaries = [item["highlights"] for item in selected_data]
return train_test_split(texts, summaries, test_size=0.2, random_state=42)
class SummarizationDataset(Dataset):
def __init__(self, texts, summaries, tokenizer):
[Link] = texts
[Link] = summaries
[Link] = tokenizer
def __len__(self):
return len([Link])
def __getitem__(self, idx):
encoding = [Link]([Link][idx], return_tensors="pt", truncation=True, padding="max_length",
max_length=512)
target_encoding = [Link]([Link][idx], return_tensors="pt", truncation=True,
padding="max_length", max_length=128)
item = {key: [Link](0) for key, val in [Link]()}
item['labels'] = target_encoding['input_ids'].squeeze(0)
return item
def load_model():
model_name = "patrickvonplaten/bert2bert_cnn_daily_mail"
tokenizer = BertTokenizer.from_pretrained(model_name)
model = EncoderDecoderModel.from_pretrained(model_name)
return tokenizer, model
def train_model(train_texts, train_summaries, tokenizer, model, eval_texts=None, eval_summaries=None):
train_dataset = SummarizationDataset(train_texts, train_summaries, tokenizer)
eval_dataset = SummarizationDataset(eval_texts, eval_summaries, tokenizer) if eval_texts and
eval_summaries else None
training_args = TrainingArguments(
output_dir="./results",
num_train_epochs=5,
4
per_device_train_batch_size=8,
fp16=True,
save_steps=500,
save_total_limit=2,
evaluation_strategy="epoch",
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
)
[Link]()
return model
def summarize_text(text, tokenizer, model):
inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True, max_length=512)
# Move inputs to the same device as the model
inputs = {k: [Link]([Link]) for k, v in [Link]()}
with torch.no_grad():
summary_ids = [Link](**inputs, max_length=128, num_beams=4, early_stopping=True)
summary = [Link](summary_ids[0], skip_special_tokens=True)
return summary
if __name__ == "__main__":
tokenizer, model = load_model()
# Load dataset from Hugging Face
train_texts, test_texts, train_summaries, test_summaries = load_builtin_dataset(num_samples=5000,
min_length=200, max_length=1000)
print("Training model on built-in dataset...")
model = train_model(train_texts, train_summaries, tokenizer, model, eval_texts=test_texts,
eval_summaries=test_summaries)
test_text = "BERT is a powerful NLP model developed by Google that has achieved state-of-the-art
performance on many tasks."
5
summary = summarize_text(test_text, tokenizer, model)
print(f"Generated Summary: {summary}")
test_text = "Researchers at MIT have developed an advanced AI model, CogniNet, capable of reasoning and
decision-making similar to human cognition. This breakthrough could revolutionize artificial intelligence
applications across industries. Unlike traditional AI models that rely on statistical probability to generate
responses, CogniNet integrates a structured reasoning framework inspired by human thought patterns. “This
model can analyze context, draw conclusions, and make logical predictions far more effectively than existing
systems,” said Dr. Elaine Brooks, lead researcher at MIT. Trained on a combination of textual datasets and
cognitive neuroscience insights, CogniNet has shown superior performance in complex tasks like legal
analysis, medical diagnosis, and financial forecasting. Early tests indicate that it outperforms existing models
in comprehension, problem-solving, and contextual understanding. Tech giants, including Google and
Microsoft, have expressed interest in the new AI system, foreseeing its potential in automation, research,
and customer service. Experts believe that CogniNet could pave the way for AI systems that think more like
humans, reducing errors and increasing efficiency across sectors. The researchers plan to release a public
beta version later this year, allowing developers worldwide to experiment with the groundbreaking
technology."
summary = summarize_text(test_text, tokenizer, model)
print(f"Generated Summary: {summary}")
OUTPUT
6
7
8
7.0. Actual Resources Used:
S. No. Name of Specifications Qty Remarks
Resource/Material
1 Computer System Windows OS, i3 processor, 2GB RAM 01 -
2 Software Google Collab, MS Office Word, 01 -
PowerBI
3 Printer, Pages Canon LaserJet 01 -
8.0. Skill Developed / Learning outcome from this Mini-Project:
1. Understanding the concept of Transformer
2. How to fine tune transformer
9.0. Applications of the Mini Project:
1. To understand the working and fine tuning the transformer BERT
Evaluated by: Mrs. Akshata Dighe
Date: Name & Signature of Guide