Abstract

This thesis investigates the impact of dataset descriptions, training dataset size, and LLM
architectures on the quality of Verilog code generation. With the growing adoption of large
language models (LLMs) in domain-specific tasks, this study focuses on fine-tuning state-of-the-
art models, including DeepSeek Coder 6.7B Instruct, Qwen 2.5 Coder 7B Instruct, CodeLlama-
7B-Instruct, and CodeGemma 7B Instruct, to generate Verilog code effectively. Four types of
dataset descriptions were evaluated: Verilog PoT summaries, detailed level summaries, high-
level summaries, and block-level summaries. VerlogEval Evaluation framework was used to
assess the performance of the fine-tuned models.

The results demonstrate that the Verilog PoT dataset consistently delivers the highest
performance across all evaluation scenarios, achieving among the top two rankings in all three
types of prompts: High level, Detail level and Block level prompts. The dataset’s balanced
structure—integrating all three level of descriptions—enables robust generalization. Detailed
global summaries also performed competitively, offering an effective balance of generality and
precision. In contrast, training solely with highly detailed or overly generalized datasets resulted
in suboptimal performance, highlighting the importance of dataset structure.

Among the LLM architectures, DeepSeek Coder 6.7B Instruct achieved the best overall results,
demonstrating strong convergence and adaptability. Qwen 2.5 Coder 7B Instruct showed
competitive performance but exhibited minor instability, while CodeLlama and CodeGemma
lagged in performance and convergence.

Additionally, the influence of training dataset size revealed significant improvements in pass rates
with larger datasets, particularly up to 6000 samples. However, diminishing returns were observed
beyond this threshold, suggesting that dataset diversity and quality are more critical than sheer
size. The findings emphasize the importance of balanced datasets, efficient model architectures,
and scalable training approaches in achieving high-quality Verilog code generation.

This study contributes to the field of AI-driven hardware design automation by providing actionable
insights into dataset engineering and model optimization for domain-specific tasks. Future work
can explore alternative methods such as retrieval-augmented generation and scaling larger
models to further improve performance and applicability.

i
Table of Contents
1. Introduction .......................................................................................................................... 1
1.1 Historical Background ...................................................................................................... 1
1.2 Motivation ......................................................................................................................... 1
1.3 Problem Description ......................................................................................................... 2
1.4 Overview of Chapters ....................................................................................................... 3
2. Theoretical Background ...................................................................................................... 5
2.1 Introduction ...................................................................................................................... 5
2.2 Introduction to Verilog HDL............................................................................................... 6
2.3 MG-Verilog Dataset .......................................................................................................... 7
2.3.1 Multi-Grained Dataset Structure ................................................................................. 7
2.3.2 Impact of Dataset Granularity on Code Quality .........................................................10
2.4 Large Language Models (LLMs) ......................................................................................12
2.4.1 Overview of LLMs .....................................................................................................12
2.4.2 Transformer: The Backbone of LLMs ........................................................................13
2.4.3 LLMs Used in Our Research .....................................................................................14
2.5 Fine-Tuning Techniques for LLMs....................................................................................23
2.5.1 Overview of Fine-Tuning ...........................................................................................25
2.5.2 General Steps in Fine-Tuning....................................................................................26
2.5.3 Types of Fine-Tuning Techniques .............................................................................28
2.5.4 PEFT Fine-Tuning Method ........................................................................................30
2.5.5 Low-Rank Adaptation (LoRA) ....................................................................................33
2.5.6 Quantized Low-Rank Adaptation (QLoRA) ................................................................37
2.6 Evaluation Framework for Verilog Code ..........................................................................40
2.6.1 Overview of Code Evaluation in LLMs .......................................................................40
2.6.2 VerilogEval Library ....................................................................................................40
2.7 Hyperparameter Tuning in LLM Fine-Tuning ...................................................................42
2.7.1 Importance of Hyperparameters ................................................................................42
2.7.2 Variation in Hyperparameters ....................................................................................43
2.8 Summary.........................................................................................................................45
3. Literature Review ................................................................................................................46
3.1 Introduction .....................................................................................................................46

ii
 3.2 Large Language Models for Code Generation .................................................................46
3.3 Verilog Code Generation .................................................................................................47
3.4 Effect of Different LLM Models in Code Generation .........................................................48
3.5 Fine-Tuning Techniques in Code Generation ...................................................................49
3.6 Research Gap .................................................................................................................50
4. Implementation ...................................................................................................................51
4.1 Introduction .....................................................................................................................51
4.2 Experimental Setup .........................................................................................................52
4.2.1 Dataset Preparation ..................................................................................................52
4.2.2 Model Fine-Tuning ....................................................................................................52
4.2.3 Model Evaluation.......................................................................................................53
4.2.4 Inference ...................................................................................................................53
4.3 Dataset Preparation ........................................................................................................53
4.3.1 Dataset Description ...................................................................................................54
4.3.2 Data Preprocessing Pipeline .....................................................................................55
4.3.3 Dataset Splits ............................................................................................................56
4.3.4 Challenges in Dataset Preparation ............................................................................56
4.4. Model Fine-Tuning .........................................................................................................56
4.4.1 Dataset-Specific Training Arguments ........................................................................57
4.4.2 Fine-Tuning Framework ............................................................................................59
4.4.3 Checkpointing and Model Saving ..............................................................................65
4.4.4 Logging and Monitoring .............................................................................................67
4.5 Model Evaluation .............................................................................................................68
4.5.1 Evaluation Datasets ..................................................................................................68
4.5.2 Step-by-Step Evaluation Process ..............................................................................70
4.6 Summary.........................................................................................................................74
5. Results and Discussion ......................................................................................................75
5.1 Introduction .....................................................................................................................75
5.2 Influence of Dataset Descriptions on Model Performance ...............................................76
5.2.1 Results ......................................................................................................................76
5.2.2 Analysis ....................................................................................................................78
5.3 Influence of Model Architecture on Code Generation.......................................................79
5.3.1 Evaluation Loss Analysis for Verilog PoT Dataset .....................................................79

iii
 5.3.2 Evaluation Loss for High-Level Summaries Dataset ..................................................81
5.3.3 Overall Observations .................................................................................................82
5.4 Influence of Training Samples in Fine Tuning ..................................................................83
5.4.1 Analysis of the Impact of Increasing Training Samples..............................................83
5.4.2 Diminishing Returns ...............................................................................................84
6. Conclusion ..........................................................................................................................85
7. Limitations and Future Work ..............................................................................................86
7.1 Limitations .................................................................................................................86
7.2 Future Work ....................................................................................................................86
8. References...........................................................................................................................87

iv
List of Figures
Figure 1: Different level of code descriptions [9] ......................................................................... 9
Figure 2: Transformer Architecture [8] .......................................................................................13
Figure 3: Model Architecture of CodeLlama 7B Instruct ............................................................16
Figure 4: Model Architecture of Qwen 2.5 Coder 7B Instruct .....................................................18
Figure 5: Model Architecture of DeepSeek Coder 6.7B Instruct ................................................20
Figure 6: Model Architecture of CodeGemma 7B Instruct .........................................................22
Figure 7: Techniques for improvement of LLM on Targeted Domain .........................................24
Figure 8: Fine-Tuning Process [14] ...........................................................................................25
Figure 9: General Steps in Fine-Tuning [16]..............................................................................26
Figure 10: Overview of Fine-Tuning Techniques .......................................................................28
Figure 11: PEFT vs Full Fine-tuning [18] ...................................................................................30
Figure 12: Overview of PEFT [19] .............................................................................................31
Figure 13: PEFT Techniques [20]..............................................................................................32
Figure 14: Concept of LoRA [17] ...............................................................................................33
Figure 15: Transformer Architecture [20] ...................................................................................35
Figure 16: Add LoRA to the self-attention layer [18] ..................................................................35
Figure 17: Low Rank Decomposition with r = 8 [20] ..................................................................36
Figure 18: Steps to update the model for inference [20] ............................................................36
Figure 19: Quantization and Dequantization [13].......................................................................39
Figure 20: Example of VerilogEval Evaluation dataset [10] .......................................................41
Figure 21: Overall Experimental Setup for the Research...........................................................52
Figure 22: Hyperparameters used for High-Level Global Summaries, Detailed Global Summaries,
and Block-Level Summaries datasets .......................................................................................58
Figure 23: Hyperparameters used for Verilog Pyramid-of-Thoughts (PoT) Dataset ...................59
Figure 24: Overview of Evaluation Process ...............................................................................68
Figure 25: Evaluation Loss during Finetuning of Verilog PoT Dataset .......................................79
Figure 26: Evaluation Loss during Finetuning of High-Level Summaries Dataset ......................81
Figure 27: The Impact of Training Samples during Finetuning ..................................................83

v
List of Tables
Table 1: Overview of Research Framework ............................................................................... 5
Table 2 : Levels of Abstraction in Verilog HDL ........................................................................... 7
Table 3: Example of Multi-Grained Descriptions for the Verilog Module "cordic" .......................10
Table 4: Strengths and Limitations of different summaries ........................................................11
Table 5: Differences between General-Purpose LLMs and Code-Specific LLMs .......................15
Table 6: Difference Between LoRA and QLoRA ........................................................................37
Table 7: Advantages of QLoRA.................................................................................................38
Table 8: Metrics Used in VerilogEval.........................................................................................41
Table 9: Effects of Hyperparameters Tuning .............................................................................44
Table 10: Pass@1 ....................................................................................................................76
Table 11: Pass@5 ....................................................................................................................77
Table 12: Pass@10 ..................................................................................................................78

vi
Abbreviations

AI Artificial intelligence
BERT Bidirectional Encoder Representations from Transformers
BLEU Bilingual Evaluation Understudy
DoRA Weight-Decomposed Low-Rank Adaptation
DSLs Domain-specific languages
FIM fill-in-the-middle
GPT Generative Pre-trained Transformer
HDL Hardware description language
LLM Large Language Model
LoRA Low-Rank Adaptation
LSTM Long Short-Term Memory
MG-Verilog Multi-Grained Verilog
NF4 Normalized Float 4-bit
NLP Natural Language Processing
PEFT Parameter-Efficient Fine-Tuning
PoT Pyramid of Thoughts
QLoRA Quantized LoRA
RAG Retrieval-augmented generation
RNNs Recurrent Neural Network
RTL Register-Transfer Level
SFT Supervised Fine-Tuning
SoC System-on-Chip

vii
1. Introduction

1.1 Historical Background

The incorporation of artificial intelligence (AI) into software development has advanced
significantly in recent years, especially in the area of code generation. Large Language Models
(LLMs), utilizing deep learning methods, have proven capable of producing programming code
with impressive precision and complexity. Notable examples, including OpenAI’s Codex [1],
Meta’s CodeLlama [2], and Google’s PaLM [3], have demonstrated their effectiveness in
supporting various software engineering tasks, ranging from debugging to full program synthesis.

Despite their success with general-purpose programming languages like Python, C++, and
JavaScript, their application to domain-specific languages (DSLs) remains an emerging field.
Verilog, a hardware description language (HDL) crucial for digital system design and simulation,
exemplifies such a DSL. The structured and hierarchical nature of Verilog code, coupled with its
emphasis on parallelism and hardware-level design logic, presents a unique set of challenges
that differ significantly from conventional software development tasks.

1.2 Motivation
The use of LLMs for generating Verilog code has been relatively unexplored, resulting in a limited
understanding of their capabilities in this area. Unlike traditional programming languages, Verilog
requires adherence to strict rules regarding parallelism, timing constraints, and hierarchical
module interactions, which make code generation particularly challenging.

One of the key problems lies in the datasets used for fine-tuning these models. While dataset size
and diversity are known to impact model performance [4], there is limited research on how dataset
descriptions—such as modular global summaries or fine-grained block-level annotations—affect
the quality of generated code in domain-specific languages.

Moreover, different LLM architectures, optimized for general-purpose programming tasks, may
vary in their ability to handle Verilog. While recent models like CodeLlama [2], Qwen [5],
CodeGemma [6] and DeepSeek [7] have shown proficiency in general-purpose programming,
their adaptability to domain-specific requirements, such as hardware design and Verilog
synthesis, remains unquantified. A thorough evaluation is required to determine the factors that
influence their effectiveness or shortcomings in generating Verilog code.

Failure to address these issues limits the practical application of LLMs in hardware design
automation, a domain where efficiency, correctness, and modularity are paramount. By analysing
the interplay between dataset descriptions and model architectures, this study aims to provide
actionable insights for optimizing LLMs for Verilog code generation.

1
1.3 Problem Description
The quality and efficacy of LLM-generated Verilog code are inherently tied to the training datasets
used during fine-tuning. Training data with different levels of detail—ranging from global
summaries of code behaviour to highly detailed block-level descriptions—can significantly impact
the model's ability to learn Verilog’s intricacies. However, the exact effects of dataset granularity
and type on Verilog code generation remain unclear, necessitating further investigation.

Another critical aspect is the diversity in LLM architectures. While all large models leverage
transformer-based architectures [8], subtle differences in pretraining strategies, parameter
optimizations, and dataset usage can lead to variability in their performance for domain-specific
applications. Comparing state-of-the-art models like CodeLlama 7B Instruct, Qwen 2.5 Coder 7B
Instruct, CodeGemma 7B Instruct and DeepSeek Coder 6.7B Instruct on Verilog code generation
can shed light on which architectural features and training paradigms are best suited for such
tasks.

This study aims to explore the following key question:

How do dataset descriptions, size of training dataset and LLM architectures influence the
quality of Verilog code generation?

This question is broken down into the following sub-questions:

● What are the effects of different dataset descriptions—Verilog Pyramid of Thoughts (PoT)
summaries, global-level summaries, detailed-level summaries, and block-level
summaries—on the accuracy and quality of Verilog code generation by LLMs?

● How do the performances of CodeLlama 7B Instruct, Qwen 2.5 Coder 7B Instruct,

CodeGemma 7B Instruct and DeepSeek Coder 6.7B Instruct compare in generating
Verilog code when fine-tuned with these dataset descriptions?

● What role do hyperparameters play in influencing the model's capacity to produce

accurate and efficient Verilog code?

● How does the size of training dataset affect the model's ability to generate accurate and
efficient Verilog code?

Addressing these questions will provide a deeper understanding of how to optimize LLMs for
domain-specific tasks, particularly in hardware description languages like Verilog.

The overarching aim of this thesis is to evaluate the performance of LLMs in generating Verilog
code, focusing on the effects of dataset descriptions, training dataset size and model
architectures. Specifically, the objectives of this research are:

2
 ● Analysing Dataset Descriptions:
○ Examine the influence of four dataset description types— Verilog Pyramid of Thoughts
(PoT) summaries, global-level summaries, detailed-level summaries, and block-level
summaries—on the performance of LLMs.
○ Identify the dataset description type that maximizes the quality of the generated Verilog
code.
● Evaluating Model Architectures:
○ Compare the capabilities of CodeLlama 7B Instruct, Qwen 2.5 7B Coder Instruct, and
DeepSeek Coder 6.7B Instruct in generating Verilog code.
○ Highlight the strengths and weaknesses of each model in handling the structural and
semantic requirements of Verilog.
● Developing Optimization Strategies:
○ Propose best hyperparameter practices for fine-tuning LLMs.
● Analysing Training Dataset Size:
○ Examine the influence of training dataset size on the performance of LLMs in Verilog
code generation.

By achieving these aims, this study will contribute to advancing the use of AI in hardware
description and digital system design.

1.4 Overview of Chapters

The following provides a concise overview of the thesis structure:

Chapter 2: Theoretical Background

This chapter provides a detailed discussion of the foundational concepts of tools and techniques
that have been used in this research. It reviews the evolution of LLM architectures, their
application to code generation tasks, and the technique used for finetuning LLM in hardware
description languages like Verilog.

Chapter 3: Literature Review

This chapter critically examines existing studies on LLMs and their application to code generation.
It provides insights into model architectures, and fine-tuning techniques in code generation,
emphasizing research gaps in Verilog and other domain-specific languages.

Chapter 4: Implementation
The methodology chapter outlines the experimental setup for this study. It describes the dataset
preparation process, including the creation of four dataset description types, and details the fine-
tuning of CodeLlama, Qwen, CodeGemma and DeepSeek models. It also specifies the evaluation
metrics and tools used for performance analysis.

3
Chapter 5: Results and Discussion
This chapter presents the findings of the experiments, comparing the effects of dataset
descriptions and LLM architectures on Verilog code generation. The analysis includes quantitative
metrics, such as code accuracy and correctness.

This chapter also interprets the experimental results in relation to the research questions and
objectives. It explores the implications of the findings for the use of LLMs in domain-specific
applications, providing insights into optimizing dataset descriptions and model architectures for
Verilog code generation.

Chapter 7: Conclusion
The final chapter summarizes the research contributions, highlighting the advancements made in
understanding the interplay between dataset descriptions and LLM architectures.

Chapter 8: Limitations and Future Work

This chapter discusses the limitations of the study, including constraints related to dataset size,
diversity, and model scalability. It proposes directions for future research, such as exploring
additional dataset descriptions, leveraging larger and more advanced LLMs, and extending the
findings to other hardware description languages. By outlining these possibilities, the chapter
provides a roadmap for advancing research in AI-driven hardware design.

4
2. Theoretical Background

2.1 Introduction
In the rapidly evolving field of artificial intelligence (AI), large language models (LLMs) have
become pivotal tools for automating complex tasks. Their ability to understand and generate
human-like text has extended to specialized domains, including code generation for programming
and domain-specific languages. This chapter serves as the foundational groundwork for
understanding the core concepts, methods, and tools relevant to this thesis, which focuses on
fine-tuning LLMs for Verilog code generation.

The purpose of this chapter is to bridge the gap between the general capabilities of LLMs and the
specific requirements of hardware description languages (HDLs) like Verilog. It provides an
overview of the historical evolution of LLMs, delves into their architectural nuances, and explores
the challenges posed by domain-specific code generation. The chapter also discusses critical
components such as dataset preparation, fine-tuning techniques, and evaluation frameworks that
are central to the thesis.

The insights presented in this chapter are directly aligned with the thesis objectives, which aim to
investigate the interplay between dataset descriptions and LLM architectures in the context of
Verilog code generation. By systematically exploring prior research and contextualizing it within
the scope of the current study, this chapter sets the stage for the experimental design and analysis
presented in later chapters.

Table 1 provides an illustrative overview of how the core components discussed in this chapter—
LLMs, dataset descriptions, fine-tuning, and evaluation—are interrelated and contribute to the
research framework of this thesis.

Component Description

CodeLlama, Qwen, and DeepSeek, each with unique strengths in

LLM Architectures
code generation.
MG Verilog, global-level summaries, detailed-level summaries, block-
Dataset Descriptions
level summaries.
QLoRA method with variations in hyperparameters for model
Fine-Tuning Technique
optimization.

Evaluation Framework VerilogEval for assessing the quality of generated Verilog code.

Table 1: Overview of Research Framework

5
2.2 Introduction to Verilog HDL
Verilog HDL (Hardware Description Language) is a cornerstone in digital circuit design, providing
a means to describe the structure and behaviour of electronic systems. Developed in 1984,
Verilog was initially created to support simulation before synthesis became a key focus. Over the
years, Verilog has evolved into an industry-standard language widely used for designing systems
ranging from simple digital circuits to complex processors and System-on-Chip (SoC) designs.

Verilog allows designers to define circuit functionality at multiple abstraction levels:

● Gate Level: Describes circuits using basic logic gates and interconnections.
● Register-Transfer Level (RTL): Specifies how data flows between registers and the
combinational logic that processes it.
● Behavioural Level: Focuses on system functionality without detailing structural design.

Its capabilities are integral for prototyping, simulating, and testing digital circuits. Verilog’s support
for synthesis also enables seamless transitions from abstract design to physical implementation.

For those familiar with programming, Verilog resembles traditional coding languages. Below is an
example of Verilog code that describes a simple register:

// MODULE DECLARATION
module Main(
input clk, // Clock signal
input reset, // Reset signal
input [7:0] input_data, // 8-bit input
output reg [7:0] output_data // 8-bit output
);

// ALWAYS BLOCK TO HANDLE REGISTER OPERATIONS

always @(posedge clk or posedge reset)
begin
if (reset)
output_data <= 8'b0; // Clear output on reset
else
output_data <= input_data; // Assign input to output on clock edge
end

endmodule

The above Verilog code describes a register that can store an 8-bit input and outputs the same
data. On a reset signal, the output is cleared to 0. The design mimics the behaviour of hardware
components, enabling engineers to translate functional requirements into a circuit.

6
While Verilog is robust and widely adopted, it presents notable challenges:

1. Syntax Complexity: Verilog's strict syntax leaves little room for error. Any deviation from
correct practices can result in malfunctioning or non-synthesizable designs.
2. Limited Open-Source Resources: Unlike software languages like Python or Java,
Verilog lacks a rich repository of pre-existing open-source designs. This scarcity stems
from the proprietary nature of hardware design, as companies closely guard their HDL
work for competitive advantage.
3. Scalability in Complex Designs: As circuit complexity grows, Verilog’s modular design
becomes harder to manage without automated tools.

Level Description Example

Gate Level Describes logic gates and their and gate1 (output, input1,
interconnections. input2);

Register- Defines data flow between always @(posedge clk) q <= d;

Transfer registers and combinational logic.

Behavioural Specifies functionality without if (enable) sum = a + b;

structural details.

Table 2 : Levels of Abstraction in Verilog HDL

To overcome these limitations, our research focuses on creating customized models for Verilog
code generation. By curating unique datasets and developing domain-specific models, we ensure
that the generated Verilog code meets specific requirements while addressing challenges like
syntax accuracy and design optimization.

2.3 MG-Verilog Dataset

Fine-tuning large language models (LLMs) for hardware description languages like Verilog
requires datasets that are both diverse and domain specific. The MG-Verilog dataset, as proposed
by Zhang et al., addresses key challenges in this domain by providing multi-grained descriptions
at various levels of detail [9]. These structured descriptions are paired with Verilog code, enabling
LLMs to learn both high-level and detailed aspects of hardware design.

2.3.1 Multi-Grained Dataset Structure

The MG-Verilog dataset consists of Verilog modules paired with natural language descriptions
across varying levels of abstraction. The dataset has two columns:

7
1. Code Column: Contains Verilog modules with line-by-line comments and implementations of
varying complexity, from single-module to multi-module designs.
2. Description Column: The MG-Verilog dataset is unique in its multi-grained structure,
which provides descriptions at four distinct levels of granularity. These levels are designed to
balance the trade-offs between user-friendliness and the need for detail. The granularity levels
include:

a. High-Level Global Summary: These summaries offer a general description of the

module, outlining its core functionalities and purpose. They provide an overall picture,
detailing the top-level communication and control interface without going into signal-
specific details. They make the dataset user-friendly and allow LLMs to start code
generation from general-purpose prompts. The high-level summary generally includes:

■ Module Function: An overview description of the module's core functionality. For

instance, "manages data transmission between components with buffering and clock
synchronization" could describe a communication module.
■ Key Features: Outlines the major components or sub-modules involved, like "dual
FIFOs for RX and TX," or "clock synchronization."
■ General Interface: Highlights the main inputs and outputs, such as "input clock, reset,
and data signals," but without detailing individual signal interactions.
■ Scope of Application: Briefly mention what kind of system or environment the module
is designed for, such as "for data transfer in communication systems."

b. Detailed Global Summary: The detailed level summaries provide a deeper look into the
module's functionalities and interactions between signals and sub-modules. It expands
upon the high-level summary by explaining specific design components and their roles. It
covers critical operations, control signals, and timing considerations that define how the
module works. This level of detail is beneficial for LLM for a more thorough understanding
of the module’s structure and logic flow. The detailed summary usually contains:

■ Functional Breakdown: Describes specific tasks the module performs, such as

"handling data transmission with clock management" or "filtering signals for noise
reduction."
■ Signal Roles: Mentions important control signals (e.g., reset, enable, full/empty status)
and how they govern the module’s operation.
■ Interaction with Sub-Modules: Outlines how the main module interacts with internal
sub-modules, like debouncing circuits, clock reduction modules, or token detection
systems.
■ Signal Mapping and Flow: Provides information on how the signals are routed within
the module, such as "data flow from RX to TX FIFOs, managed by control signals."

c. Block-Level Summaries: These summaries break down the module at the line or logical
block level, describing each section's purpose and function in detail. The block-level
summary is designed to offer granular, step-by-step information on individual sections of
the code. It’s useful for LLM to understand or modify specific parts of the module in detail.
This level of summary is essential for debugging, editing, or developing specific features

8
 of the module since it explains each functional block’s purpose and behaviour. This level
provides comprehensive context, which is especially useful for tasks requiring high
accuracy. The block-level summary includes:

■ Block Identification: Each block is labelled (e.g., block_0, block_1) and has a unique
role in the overall functionality.
■ Detailed Function of Each Block: Describes the purpose of each code block, such as
"initializes the reset and enable signals" or "handles data write and read logic for
FIFOs."
■ Signal-Specific Roles: Identifies the specific signals involved in each block and what
they control or influence within the module.
■ Sub-Module Instantiation and Logic Operations: Details any sub-module
instantiations, control statements (like assign or always blocks), and logical conditions
affecting the operation of that particular block code generation.

d. Line-by-Line Comments: These are the most granular descriptions, offering detailed
comments for every line of Verilog code.

Example: "// Assign rotated xO and yO to output ports"

Purpose: Line-by-line annotations are valuable for debugging and understanding the exact
purpose of each line of code but are computationally intensive for fine-tuning.

Figure 1: Different level of code descriptions [9]

Figure 1 depicts the hierarchical structure of the dataset, showing how each level builds upon the
previous one.

9
 Granularity Level Description Example

High-Level Global The 'cordic' module computes trigonometric functions for given inputs
Summary using CORDIC rotation logic.

Detailed Global The module uses input coordinates, angle values, and an internal
Summary lookup table for iterative calculations.

Block-Level Summary Block 1: Declares input/output ports. Block 8: Assigns final results to
outputs after iterative calculations.

Line-by-Line Line 1: Assigns rotated x-coordinate. Line 2: Implements quadrant

Comments selection logic.

Table 3: Example of Multi-Grained Descriptions for the Verilog Module "cordic"

The description column in the dataset incorporates a structured meta-prompting template to

enhance model understanding and performance. The template follows the format:

```<s>[INST] <<SYS>>\n{context}\n<</SYS>>\n\n{question} [/INST] </s>```

In this template:

● {context}contains a predefined instruction:

"You only complete chats with syntax-correct Verilog code. End the Verilog module code
completion with 'endmodule'. Do not include module, input, and output definitions."
● {question} is populated with one of the following summary types: high-level, detailed,
or block-level summaries.

This structured format ensures the model receives clear, consistent, and actionable prompts,
facilitating accurate Verilog code generation during training and inference. It is applied uniformly
to all 11,100 Verilog code descriptions in the dataset, creating a cohesive and comprehensive
MG-Verilog resource.

2.3.2 Impact of Dataset Granularity on Code Quality

The varying granularity levels in the MG-Verilog dataset allow LLMs to learn at different levels of
abstraction. This multi-grained structure enables the models to:

1. Generate functional Verilog code using only high-level summaries, simulating real-world
usage scenarios.
2. Create detailed and optimized designs when provided with fine-grained descriptions.

Zhang et al. propose a unique balanced fine-tuning scheme that uses a mix of high-level and low-
level descriptions during training [9]. This method ensures that models can handle a wide variety

10
of input instructions—ranging from abstract user prompts to highly detailed technical
requirements. The balanced approach improves both generalization and accuracy in generated
code.

Zhang et al. conducted extensive experiments to evaluate the effectiveness of the MG-Verilog
dataset. Key findings include:

● Models trained on a mix of high-level and block-level summaries exhibited better

performance across diverse tasks compared to those trained exclusively on one
granularity level.
● The dataset's inclusion of detailed descriptions improves the model's ability to handle
complex design scenarios.

Granularity Level Strengths Limitations

High-Level Summary Simplifies user interaction. Limited detail for complex designs.

Balances abstraction and

Detailed Summary Annotation-intensive.
detail.

Enables precise code Computationally expensive for fine-

Block-Level Summary
generation. tuning.

Improves debugging and May not generalize well to broader

Line-by-Line Comments
transparency. prompts.

Table 4: Strengths and Limitations of different summaries

Table 4 summarizes how different granularity levels influence model performance metrics such
as pass@1, pass@5, and pass@10.

The MG-Verilog dataset overcomes limitations in prior datasets used for Verilog code generation,
such as:

● Limited size and diversity (e.g., datasets used in VerilogEval [10] and ChipNemo [11]
projects).
● Lack of alignment between code and descriptions.
● Over-reliance on either high-level or detailed data, which reduces flexibility in model
applications.

These improvements make MG-Verilog a comprehensive dataset suitable for both pretraining and
fine-tuning.

11
2.4 Large Language Models (LLMs)

2.4.1 Overview of LLMs

Large Language Models are deep learning systems trained to predict and generate text based on
contextual inputs. By analysing patterns within extensive text corpora, LLMs excel in tasks ranging
from text summarization and translation to code completion. Their defining features include:

1. Scale: Modern LLMs, such as GPT-3 [4], utilize billions of parameters to capture nuanced
linguistic and logical relationships.
2. Contextual Understanding: Self-attention mechanisms allow LLMs to process input
holistically, maintaining consistency and relevance in outputs [8].
3. Versatility: LLMs can adapt to a wide range of tasks through fine-tuning or prompt
engineering, making them powerful tools for domain-specific applications.
4. Emergent Behaviours: Larger models exhibit behaviours such as reasoning and multi-step
logic inference, which emerge as a result of scale and training complexity.

The trajectory of LLMs can be divided into three distinct phases:

1. Initial Transformer Models:

The transformer architecture, introduced by Vaswani [8], revolutionized NLP by

overcoming the limitations of RNNs and LSTMs. Early models like BERT [12] focused on
understanding text but were limited to natural language comprehension tasks.

2. Scaling to General-Purpose LLMs:

GPT-3 [4] demonstrated that increasing model size and training on diverse datasets
enabled strong performance across numerous tasks. Despite its success, GPT-3 was not
specifically optimized for code generation.

3. Specialization for Programming Tasks:

Fine-tuning general-purpose LLMs on code datasets led to models such as Codex [1].
More recently, dedicated code models like CodeLlama, Qwen 2.5 Coder, and DeepSeek
Coder have emerged, optimized for programming languages with curated datasets and
advanced tokenization methods.

12
2.4.2 Transformer: The Backbone of LLMs

Figure 2: Transformer Architecture [8]

The transformer architecture forms the backbone of LLMs, enabling parallel processing and long-
range dependency capture. Key components include:

1. Input Representation

The process begins with the input text, which is tokenized into numerical representations that the
model can process. Each token is mapped to a high-dimensional vector through embeddings,
capturing semantic meaning.

2. Positional Encoding

Since transformers process tokens in parallel rather than sequentially, positional encoding is
added to the token embeddings to preserve the order of words. This ensures the model
understands the syntax and flow of the input sequence, which is crucial for maintaining context
and meaning.

13
 3. Self-Attention

At the core of the transformer is the self-attention mechanism. This dynamically calculates the
importance of each token in relation to every other token in the sequence. It enables the model
to focus on relevant parts of the input, understanding dependencies and relationships, such as
long-range interactions between words.

4. Multi-Head Attention

To enhance its representational power, the transformer employs multi-head attention. Multiple
attention heads allow the model to focus on different aspects of the input simultaneously,
capturing diverse relationships and patterns across the sequence.

5. Feed-Forward Layers

Each attention layer is followed by feed-forward layers. These non-linear transformations help the
model learn complex patterns and features. The feed-forward layers operate independently on
each token, further refining the representation.

6. Normalization and Residual Connections

To stabilize training and improve gradient flow, the transformer uses residual connections and
layer normalization around the attention and feed-forward layers. These components ensure that
information flows efficiently through the network.

7. Scalability

The transformer architecture is highly scalable, making it ideal for handling large datasets and
massive models. Its parallel processing capability allows it to train on billions of parameters
efficiently, enabling the development of powerful large language models.

8. Output Representation

After processing through multiple layers of attention and feed-forward networks, the transformer
outputs a sequence of token representations. These representations can be used for various
tasks, such as predicting the next word, generating text, or understanding complex input queries.

2.4.3 LLMs Used in Our Research

This research employs four cutting-edge Code-specific LLMs tailored for code generation:
CodeLlama 7B Instruct, Qwen 2.5 Coder 7B Instruct, DeepSeek Coder 6.7B Instruct and
CodeGemma 7B Instruct. These models were chosen for their state-of-the-art architectures and
proven effectiveness in generating code for domain-specific languages like Verilog.

Code-specific LLMs integrate programming domain knowledge into their architecture and training.
Table 5 shows the key differences between General-Purpose LLMs and Code-Specific LLMs:

14
 Aspect General-Purpose LLMs Code-Specific LLMs

Text from diverse sources (books, Programming code repositories like

Training Data
articles). GitHub.
General-purpose tokenizers (e.g.,
Tokenization Specialized tokenizers for code syntax.
byte-pair encoding).
Adapted to tasks like summarization or Fine-tuned on code-specific tasks, such
Fine-Tuning
translation. as bug detection.
Evaluation Language metrics (e.g., BLEU, Code metrics (e.g., pass@k, functional
Metrics perplexity). accuracy).

Table 5: Differences between General-Purpose LLMs and Code-Specific LLMs

1. CodeLlama 7B Instruct

CodeLlama-7B Instruct, developed by Meta AI, is a 7-billion-parameter transformer-based model

fine-tuned for coding and conversational tasks. As part of the Code Llama family—which includes
models with 7B, 13B, 34B, and 70B parameters—this model is based on the Llama 2 architecture
and has been trained on sequences of up to 16,000 tokens, showing improvements on inputs with
up to 100,000 tokens [2]. The 7B and 13B versions also support code infilling capabilities, allowing
the model to fill in missing sections of code given the surrounding context. The Instruct variant is
specifically fine-tuned to follow instructions, making it particularly useful for tasks that require
natural language interpretation and safer deployment [2]. CodeLlama-7B Instruct has been
evaluated on major code generation benchmarks, including HumanEval and MBPP,
demonstrating its proficiency in generating syntactically accurate and semantically valid code
across various programming languages [2]. Its robust instruction-following capabilities and
versatility make it a powerful tool for developers seeking assistance in a wide range of
programming tasks.

15
 Figure 3: Model Architecture of CodeLlama 7B Instruct

Detailed Breakdown of the architecture components:

● LlamaForCausalLM: This is the main class representing the language model designed for
causal language modelling tasks. It encompasses the core components necessary for
processing input text and generating outputs.
● model (LlamaModel): The core structure of the model, containing the embedding layer,
multiple decoder layers, and normalization components.
● embed_tokens (Embedding): This layer transforms input tokens into a 4096-dimensional
vector space, facilitating the model's understanding of the input. The vocabulary size is
32,016, indicating the range of unique tokens the model can process.
● layers (ModuleList): A collection of 32 LlamaDecoderLayer instances, each responsible
for processing the input data through attention mechanisms and feedforward neural networks.
● LlamaDecoderLayer: Each decoder layer comprises several sub-components:
● self_attn (LlamaSdpaAttention): Implements the self-attention mechanism, allowing the
model to weigh the importance of different tokens in the input sequence.
● q_proj, k_proj, v_proj (Linear): Linear transformations applied to the input to generate
queries, keys, and values for the attention mechanism. Each operates on 4096 input features
and produces 4096 output features without a bias term.

16
● o_proj (Linear): A linear transformation that projects the output of the attention mechanism
back to the model's dimensionality.
● rotary_emb (LlamaRotaryEmbedding): Incorporates positional information into the
attention mechanism using rotary positional embeddings, enhancing the model's ability to
understand the order of tokens.
● mlp (LlamaMLP): A feedforward neural network within each decoder layer that processes the
output of the attention mechanism.
● gate_proj, up_proj, down_proj (Linear): Linear layers that transform the input through an
intermediate higher-dimensional space (11,008 dimensions) and back to the original
dimensionality (4096 dimensions).
● act_fn (SiLU): The activation function applied within the MLP, introducing non-linearities to
the model.
● input_layernorm, post_attention_layernorm (LlamaRMSNorm): Normalization layers
applied before the attention mechanism and after it, respectively, to stabilize and normalize
the input and output activations within the decoder layer.
● norm (LlamaRMSNorm): A final normalization layer applied after all decoder layers to ensure
stable activations before generating the final output.
● rotary_emb (LlamaRotaryEmbedding): An additional instance of rotary positional
embeddings, potentially used globally within the model to encode positional information.
● lm_head (Linear): The final linear layer that maps the model's output to the vocabulary size
(32,016), producing logits for each token in the vocabulary, which can be converted to
probabilities for language modelling tasks.

2. Qwen 2.5 Coder 7B Instruct

Qwen 2.5 Coder 7B Instruct, developed by Alibaba Cloud's Qwen team, is a transformer-based
model optimized for code-related tasks. Supporting 92 programming languages, including Verilog
and System Verilog, it excels in code generation, completion, and repair [5]. The model's
architecture accommodates a context length of up to 128,000 tokens, enabling it to handle
complex and extensive codebases effectively [5]. Its 32B variant, Qwen 2.5 Coder 32B Instruct,
has achieved state-of-the-art performance among open-source models, demonstrating coding
capabilities comparable to proprietary models like GPT-4o. This makes it particularly well-suited
for hardware design applications that require hierarchical and reusable code structures [5].

17
 Figure 4: Model Architecture of Qwen 2.5 Coder 7B Instruct

Detailed Breakdown of the architecture components:

● Qwen2ForCausalLM: This is the main class representing the language model designed for
causal language modelling tasks. It encompasses the core components necessary for
processing input text and generating outputs.
● model (Qwen2Model): The core structure of the model, containing the embedding layer,
multiple decoder layers, and normalization components.
● embed_tokens (Embedding): This layer transforms input tokens into a 3584-dimensional
vector space, facilitating the model's understanding of the input. The vocabulary size is
152,064, indicating the range of unique tokens the model can process.
● layers (ModuleList): A collection of 28 Qwen2DecoderLayer instances, each responsible
for processing the input data through attention mechanisms and feedforward neural networks.
● Qwen2DecoderLayer: Each decoder layer comprises several sub-components:
● self_attn (Qwen2SdpaAttention): Implements the self-attention mechanism, allowing the
model to weigh the importance of different tokens in the input sequence.
● q_proj (Linear): A linear transformation applied to the input to generate queries for the
attention mechanism. It operates on 3584 input features and produces 3584 output features
with a bias term.

18
● k_proj (Linear): A linear transformation applied to the input to generate keys for the attention
mechanism. It operates on 3584 input features and produces 512 output features with a bias
term.
● v_proj (Linear): A linear transformation applied to the input to generate values for the
attention mechanism. It operates on 3584 input features and produces 512 output features
with a bias term.
● o_proj (Linear): A linear transformation that projects the output of the attention mechanism
back to the model's dimensionality (3584 features) without a bias term.
● rotary_emb (Qwen2RotaryEmbedding): Incorporates positional information into the
attention mechanism using rotary positional embeddings, enhancing the model's ability to
understand the order of tokens.
● mlp (Qwen2MLP): A feedforward neural network within each decoder layer that processes
the output of the attention mechanism.
● gate_proj (Linear): A linear layer that transforms the input through an intermediate higher-
dimensional space (18,944 dimensions) without a bias term.
● up_proj (Linear): Another linear layer that projects the input to the higher-dimensional space
(18,944 dimensions) without a bias term.
● down_proj (Linear): A linear layer that projects the higher-dimensional representation back
to the original dimensionality (3584 dimensions) without a bias term.
● act_fn (SiLU): The activation function applied within the MLP, introducing non-linearities to
the model.
● input_layernorm (Qwen2RMSNorm): A normalization layer applied before the attention
mechanism to stabilize and normalize the input activations within the decoder layer.
● post_attention_layernorm (Qwen2RMSNorm): A normalization layer applied after the
attention mechanism to stabilize and normalize the output activations within the decoder layer.
● norm (Qwen2RMSNorm): A final normalization layer applied after all decoder layers to
ensure stable activations before generating the final output.
● rotary_emb (Qwen2RotaryEmbedding): An additional instance of rotary positional
embeddings, potentially used globally within the model to encode positional information.
● lm_head (Linear): The final linear layer that maps the model's output to the vocabulary size
(152,064), producing logits for each token in the vocabulary, which can be converted to
probabilities for language modelling tasks.

3. DeepSeek Coder 6.7B Instruct

DeepSeek Coder 6.7B Instruct is a transformer-based model developed by DeepSeek AI,

designed to enhance code generation, particularly in error-prone and low-resource environments.
Trained on a substantial dataset of 2 trillion tokens—comprising 87% code and 13% natural
language in both English and Chinese—the model is fine-tuned on 2 billion tokens of instruction
data to support project-level code completion and infilling tasks [7]. Its architecture employs a
16,000-token window size and incorporates a fill-in-the-blank task, enabling it to handle complex
coding scenarios effectively [7]. The model demonstrates state-of-the-art performance across
multiple programming languages and benchmarks, including HumanEval, MultiPL-E, MBPP, DS-
1000, and APPS [7]. Notably, it excels in generating Verilog code with minimal syntactical and
logical errors, making it a valuable tool for developers focused on debugging and edge-case

19
scenarios. Additionally, its scalable design—available in sizes ranging from 1B to 33B
parameters—facilitates efficient training and deployment, allowing users to select the model size
that best fits their requirements [7].

Figure 5: Model Architecture of DeepSeek Coder 6.7B Instruct

Detailed Breakdown of the architecture components:

 LlamaForCausalLM: This is the main class representing the language model designed for
causal language modelling tasks. It encompasses the core components necessary for
processing input text and generating outputs.
 model (LlamaModel): The core structure of the model, containing the embedding layer,
multiple decoder layers, and normalization components.
 embed_tokens (Embedding): This layer transforms input tokens into a 4096-dimensional
vector space, facilitating the model's understanding of the input. The vocabulary size is
32,256, indicating the range of unique tokens the model can process.
 layers (ModuleList): A collection of 32 LlamaDecoderLayer instances, each responsible
for processing the input data through attention mechanisms and feedforward neural networks.
 LlamaDecoderLayer: Each decoder layer comprises several sub-components:

20
 self_attn (LlamaSdpaAttention): Implements the self-attention mechanism, allowing the
model to weigh the importance of different tokens in the input sequence.
 q_proj, k_proj, v_proj (Linear): Linear transformations applied to the input to generate
queries, keys, and values for the attention mechanism. Each operates on 4096 input features
and produces 4096 output features without a bias term.
 o_proj (Linear): A linear transformation that projects the output of the attention mechanism
back to the model's dimensionality.
 rotary_emb (LlamaRotaryEmbedding): Incorporates positional information into the
attention mechanism using rotary positional embeddings, enhancing the model's ability to
understand the order of tokens.
 mlp (LlamaMLP): A feedforward neural network within each decoder layer that processes the
output of the attention mechanism.
 gate_proj, up_proj, down_proj (Linear): Linear layers that transform the input through an
intermediate higher-dimensional space (11,008 dimensions) and back to the original
dimensionality (4096 dimensions).
 act_fn (SiLU): The activation function applied within the MLP, introducing non-linearities to
the model.
 input_layernorm, post_attention_layernorm (LlamaRMSNorm): Normalization layers
applied before the attention mechanism and after it, respectively, to stabilize and normalize
the input and output activations within the decoder layer.
 norm (LlamaRMSNorm): A final normalization layer applied after all decoder layers to ensure
stable activations before generating the final output.
 rotary_emb (LlamaRotaryEmbedding): An additional instance of rotary positional
embeddings, potentially used globally within the model to encode positional information.
 lm_head (Linear): The final linear layer that maps the model's output to the vocabulary size
(32,256), producing logits for each token in the vocabulary, which can be converted to
probabilities for language modelling tasks.

4. CodeGemma 7B It / CodeGemma 7B Instruct

CodeGemma-7B-it or CodeGemma-7B-Instruct, developed by Google, is a 7-billion-parameter

instruction-tuned variant of the CodeGemma model family, which is built upon Google's Gemma
architecture. This model specializes in code completion, code generation, and engaging in
conversational interactions about programming and technical problems. Trained on an extensive
dataset of 500 billion tokens, primarily comprising English language data from publicly available
code repositories and open-source mathematics datasets, CodeGemma-7B-it is designed to
handle both fill-in-the-middle (FIM) tasks and natural language prompts. The model employs
techniques such as dependency graph-based packing and unit test-based lexical packing to
enhance its alignment with real-world applications. Evaluation results indicate that CodeGemma-
7B-it achieves a 56.1% pass rate on the HumanEval benchmark and 54.2% on the MBPP
benchmark, demonstrating its proficiency in generating accurate and reliable code [6].

21
 Figure 6: Model Architecture of CodeGemma 7B Instruct

Detailed Breakdown of the architecture components:

● GemmaForCausalLM: This is the primary class representing a language model tailored for
causal language modeling tasks. It integrates the essential components required for
processing input text and generating coherent outputs.
● model (GemmaModel): The foundational structure of the model, comprising the embedding
layer, multiple decoder layers, and normalization components.
● embed_tokens (Embedding): This layer converts input tokens into a 3,072-dimensional
vector space, facilitating the model's comprehension of the input. The vocabulary size is
256,000, indicating the range of unique tokens the model can process. The padding_idx=0
parameter specifies that the token with index 0 is used for padding sequences.
● layers (ModuleList): A collection of 28 GemmaDecoderLayer instances, each responsible
for processing the input data through attention mechanisms and feedforward neural networks.
● GemmaDecoderLayer: Each decoder layer comprises several sub-components:
● self_attn (GemmaSdpaAttention): Implements the self-attention mechanism, enabling the
model to weigh the significance of different tokens in the input sequence.
● q_proj, k_proj, v_proj (Linear): Linear transformations applied to the input to generate
queries, keys, and values for the attention mechanism. Specifically:
● q_proj: Projects 3,072 input features to 4,096 output features.

22
● k_proj: Projects 3,072 input features to 4,096 output features.
● v_proj: Projects 3,072 input features to 4,096 output features. All these projections are
performed without bias terms.
● o_proj (Linear): A linear transformation that projects the output of the attention mechanism
from 4,096 features back to the model's dimensionality of 3,072 features, also without a bias
term.
● rotary_emb (GemmaRotaryEmbedding): Incorporates positional information into the
attention mechanism using rotary positional embeddings, enhancing the model's ability to
understand the order of tokens.
● mlp (GemmaMLP): A feedforward neural network within each decoder layer that processes
the output of the attention mechanism.
● gate_proj, up_proj, down_proj (Linear): Linear layers that transform the input through an
intermediate higher-dimensional space (24,576 dimensions) and back to the original
dimensionality (3,072 dimensions), all without bias terms.
● act_fn (PytorchGELUTanh): The activation function applied within the MLP, introducing non-
linearities to the model. The PytorchGELUTanh is a fast C implementation of the tanh
approximation of the Gaussian Error Linear Units (GELU) activation function. This
approximation is designed to provide the benefits of GELU with improved computational
efficiency.
● input_layernorm, post_attention_layernorm (GemmaRMSNorm): Normalization layers
applied before the attention mechanism and after it, respectively, to stabilize and normalize
the input and output activations within the decoder layer. Both use an epsilon value of 1e-06
to prevent division by zero errors.
● norm (GemmaRMSNorm): A final normalization layer applied after all decoder layers to
ensure stable activations before generating the final output, also with an epsilon value of 1e-
06.
● lm_head (Linear): The concluding linear layer that maps the model's output to the vocabulary
size (256,000), producing logits for each token in the vocabulary. This layer operates without
a bias term.

2.5 Fine-Tuning Techniques for LLMs

In the rapidly advancing fields of Artificial Intelligence (AI) and Natural Language Processing
(NLP), Large Language Models (LLMs) have become essential tools across a wide variety of
applications. To achieve optimal results from these models, there are three primary approaches:
Prompt Engineering, Fine-Tuning, and Training a New Model from Scratch.

23
 Figure 7: Techniques for improvement of LLM on Targeted Domain

These methods differ in complexity, resource requirements, and the level of customization they
offer. As we move from one approach to the next, the associated computational costs and
infrastructure needs increase substantially.

1. Prompt Engineering Using Pre-Trained Models: At the most fundamental level, prompt
engineering involves carefully designing inputs or prompts to guide the model toward
producing the desired output. By adjusting the wording, structure, or instructions within the
prompt, users can elicit responses that align with their goals. This method is resource efficient
as it relies solely on pre-trained models without any additional training. It is particularly
effective for straightforward tasks where general-purpose model knowledge suffices, such as
text summarization, answering simple questions, or generating basic code snippets [13].

2. Fine-Tuning Pre-Trained Models: Fine-tuning offers a deeper level of specialization

compared to prompt engineering. It involves further training a pre-existing pre-trained model
on a smaller, domain-specific dataset. This process allows the model to adjust its parameters
to focus on a particular task or subject area. Fine-tuning achieves significantly better results
for specialized tasks, such as generating domain-specific code like Verilog. By building on the
model's pre-trained knowledge, fine-tuning requires far fewer resources compared to training
a model from scratch while still improving task-specific accuracy [13].

3. Training a New Model from Scratch: For the highest level of customization, a new model
can be trained entirely from the ground up using a large dataset tailored to a specific task or
domain. This method provides complete control over the model’s architecture, training data,
and behaviour. However, it is also the most resource-intensive approach, requiring enormous
amounts of computational power, labelled training data, and time. As such, this method is
generally only employed for highly specialized or large-scale projects where existing pre-
trained models are inadequate [13].

Given the resource constraints often associated with fine-tuning LLMs, innovative techniques like
Parameter-Efficient Fine-Tuning (PEFT) have been developed. PEFT approaches, such as

24
Low-Rank Adaptation (LoRA) and Quantized LoRA (QLoRA), strike a balance between
efficiency and performance. They enable fine-tuning of large models using minimal infrastructure,
making it possible to achieve task-specific customization without compromising performance.
This section discusses the importance of fine-tuning, the general steps involved, different
techniques of fine tuning and the specific advantages of QLoRA (Quantized Low-Rank
Adaptation), a state-of-the-art fine-tuning method that balances efficiency and performance.

2.5.1 Overview of Fine-Tuning

Fine-tuning refers to the process of adapting a pre-trained language model to a specific domain,
task, or dataset by adjusting its parameters. While pre-trained LLMs possess general knowledge
acquired from diverse datasets, fine-tuning allows the model to specialize in specific contexts,
such as hardware description languages (HDLs) like Verilog.

Figure 8: Fine-Tuning Process [14]

Fine-tuning can be likened to refining a general-purpose tool to perform a highly specialized task
with precision. Imagine guiding a multi-talented individual to focus exclusively on mastering a
single skill for a particular occasion. Similarly, fine-tuning involves training a pre-trained language
model on a smaller, domain-specific dataset to adapt its capabilities for a specialized task.

The Need for Fine-Tuning LLMs

While pre-trained language models demonstrate impressive versatility, they are not inherently
designed for specific tasks. Fine-tuning bridges this gap by aligning the model’s general-purpose
knowledge with the nuances of a specialized task, such as sentiment analysis or hardware
description language generation. It enables the model to understand domain-specific details and
task-specific requirements. This process brings three main advantages:

25
 1. Task Specialization: Pre-trained LLMs are designed to handle a broad spectrum of tasks
but may lack the precision needed for domain-specific languages. Fine-tuning improves
the model's understanding of specialized syntax, semantics, and domain constraints [4].
2. Enhanced Performance: Fine-tuning significantly enhances the quality of model outputs
by aligning them with the desired task. For example, a fine-tuned model can generate
syntactically accurate and logically correct Verilog code.
3. Data Alignment: Fine-tuning ensures that the model adapts to the idiosyncrasies of the
target dataset, such as the hierarchical structure and parallelism inherent in Verilog
designs [15].

2.5.2 General Steps in Fine-Tuning

Fine-tuning large language models (LLMs) involves a systematic process of adapting pre-trained
models to perform specialized tasks effectively. This section outlines the steps involved in fine-
tuning, incorporating modifications for clarity and originality.

Figure 9: General Steps in Fine-Tuning [16]

1. Dataset Preparation

The first step in fine-tuning is to assemble a high-quality dataset tailored to the task. For Verilog
code generation, this involves creating datasets with structured descriptions, such as Verilog
Pyramid of Thoughts summaries, global-level summaries, detailed-level summaries, and block-
level summaries. The dataset should comprehensively represent the target domain while
maintaining diversity and balance [16].

2. Preprocessing

26
The dataset is prepared by cleaning, tokenizing, and formatting it to match the pre-trained model's
input requirements. This step includes splitting the data into training, validation and test sets to
facilitate effective model training and evaluation. Proper preprocessing ensures compatibility with
the model architecture and enhances training efficiency [16].

3. Selecting a Pre-Trained Model

Next step is to choose a pre-trained model that best aligns with the requirements of the task at
hand. These models are pre-trained on large-scale, diverse datasets of unlabelled text, giving
them a solid foundation of general knowledge. By selecting a model designed for tasks like code
generation, such as GPT-3, GPT-4, or specialized models like CodeLlama, one ensures a strong
starting point for adaptation [16].

4. Fine Tuning

During the training phase, the model learns task-specific patterns by updating its parameters
based on the provided dataset. This involves techniques like gradient descent and
backpropagation to iteratively refine the model’s understanding of the domain. Training is
conducted using an optimized configuration of hyperparameters to balance learning efficiency
and accuracy [16].

5. Hyperparameter Tuning

Fine-tuning also requires careful selection of hyperparameters such as learning rate, batch size,
and regularization strength. These hyperparameters play a critical role in determining the model's
learning dynamics. Adjusting them ensures the model generalizes effectively to unseen data while
minimizing the risk of overfitting [16].

6. Validation

Throughout the training process, the model’s performance on a validation set is monitored. This
step is crucial for assessing how well the model is adapting to the task and identifying potential
overfitting. By analysing validation results, one can adjust training strategies to improve overall
performance [16].

7. Testing

After completing training, the model on an unseen test dataset is evaluated. Testing provides an
unbiased measure of the model's performance and its ability to handle new data. This step helps
ensure the fine-tuned model will be reliable in real-world applications [16].

8. Early Stopping

To avoid overfitting, early stopping mechanisms during training can be employed. If the model's
performance on the validation set stagnates or begins to degrade, the training process is halted.

27
Early stopping not only preserves generalization ability but also saves computational resources
[16].

9. Iterative Refinement

Fine-tuning is often iterative. Based on performance metrics from the validation and test sets, the
model’s architecture, dataset, or hyperparameters are refined. This iterative approach allows for
incremental improvements and ensures optimal results [16].

10. Deployment

Once the fine-tuned model demonstrates reliable performance, it is deployed for real-world use.
This can involve integrating the model into software systems or services for specific tasks like
Verilog code generation, text summarization, or other domain-specific applications [16].

2.5.3 Types of Fine-Tuning Techniques

Fine-tuning large language models (LLMs) can be approached through a variety of techniques,
including standard fine-tuning, adapter-based methods, prefix-tuning, chain-of-thought prompting,
sequential regularization, zero-shot learning, and others.

To simplify, these fine-tuning techniques can be broadly grouped into two main categories:

Figure 10: Overview of Fine-Tuning Techniques

1. Instruction Tuning: The goal here is to improve the model’s ability to follow instructions
by training it on datasets containing prompts paired with corresponding responses [17].

28
 {

"instruction": "...",

"output": "...",

2. Preference Optimization: This focuses on aligning the model's outputs with specific
preferences, often reflecting user requirements or desired behaviours [17].

"instruction": "...",

"preferred": "...",

"rejected": "...",

These approaches are not mutually exclusive and can be combined, with instruction tuning
forming the foundation and preference optimization enhancing alignment with user expectations.

Instruction Tuning

Supervised Fine-Tuning (SFT): Supervised fine-tuning involves training a pre-trained language

model on a labelled dataset tailored to specific tasks. This enables the model to adjust its
parameters to predict the correct output for a given input. By leveraging the foundational
knowledge from pre-training, SFT enhances the model's performance on specific applications.
There are two approaches to SFT:

1. Full Fine-Tuning: Full fine-tuning refers to adjusting all of the model's weights during the
training process. While this approach allows for extensive adaptation to new tasks, it is
computationally intensive. It requires calculating gradients for all parameters to minimize the
loss between the model’s predictions and the actual outputs. This process involves billions of
floating-point operations and frequent data movement within the GPU memory, making it a
memory-demanding task. Additionally, it poses risks such as catastrophic forgetting, where
the model loses general knowledge from its pre-trained stage. Full fine-tuning also results in
large, task-specific models that lack flexibility, making them impractical for applications
requiring modularity or adaptation across multiple domains.

2. Parameter-Efficient Fine-Tuning (PEFT): To address the challenges of Full fine-tuning ,

techniques like Parameter-Efficient Fine-Tuning (PEFT) have emerged. These methods
minimize the number of trainable parameters, significantly reducing computational
requirements while preserving the model's ability to generalize. PEFT methods also avoid

29
 altering all parameters, making the model adaptable and more efficient for resource-
constrained settings.

2.5.4 PEFT Fine-Tuning Method

Advancements in parameter-efficient fine-tuning techniques, such as QLoRA, have revolutionized

the fine-tuning process by reducing computational costs while maintaining performance. This
section explores Parameter-Efficient Fine-Tuning (PEFT) approaches, including LoRA and
QLoRA, and highlights the advantages of QLoRA for Verilog code generation.

Parameter-Efficient Fine-Tuning (PEFT)

Parameter-Efficient Fine-Tuning (PEFT) has emerged as a transformative approach for adapting

large language models (LLMs) to specific tasks. Unlike traditional full fine-tuning, which requires
modifying all of a model's millions of parameters, PEFT strategically updates only a small subset.
This selective approach makes PEFT particularly appealing for resource-constrained
environments while maintaining impressive performance levels.

PEFT Full fine-tuning

Figure 11: PEFT vs Full Fine-tuning [18]

How PEFT Works

PEFT techniques typically involve freezing the majority of a model’s parameters, especially those
in early layers responsible for foundational language understanding. These frozen layers preserve
the core knowledge acquired during pretraining, allowing the model to retain its general-purpose
capabilities. Fine-tuning is then focused on the later layers or on newly introduced parameters,
enabling task-specific adaptations without overwriting the original weights.

30
 Figure 12: Overview of PEFT [19]

In traditional fine-tuning, significant memory is required to store not only the model but also the
training-related elements, such as optimizer states, gradients, forward activations, and temporary
buffers. These elements often demand more memory than the model itself, which can quickly
exceed the capabilities of consumer-grade hardware. In contrast, PEFT alleviates these
challenges by limiting the number of trainable parameters, ensuring that memory usage remains
manageable even on smaller GPUs [19].

Why Use PEFT?

PEFT stands out due to several key advantages that make it an appealing alternative to full fine-
tuning:

1. Reduced Computational Costs: By updating only a subset of the model's parameters,

PEFT significantly lowers computational requirements. This allows large-scale models to
be fine-tuned with fewer GPUs and reduced GPU time, making the process more
affordable and accessible [19].

2. Faster Training Times: Since fewer parameters are being optimized, training progresses
much faster. This acceleration enables quicker iterations and shorter deployment cycles,
making it ideal for dynamic real-world applications [19].

3. Lower Hardware Requirements: PEFT is highly efficient on smaller GPUs, requiring less
memory compared to full fine-tuning. This makes it feasible for users working with limited
hardware resources [19].

4. Improved Modelling Performance: By focusing on task-relevant parameters and

reducing overfitting, PEFT often produces models that are more robust and accurate
across diverse tasks [19].

31
 5. Space-Efficient Storage: PEFT minimizes storage needs by sharing weights across
tasks. This efficiency simplifies model deployment and management, especially when the
same base model is fine-tuned for multiple applications [19].

PEFT Techniques

PEFT encompasses several techniques that allow fine-tuning of pre-trained models by modifying
a small subset of their parameters or introducing additional components. Figure 13 an overview
of key PEFT methods:

Figure 13: PEFT Techniques [20]

1. Adapter-Based Models: This method involves adding additional layers, known as

adapters, on top of the frozen pre-trained model. While effective for task-specific
customization, the added components increase inference overhead and complicate
deployment, making this approach less efficient compared to other PEFT techniques [20].

2. Soft Prompts: This method adds learnable, soft tokens to the input prompts and trains
their embeddings while keeping the model’s weights unchanged. Although soft prompts
can be effective in certain scenarios, their performance often lags behind other PEFT
methods. Additionally, the added tokens introduce significant overhead during inference,
limiting their practicality [20].

3. LoRA (Low-Rank Adaptation): LoRA introduces low-rank matrices into the model’s
architecture to adjust specific weights during fine-tuning. Unlike adapters, LoRA does not
require additional components at inference time, making it a highly efficient method for
PEFT. The original model weights remain intact, and the low-rank adaptations are
seamlessly integrated into the existing layers [20].

4. DoRA (Decomposed Rank Adaptation): Building on LoRA, DoRA refines the adaptation
process by decomposing weights into their magnitude and direction. This decomposition
allows for more targeted and focused fine-tuning, improving efficiency and performance
compared to traditional LoRA [20].

32
 5. Selective Fine-Tuning: This method involves selecting and retraining a subset of the
model’s original parameters. While it can be effective in some cases, results from this
approach have been inconsistent across different tasks [20].

Among these techniques, LoRA stands out as one of the most effective and widely used PEFT
methods due to its simplicity, efficiency, and lack of additional inference overhead. Its proven
performance makes it a valuable choice for fine-tuning large language models for specialized
tasks, such as Verilog code generation.

2.5.5 Low-Rank Adaptation (LoRA)

Low-Rank Adaptation (LoRA) is a Parameter-Efficient Fine-Tuning (PEFT) technique designed to

enhance the efficiency of fine-tuning large language models (LLMs). Instead of updating the entire
model, LoRA introduces low-rank matrices into the weight updates, significantly reducing
computational and memory requirements. By focusing on adjusting only a small subset of the
model’s parameters, LoRA makes fine-tuning feasible even for models with extensive parameter
counts, without sacrificing accuracy. This approach has proven particularly effective for various
natural language processing (NLP) and code generation tasks, enabling large-scale models to be
adapted efficiently to specific applications [21].

Concept of LoRA

LoRA leverages low-rank adaptation to fine-tune LLMs more efficiently. Instead of adjusting all
the parameters of the pre-trained model, LoRA introduces additional low-rank matrices that
approximate the changes needed for task-specific adaptation.

● Efficient Parameter Updates: LoRA focuses on tracking updates through smaller low-
rank matrices rather than modifying the full set of model weights [14].
● Reduced Resource Usage: This approach significantly decreases the memory and
computational demands, making it possible to fine-tune large models on resource-limited
hardware [14].

Figure 14: Concept of LoRA [17]

33
In LoRA, instead of fine-tuning the entire weight matrix (W) of the pre-trained model, two smaller
matrices, A and B, are fine-tuned to approximate the updates to W.

Rank Decomposition Matrix

The core of LoRA lies in its use of rank decomposition matrices, which consist of two linear
transformations. These transformations reduce and then restore the dimensionality of the input.
The outputs of these transformations are added to the outputs derived from the pre-trained
weights, effectively creating a modified layer [22].

● The matrix product AB has the same dimensions as a full fine-tuning update but is
decomposed into two smaller matrices. This low-rank decomposition reduces the number
of parameters that need to be optimized [22].
● Instead of adjusting the original weights in the model’s layers, LoRA focuses on optimizing
the rank decomposition matrices. This process approximates the full fine-tuning update
while being far more efficient [22].
● The matrices A and B are initialized with specific values: A starts with small, random
values, while B begins with zero. This ensures that the model starts with its original pre-
trained weights intact [22].

How to Apply LoRA

To understand how LoRA is applied, let us consider the traditional Transformer architecture,
which consists of dense layers with two types of neural networks: the self-attention network and
the feed-forward network. The weights for these networks are learned during the pre-training
process.

After the embedding vectors are generated, they pass through the self-attention layers where
attention scores are computed. Applying LoRA specifically to these self-attention layers has been
found to be highly effective, delivering strong results while significantly reducing computational
costs.

Although LoRA can also be applied to the feed-forward layers, the majority of a large language
model's parameters are concentrated in the self-attention layers. Therefore, applying LoRA to
these weight matrices offers the most substantial reduction in trainable parameters.

LoRA can be used on both the encoder and decoder components of Transformer-based models.
However, to better understand its application, we can focus on how it operates within the encoder
component to gain a clearer intuition of its functionality.

34
 Figure 15: Transformer Architecture [20]

LoRA process in high level

Figure 16: Add LoRA to the self-attention layer [18]

Let us consider a base Transformer model with dimensions 512×64. To apply low-rank
decomposition, we use a mathematical approach where a matrix is expressed as the product of
two smaller matrices. For this example, let’s choose a rank of 8 to decompose the original

35
Transformer matrix, which initially has 32,768 trainable parameters. Using a rank of 8 reduces
this to just 4,608 trainable parameters.

Figure 17: Low Rank Decomposition with r = 8 [20]

Steps to update the model for inference

Figure 18: Steps to update the model for inference [20]

Importantly, the overall number of parameters in the model remains unchanged, ensuring there
is no added latency during inference.

One of LoRA’s key strengths is its flexibility. For instance, if we train a pair of LoRA matrices for
a specific task, Task A, these matrices can be multiplied and added to the frozen layers during
inference. The updated weights can then be replaced seamlessly in the original model. If we need
to fine-tune for another task, Task B, the same process can be repeated. Additionally, storing
these LoRA matrices requires very little memory, allowing us to switch between task-specific
weights efficiently as needed.

36
2.5.6 Quantized Low-Rank Adaptation (QLoRA)

Quantized Low-Rank Adaptation (QLoRA) is an advanced extension of Low-Rank Adaptation

(LoRA), designed to further optimize the efficiency of fine-tuning large language models (LLMs).
It builds on LoRA by integrating quantization, a process that reduces the precision of the original
model’s weights from high-resolution data types like Float32 to lower-resolution formats such as
int4, resulting in decreased memory usage and computational requirements while maintaining
high performance.

Aspect LoRA QLoRA

Weight Precision Full precision 4-bit quantization

Memory Usage Moderate Significantly lower

Computational Efficiency Moderate Highly efficient

Accuracy High Comparable to full-precision fine-tuning

Suitability for Resource-Constrained

Limited Excellent
Environments

Table 6: Difference Between LoRA and QLoRA

The introduction of quantization in QLoRA offers a significant edge over LoRA by enabling fine-
tuning on devices with limited computational resources, such as GPUs with reduced memory
capacity.

Why QLoRA is Suitable for Verilog Code Generation

LoRA and QLoRA are among the most popular and effective techniques for Parameter-Efficient
Fine-Tuning, combining innovations in memory optimization and computational efficiency to adapt
large models for specific tasks.

1. Domain-Specific Efficiency: Verilog requires understanding both syntax and logic-level

correctness. QLoRA’s efficiency ensures that models can be fine-tuned on detailed and
specialized datasets, such as the block-level summaries of Verilog, without requiring
extensive computational resources.
2. Handling Dataset Granularity: QLoRA enables experimentation with diverse dataset
descriptions (e.g., global-level vs. block-level summaries), providing insights into their
impact on model performance.
3. Resource-Constrained Training: Given the high computational requirements of fine-
tuning large LLMs, QLoRA's memory and cost efficiency make it practical for fine-tuning
Verilog-specific models on mid-tier hardware setups.

37
 Advantage Description

Cost Efficiency Significantly reduces fine-tuning costs by leveraging quantization.

Memory Allows large models to be fine-tuned on resource-constrained hardware.

Optimization
Scalability Facilitates fine-tuning of models with billions of parameters.

Task-Specific Efficiently adapts LLMs for domain-specific applications like Verilog.

Adaptation
Table 7: Advantages of QLoRA

Key Optimizations in QLoRA

QLoRA brings three key innovations on top of LoRA, making it one of the most efficient
Parameter-Efficient Fine-Tuning (PEFT) methods:

1. 4-bit NF4 Quantization

4-bit NormalFloat4 (NF4) is an optimized data type used to store model weights, substantially
reducing memory usage. The 4-bit NF4 quantization process involves three main steps:

a. Normalization and Quantization:

During this step, weights are adjusted to have a zero mean and a unit variance. Because a 4-bit
data type can represent only 16 distinct values, weights are mapped to these 16 numbers in a
zero-centred distribution. Instead of storing the precise weight value, the nearest mapped position
is saved. For instance, if a weight value of 0.2121 (in FP32 format) is closest to 0.1997 (the 10th
position in the range of -1 to 1), only the position "10" is stored [13].

The quantization process can be represented with the following formula:

In this formula:

● totalNumberOfPositions refers to the number of discrete values that the int4 data type
can represent, which is 16 in this example [13].
● The value totalNumberOfPositions/absmax(inputXTensor) is called the
quantization constant, which scales the FP32 weights to fit into the reduced precision
format [13].

38
While this introduces some data loss due to reduced precision, the impact is minimal as long as
the input tensor does not contain outliers that could skew the normalization. To mitigate this,
weights are often quantized in smaller blocks, which helps normalize the outliers effectively.

b. Dequantization:

Dequantization reverses the process, reconstructing the approximate original weights from the
quantized values.

Where, totalNumberOfPositions = 16

While the LoRA adapter weights remain in FP32 during training, the original model weights are
dequantized after training is completed.

Figure 19: Quantization and Dequantization [13]

2. Double Quantization

Double quantization further reduces the memory footprint by compressing the quantization
constants themselves. In the 4-bit NF4 quantization step, a quantization constant is computed for
each block of weights. These constants, initially stored as FP32 values, can also be quantized for
better efficiency [13].

For example, if a block consists of 64 weights, each quantization constant (32 bits in FP32) adds
an average of 0.5 bits per parameter, which amounts to approximately 500,000 bits for a model
with one million parameters. By applying 8-bit quantization to groups of 256 quantization
constants, the memory usage can be reduced significantly, achieving approximately 0.127 bits
per parameter [13].

Here’s the breakdown:

● A 64-weight block with 256 quantization constants requires 32 bits per constant, leading
to 32/(64*256) = 0.001953125 bits per parameter.
● We have 8bits for 64 weights which is 8/64 = 0.125
● If we add it up 0.125+0.001953125 which is 0.127 approximately

This approach reduces memory usage drastically while maintaining efficiency.

39
3. Unified Memory Paging

QLoRA also leverages Nvidia’s unified memory feature, which facilitates seamless data transfers
between the GPU and CPU. When GPU memory is fully utilized, this feature prevents memory
overflows by dynamically offloading excess data to the CPU. Unified memory paging effectively
manages memory spikes, preventing bottlenecks and ensuring smooth training [13].

2.6 Evaluation Framework for Verilog Code

2.6.1 Overview of Code Evaluation in LLMs

The evaluation of Verilog code generated by large language models (LLMs) is a critical step to
assess the capability of these models in hardware description tasks. Evaluating generated Verilog
code entails unique challenges compared to general-purpose programming languages. These
challenges include:

1. Syntax: Verilog code must adhere to precise syntactical rules, including correct signal
declarations, module instantiations, and proper formatting. Even minor deviations, such as
missing semicolons or incorrect signal widths, can render the code invalid for simulation or
synthesis.

2. Semantics: Beyond syntax, the generated code must capture the intended behaviour
described in the problem statement. For example, correct Boolean logic, state transitions, and
module hierarchies are vital for ensuring the generated code meets the desired functionality.

3. Functionality: The most critical evaluation metric is functional correctness. This involves
testing the generated Verilog code in a simulation environment to compare its output against
expected results, ensuring the generated design operates as intended.

2.6.2 VerilogEval Library

To tackle the challenges that Verilog code evaluation face, researcher has come up with a
evaluation framework only for Verilog code and this is called VerilogEval, a specialized evaluation
framework tailored for Verilog code generation tasks. Developed as an open-source tool, it
integrates seamlessly with simulation environments to automate the testing of generated Verilog
code. It uses a curated dataset of 156 problems from HDLBits, encompassing a diverse range of
tasks such as combinational circuits, finite state machines, and debugging challenges [10].

The VerilogEval dataset includes problems of varying complexity sourced from HDLBits. These
tasks are meticulously curated to ensure clarity and diversity. Example tasks range from simple
bit-wise operations to more complex state transition logic and testbench generation.

40
 Figure 20: Example of VerilogEval Evaluation dataset [10]

Each task in the dataset is associated with a clear problem description in natural language,
enabling LLMs to generate corresponding Verilog modules. Functional correctness is assessed
by comparing the simulation outputs of generated designs against a golden reference solution.

VerilogEval provides a robust and automated mechanism to evaluate syntactical correctness,

semantic validity, and functional correctness of Verilog code. It evaluates the quality of generated
Verilog code using the following metrics:

Evaluation
Metric Description
Method

Syntactical Measures the percentage of generated code that is free Compilation

Accuracy from syntax errors and can be successfully compiled. Logs

Functional Verifies the correctness of the output produced by the Simulation

Correctness generated code through automated simulations. Results

Determines the likelihood that at least one of the top k Simulation and
Pass@k Metric
generated solutions passes all tests. Testing

Table 8: Metrics Used in VerilogEval

Evaluation Process

The evaluation framework operates in three main stages:

1. Syntax Checking: Generated code is parsed and compiled using Verilog compilers such
as Icarus Verilog. Any syntax errors are flagged during this step.

41
 2. Simulation and Functional Testing: VerilogEval uses a simulation-based approach to
test the behaviour of the generated code. Each problem includes a set of testbenches,
consisting of manually crafted and randomly generated test patterns. These test patterns
are applied to the generated code, and its outputs are compared with the golden reference
solution.

3. Automated Metrics Calculation: VerilogEval computes metrics such as pass@k, which

represents the fraction of tasks for which at least one of the top k generated solutions
passes the functional tests. Pass@k is calculated using the following unbiased estimator:

Where,
n is the total number of samples generated per task, and c is the count of correct solutions.
we generate n ≥ k samples per task in which c ≤ n samples pass testing.

2.7 Hyperparameter Tuning in LLM Fine-Tuning

Hyperparameter tuning is a critical process in the fine-tuning of large language models (LLMs).
Fine-tuning aims to optimize the model for specific tasks, and the choice of hyperparameters can
significantly affect its performance in terms of accuracy, efficiency, and generalization. In this
study, hyperparameter tuning was employed to adapt state-of-the-art LLMs—CodeLlama 7B
Instruct, Qwen 2.5 Coder 7B Instruct, CodeGemma 7B Instruct and DeepSeek Coder 7B
Instruct—for the task of Verilog code generation. This section details the importance of
hyperparameters, describes the specific hyperparameters varied in this thesis and discusses their
expected impacts on model performance.

2.7.1 Importance of Hyperparameters

Key Hyperparameters in LLM Fine-Tuning

Hyperparameters in LLM training govern the learning process and determine how effectively the
model can adapt to the dataset. Commonly adjusted hyperparameters include:

1. Learning Rate:
 Controls the step size during the optimization process.
 A high learning rate may lead to faster convergence but risks overshooting the optimal
solution. Conversely, a low learning rate ensures more stable convergence but may
increase training time [23].

2. Batch Size:
 Defines the number of samples processed before the model updates its weights.

42
  Larger batch sizes enable smoother gradient estimates but require more computational
resources. Smaller batch sizes can lead to noisy updates but may generalize better [24].

3. Weight Decay:
 A regularization technique that penalizes large weights to prevent overfitting.
 Helps ensure that the model does not rely excessively on certain parameters [25].

4. Gradient Accumulation Steps:

 Allows for effective training with limited memory by accumulating gradients over multiple
forward passes before updating model weights.

5. Dropout:
 Introduces randomness by disabling neurons during training, helping the model generalize
better [26].

6. Learning Rate Schedulers:

 Adjust the learning rate dynamically during training. Common schedulers include linear
decay, cosine annealing, and warm-up strategies.

These hyperparameters interact in complex ways, and their combined tuning is crucial for
achieving optimal performance.

Impact of Hyperparameters on Model Performance

The impact of hyperparameters on fine-tuning LLMs for specific tasks such as Verilog code
generation includes:

● Model Accuracy: Proper tuning of the learning rate and batch size can improve the
model's ability to understand and replicate Verilog's hierarchical and syntactic nuances.
● Efficiency: Adjusting gradient accumulation steps and batch sizes ensures efficient use
of computational resources.
● Generalization: Parameters like dropout and weight decay play a vital role in preventing
overfitting, especially when working with smaller or domain-specific datasets.
● Convergence Speed: A well-chosen learning rate scheduler can accelerate
convergence, reducing training time without sacrificing performance.

2.7.2 Variation in Hyperparameters

In this study, multiple hyperparameters were systematically varied during the fine-tuning process
to evaluate their effects on Verilog code generation. The selected hyperparameters and their
expected impacts are detailed below:

43
Hyperparameter Description Expected Impact

Higher values allow for effective training

Gradient Number of steps over which
with larger effective batch sizes,
Accumulation gradients are accumulated before
improving model stability and
Steps updating weights.
performance with limited memory.

Larger values can improve gradient

Per Device Train Batch size for training samples on estimation but may require more memory.
Batch Size each device. Smaller values may generalize better but
lead to noisier updates.

Per Device Eval Batch size for evaluation samples Determines the speed and memory
Batch Size on each device. efficiency of evaluation processes.

Dropout rate applied to the low- Prevents overfitting by introducing

LoRA Dropout rank adaptation (LoRA) layer regularization, especially useful when
during fine-tuning. fine-tuning on smaller datasets.

Helps prevent overfitting and ensures the

Regularization factor that penalizes
Weight Decay model does not become overly reliant on
large weights.
certain weights.

Impacts convergence speed and stability,

Strategy for adjusting the learning
Learning Rate with schedulers like linear decay ensuring
rate during training (e.g., linear
Scheduler Type gradual optimization and cosine
decay, cosine annealing).
annealing enabling periodic resets.

Affects how quickly or stably the model

Learning Rate Initial step size for weight updates. converges. Optimal learning rates
balance speed with convergence quality.

Determines the dimensionality of the

LoRA Rank Rank of the low-rank matrices
adaptation layer, impacting model
(LoRA R) used in LoRA fine-tuning.
flexibility and performance.

Controls the magnitude of updates from

LoRA Alpha Scaling factor for LoRA updates. the LoRA layers, influencing fine-tuning
effectiveness.

Table 9: Effects of Hyperparameters Tuning

44
2.8 Summary
This chapter has provided a comprehensive foundation for understanding the core concepts and
methodologies central to this thesis. The discussions ranged from the general capabilities and
architectures of large language models (LLMs) to the specific challenges posed by Verilog code
generation, emphasizing the interplay of dataset descriptions, fine-tuning strategies, evaluation
frameworks and hyperparameters tuning.

How These Foundational Ideas Inform Methodology and Experiments

The discussions in this chapter set the stage for the methodological approaches and experimental
designs presented in subsequent chapters:

1. Experimental Setup: The insights into dataset descriptions and fine-tuning strategies will
inform the creation and preprocessing of datasets, ensuring alignment with the research
objectives. Additionally, the comparative analysis of the four LLMs will be grounded in the
understanding of their architectural differences and domain-specific adaptations.

2. Evaluation Metrics: Building on the capabilities of VerilogEval, the methodology will

incorporate quantitative metrics such as accuracy and error rates, as well as qualitative
assessments like code modularity and readability, to provide a holistic evaluation of model
performance.

3. Optimization of Fine-Tuning: The overview of QLoRA and its advantages will guide the
tuning of hyperparameters during the model training phase, enabling the exploration of
different configurations to maximize the performance of each LLM.

4. Hypothesis Testing:
The hypotheses regarding the impact of dataset descriptions, the size of training dataset
and the comparative performance of LLMs will be tested through structured experiments,
leveraging the theoretical underpinnings discussed in this chapter.

45
3. Literature Review

3.1 Introduction
The purpose of this literature review is to situate the research within the broader context of existing
work on large language models (LLMs) and their application to code generation, specifically for
hardware description languages such as Verilog. The review synthesizes prior research on LLM
architectures, dataset preparation, fine-tuning methodologies, and evaluation frameworks,
providing a foundation for the experimental and analytical work undertaken in this thesis. By
understanding the current state of research, this chapter aims to identify gaps in the literature,
particularly concerning the influence of dataset descriptions and LLM architectures on domain-
specific language tasks like Verilog code generation.

Research on LLMs has evolved rapidly, with early models like GPT-3 [4] demonstrating
impressive capabilities in natural language understanding and code generation. Subsequent
advancements led to specialized models such as Codex [1] and CodeLlama [2], which focus on
programming tasks. However, these models are primarily trained on general-purpose
programming languages, leaving domain-specific applications underexplored.

Dataset preparation plays a critical role in the performance of LLMs, as demonstrated by

Hoffmann et al. [27], who emphasized the impact of dataset quality, diversity, and granularity on
model outputs. For Verilog, dataset granularity—ranging from high-level summaries to detailed
block-level descriptions—has yet to be systematically analysed. This thesis contributes to the field
by exploring how different levels of dataset detail affect the generation quality of Verilog code.

Fine-tuning techniques, such as LoRA [21] and QLoRA [28], have proven effective in adapting
LLMs to specific tasks while reducing computational overhead. These approaches have been
widely adopted in various applications but have not been thoroughly investigated for hardware
description languages. Additionally, evaluating generated Verilog code poses unique challenges,
as traditional programming evaluation metrics may not fully capture the correctness and efficiency
of hardware designs [3]. Tools like VerilogEval are critical for addressing this gap.

3.2 Large Language Models for Code Generation

The application of large language models (LLMs) to code generation has been extensively
explored in recent years, with promising outcomes in natural language-to-code tasks and program
synthesis. Early efforts focused on the evaluation of LLMs like GPT-3 for unseen programming
tasks, where they demonstrated a 29% success rate in generating fully correct code, showcasing
significant progress in handling diverse programming challenges [29].

Recent studies have proposed innovative methods to enhance the planning and reasoning
abilities of LLMs for code generation. Jiang et al. [30] introduced a "self-planning" mechanism,
enabling LLMs to decompose complex programming tasks into manageable steps before

46
generating solutions. This method resulted in more functionally accurate outputs and showed
promise in addressing intricate coding problems [30].

The robustness and reliability of LLMs in generating code snippets were evaluated by Zhong and
Wang [31], who analysed the impact of prompt engineering and pre-training on different
programming languages. Their findings underscored the need for structured prompts and diverse
pre-training datasets to improve the accuracy and efficiency of generated code [31].

Zhang et al. explored the use of LLMs for competitive programming tasks, emphasizing the
importance of integrating domain-specific datasets into the training process. Their work
demonstrated that domain-specific fine-tuning could significantly enhance the model's problem-
solving capabilities in niche areas [32].

Surveys on evaluating LLMs for code generation, such as those by Chen et al. (2024), highlighted
the diverse metrics used to assess model performance, including functional correctness, syntactic
accuracy, and computational efficiency. These surveys have provided valuable insights into the
benchmarks and methodologies critical for evaluating LLM capabilities [33].

Moreover, Ni et al. (2024) proposed a unified problem formulation for language-to-code tasks,
which served as a benchmark for evaluating the end-to-end capabilities of LLMs in generating
executable code from natural language instructions. This framework has set a new standard for
assessing code generation tools [34] .

3.3 Verilog Code Generation

The application of artificial intelligence (AI) to Verilog code generation has garnered increasing
attention, addressing the intricate challenges posed by hardware description languages (HDLs).
Verilog, a language pivotal for designing and modelling digital systems, necessitates strict
adherence to syntax, hierarchical design principles, and logical accuracy. Unlike general-purpose
programming languages, Verilog's inherent parallelism and module interdependencies present
unique obstacles for automated code generation systems.

Early explorations into AI-driven Verilog code generation involved fine-tuning pre-trained large
language models (LLMs) on Verilog-specific datasets. Thakur et al. (2023) fine-tuned LLMs using
Verilog code from GitHub and textbooks, resulting in models capable of producing syntactically
correct code 25.9% of the time. Notably, their fine-tuned open-source CodeGen model surpassed
the commercial Codex model in functional correctness, achieving a 6.5% success rate [35].

Advancements continued with the development of VeriGen, a specialized LLM for Verilog code
generation. Thakur et al. (2024) demonstrated that VeriGen, fine-tuned on curated Verilog
datasets, outperformed GPT-3.5-turbo by 1.1% in overall performance. The model exhibited a
41% improvement in generating syntactically correct Verilog code across various problem
categories compared to its pre-trained counterpart [36].

47
To further enhance the quality of generated Verilog code, Wang et al. (2024) introduced a
reinforcement learning approach with golden code feedback. Their model, VeriSeek, achieved
state-of-the-art results, outperforming existing models by a substantial margin. Notably, their 6.7
billion parameter model demonstrated superior performance compared to larger models,
highlighting the effectiveness of their approach [37].

A significant leap was made with the introduction of VerilogCoder by Ho et al. (2024). This system
employs multiple AI agents to autonomously generate Verilog code and rectify syntax and
functional errors. By integrating a novel task planner and an abstract syntax tree (AST)-based
waveform tracing tool, VerilogCoder achieved a 94.2% success rate in producing syntactically
and functionally correct Verilog code, surpassing previous methods by 33.9% on the VerilogEval-
Human v2 benchmark [38].

Despite these advancements, challenges persist in Verilog code generation. Many models
struggle to fully capture the parallelism and dependency structures inherent in Verilog. Moreover,
evaluation methods often rely on syntactic similarity metrics, which may not effectively measure
functional correctness. These limitations underscore the need for continued research into fine-
tuning strategies, dataset preparation, and evaluation frameworks tailored specifically for HDLs
like Verilog.

3.4 Effect of Different LLM Models in Code Generation

Numerous studies have explored the impact of different large language models (LLMs) on code
generation tasks, emphasizing their relative strengths, weaknesses, and use cases. A systematic
evaluation by Xu et al. (2022) compared the performance of LLMs like CodeParrot, Codex, and
GPT models on programming benchmarks, revealing significant disparities in the models' ability
to generate syntactically correct and functional code. The study highlighted that model size,
architecture, and pretraining data substantially influenced performance, with larger models
generally excelling in generating complex code snippets [39].

Another study by Du et al. (2024) evaluated LLMs in class-level code generation tasks. The
authors demonstrated that task-specific fine-tuning of models such as Codex and GPT-Neo
resulted in improved performance compared to zero-shot or few-shot approaches. The research
also emphasized that dataset alignment with the target domain could significantly affect the quality
of the generated code [40].

In terms of code generation efficiency, Liu et al. (2024) analysed how models like Codex and
CodeT5 performed under constrained computational resources. They observed that parameter-
efficient LLMs, when optimized with methods like QLoRA, produced results comparable to larger,
resource-intensive models, suggesting a promising direction for deploying LLMs in limited-
resource settings [41].

Furthermore, Jiang et al. (2024) introduced a self-planning code generation approach to compare
Codex with other LLMs like GPT-3.5. Their findings underscored Codex's superior ability to

48
integrate structured prompts into generating functional, error-free code, making it particularly
effective for complex programming tasks [30].

Another critical comparison by Tihanyi et al. (2025) investigated the security of code generated
by different LLMs. The study revealed variations in the generated code's vulnerability levels, with
GPT-based models often generating safer outputs than smaller, task-specific LLMs. This research
highlighted the need for robust evaluation metrics to assess not only functional correctness but
also the security of generated code [42].

Overall, these studies collectively illustrate that the choice of LLM significantly affects code
generation quality. Factors such as model architecture, pretraining data, fine-tuning techniques,
and task-specific adaptations are pivotal in determining their performance across various
benchmarks and domains.

3.5 Fine-Tuning Techniques in Code Generation

Fine-tuning techniques are crucial for adapting pre-trained large language models (LLMs) to
specialized tasks like code generation. Recent research has explored both parameter-efficient
fine-tuning (PEFT) methods and traditional full-model fine-tuning approaches, each offering
distinct benefits and trade-offs.

Weyssow et al. (2024) compared PEFT techniques such as LoRA (Low-Rank Adaptation) with
full fine-tuning for code generation. Their findings demonstrated that LoRA was effective in
reducing computational overhead while maintaining comparable performance in generating
functionally correct code [43]. Similarly, Ma et al. (2024) introduced a framework called LLaMoCo
that leveraged instruction tuning to optimize LLMs for code generation tasks, achieving significant
improvements in the quality of generated outputs [44].

Studies by Li et al. (2024) explored fine-tuning techniques for secure code generation, comparing
models fine-tuned with domain-specific datasets against those trained on general-purpose data.
They concluded that task-specific fine-tuning led to more secure and syntactically robust code
[45]. Haider et al. (2024) focused on prompt engineering as a complementary strategy to fine-
tuning, showing that well-designed prompts could enhance LLM performance even without
extensive retraining [46].

An empirical study by Storhaug and Li (2024) challenged the prevailing notion that LoRA was less
effective than full fine-tuning. Their experiments indicated that PEFT methods, including LoRA,
could outperform full fine-tuning in unit test generation and related code tasks, particularly when
computational resources were constrained [47].

Another approach explored by Poesia et al. (2022) involved fine-tuning pre-trained models like
GPT-3 and Codex on natural language descriptions for generating SQL and similar structured
code. Their results emphasized the importance of high-quality datasets and task-specific
pretraining to maximize model efficiency [48].

49
These findings collectively highlight that fine-tuning techniques must be tailored to the specific
requirements of the code generation task. Innovations such as LoRA and instruction tuning have
reduced the resource intensity of fine-tuning while ensuring performance improvements across
diverse programming languages and domains.

3.6 Research Gap

While the existing literature has made significant progress in advancing the capabilities of large
language models (LLMs) for code generation, several critical research gaps remain unaddressed,
particularly in the domain of hardware description languages like Verilog. These gaps are outlined
below:

1. Dataset Design and Granularity

Many studies have emphasized the importance of dataset quality and alignment in fine-tuning
LLMs for programming tasks. However, the impact of dataset granularity, such as global, module,
and block-level descriptions, on the performance of fine-tuned models remains insufficiently
explored. This gap limits our understanding of how different dataset designs influence the ability
of models to capture structural and contextual details in Verilog code.

2. Evaluation Metrics and Tools

Existing evaluation frameworks, such as HumanEval, primarily assess the functional correctness
of generated code using metrics like pass@k, which measures the probability that at least one of
the top k generated samples passes all unit tests. However, these frameworks do not incorporate
domain-specific metrics tailored for Verilog, such as hardware resource efficiency (e.g., area,
delay, and static power) and code modularity. The introduction of benchmarks like VerilogEval
aims to address this gap by providing a dataset of Verilog designs, enabling a more
comprehensive evaluation of code generation models for hardware implementations.

3. Model Comparison for Verilog Code Generation

Although general-purpose models like Codex and GPT-3.5 have been benchmarked extensively
for programming tasks, specific comparisons of LLMs tailored for hardware description languages
like Verilog are rare. Models such as CodeLlama, Qwen, CodeGemma and DeepSeek Coder lack
dedicated evaluations for scalability, adaptability, and performance in Verilog-specific tasks. This
gap limits our understanding of their effectiveness in this specialized domain.

Addressing Research Gaps in This Thesis

Recognizing these gaps, this thesis aims to provide targeted solutions to advance the state of
Verilog code generation using LLMs. This thesis not only bridges the identified research gaps but
also establishes a foundation for further exploration of LLMs in hardware description languages.
The outcomes will guide the development of more effective dataset designs, evaluation
frameworks, and tailored LLMs for Verilog and other specialized programming domains.

50
4. Implementation

4.1 Introduction
The methodology chapter serves as the foundation of this research, detailing the processes, tools,
and techniques employed to investigate the impact of dataset descriptions and different large
language models (LLMs) on Verilog code generation. This chapter outlines the dataset
preparation, experimental setup, fine-tuning strategies, and evaluation framework used to
address the research questions presented in Chapter 1.

The central aim of this study is to fine-tune four advanced LLMs—CodeLlama 7B Instruct, Qwen
2.5 Coder Instruct, DeepSeek Coder 6.7B Instruct, and CodeGemma 7B Instruct—on
datasets with varying levels of granularity to generate high-quality Verilog code. Each of these
models represents state-of-the-art advancements in AI-driven code generation, and their
comparison provides valuable insights into their adaptability to domain-specific languages like
Verilog.

The chapter begins by detailing the preparation of datasets used in training and evaluation,
focusing on four distinct types of dataset descriptions: High-Level Summaries, Detailed-Level
Summaries, Block-Level Summaries, and Verilog Pyramid of Thoughts (PoT) Summaries.
Each description provides varying levels of abstraction and granularity, enabling a comprehensive
analysis of their impact on model performance.

Next, the process of fine-tuning the selected LLMs is described, employing the QLoRA
methodology to efficiently adapt the models to the specific requirements of Verilog code
generation. The fine-tuning process includes configuring hyperparameters, optimizing
computational resources, and utilizing appropriate training techniques. Here we also discussed
the experimental setup, including the hardware and software configurations that ensure
reproducibility and scalability

Finally, this chapter discusses the evaluation framework, centred around the VerilogEval tool, is
also presented, highlighting the metrics used to assess the generated code’s correctness,
syntactic validity, and functional accuracy. This evaluation serves as the basis for comparing both
the dataset descriptions and the performance of the different models.

By systematically presenting these methodologies, this chapter provides a transparent and

replicable framework for understanding the dynamics of dataset descriptions and LLM
architectures in Verilog code generation. The insights derived from this approach form the basis
for the results and analysis discussed in the subsequent chapter.

51
4.2 Experimental Setup
The experimental setup for this thesis involves four distinct phases: dataset preparation, model
fine-tuning, model evaluation, and inference. The flowchart provided in Figure 21 outlines the
overall experimental workflow, illustrating the interconnected processes and tools employed in
each phase.

Figure 21: Overall Experimental Setup for the Research

4.2.1 Dataset Preparation

The first stage focuses on preparing the datasets for fine-tuning and evaluation. The MG-Verilog
dataset, consisting of Verilog code examples, forms the basis for this phase. The raw dataset
undergoes preprocessing to ensure compatibility with large language models. The dataset is then
categorized into four types of descriptions High-Level Summaries Dataset, Detailed-Level
Summaries Dataset, Block-Level Summaries Dataset, and Verilog Pyramid of Thoughts (PoT)
Dataset. After categorization, the dataset is split into training and validation subsets, ensuring
balanced representation across all description types.

4.2.2 Model Fine-Tuning

The fine-tuning phase adapts pre-trained LLMs for Verilog code generation tasks. The four LLMs
under evaluation—CodeLlama, Qwen, DeepSeek and CodeGemma—are fine-tuned using the
QLoRA (Quantized Low-Rank Adaptation) method. This process leverages the following steps:

1. Fine-Tuning Using QLoRA:

52
  QLoRA enables efficient parameter optimization by updating low-rank adapters instead of
the entire model.
 The approach reduces computational overhead while maintaining performance.
2. Hyperparameter Tuning:
 Key hyperparameters such as learning rate, batch size, and number of training epochs
are iteratively optimized to achieve the best results.
3. Loss Monitoring:
 The training and validation losses are monitored to prevent overfitting and ensure
generalizability.
4. Checkpoint Saving:
 Only the fine-tuned LoRA layers are saved, minimizing storage requirements while
preserving the model's domain-specific capabilities.

4.2.3 Model Evaluation

The fine-tuned models are evaluated using the VerilogEval framework. This phase involves:

1. Feeding the fine-tuned models with three types of prompts: high-level, detailed-level, and
block-level prompts to generate Verilog code.
2. Assessing the generated code for syntax correctness, functional validity, and adherence to
Verilog design principles using VerilogEval.
3. Metrics used for evaluation include:
 Pass@1: The percentage of correct outputs on the first attempt.
 Pass@5 and Pass@10: Success rates within the top 5 and 10 attempts, respectively.

This systematic evaluation ensures that the models' outputs align with the requirements of real-
world hardware design tasks.

4.2.4 Inference

In the final phase, the fine-tuned models are deployed to generate Verilog code based on user-
provided prompts. The workflow for inference involves:

1. Feeding a user prompt into the fine-tuned model.

2. Generating the final Verilog code output, leveraging both the pre-trained LLM layers and the
fine-tuned LoRA layers for optimal performance.

4.3 Dataset Preparation

The dataset used in this study was sourced from the publicly available MG-Verilog dataset, hosted
on Hugging Face’s Datasets repository (GaTech-EIC/MG-Verilog) [9]. Specifically designed to
enhance the performance of large language models (LLMs) in hardware design tasks, this dataset
focuses on Verilog code generation and includes multi-grained descriptions at various levels of
detail. These descriptions range from high-level summaries to detailed block-level annotations

53
paired with corresponding Verilog code samples. Rigorous preprocessing and transformation
processes were implemented to ensure the dataset’s quality, relevance, and usability for fine-
tuning the selected LLMs.

4.3.1 Dataset Description

In this research, we employ four distinct datasets derived from the MG-Verilog dataset to fine-
tune the large language models (LLMs) under study. These datasets are meticulously prepared
to explore the effects of varying levels of granularity in descriptions on Verilog code generation.
Each dataset represents a unique approach to summarizing and structuring the information within
the MG-Verilog dataset, tailored to enhance the performance of the models. The datasets are as
follows:

1. High-Level Global Summaries Dataset

 This dataset comprises 11,100 rows, each pairing Verilog code outputs with
corresponding high-level descriptions that deliver a concise and overarching summary of
the functionality of each module.
 These summaries are abstract in nature and are intended to convey the overall design
purpose without delving into intricate details.

2. Detailed Global Summaries Dataset

 Consisting of 11,100 rows, this dataset offers Verilog code outputs with corresponding
comprehensive descriptions that detail the structural and functional components of the
Verilog code at a global level.
 These summaries aim to bridge the gap between abstract high-level summaries and fine-
grained block-level descriptions, providing a complete view of the module’s architecture.

3. Block-Level Summaries Dataset

 This dataset includes 11,100 rows of Verilog code outputs with block-level summaries,
focusing on detailed explanations of individual sections within each Verilog module.
 Block-level summaries capture the specifics of each module's implementation,
emphasizing localized design elements.

4. Verilog Pyramid-of-Thoughts (PoT) Description Dataset

 The PoT dataset, comprising 44,400 rows, integrates all the above summary types into a
unified dataset.
 The dataset contains:
■ High-Level Global Summaries.
■ Detailed Global Summaries.
■ Block-Level Summaries.
■ Combination of High-Level and Block-Level summaries

54
  This dataset embodies a hierarchical organization that mimics the "pyramid of thoughts"
approach, progressively combining different levels of detail to provide a holistic training
resource.

The structured diversity of these datasets is designed to evaluate the performance of LLMs in
generating Verilog code from varying levels of abstraction. By utilizing these datasets, this study
aims to establish best practices for fine-tuning LLMs in domain-specific applications like hardware
design.

4.3.2 Data Preprocessing Pipeline

A structured preprocessing pipeline was developed to generate the datasets required for fine-
tuning. This pipeline facilitates the creation of four distinct datasets essential for our research,
derived from the MG-Verilog dataset. Key stages of the pipeline include:

1. Cleaning Summaries:
 Removed unnecessary system messages and instructions embedded in the dataset, such
as syntax-related prompts.
 Extracted meaningful content from high-level, detailed and block-level summaries while
ensuring no loss of critical information.

2. Updating Summaries:
 Replaced generic instructions with task-specific prompts to improve Verilog code
generation accuracy.
 Ensured consistent phrasing across high-level, detailed, and block-level descriptions to
improve contextual alignment.

3. Combining Description Fields:

 Merged the High-Level and Block-Level summaries into a single cohesive description field
for PoT dataset.
 Created composite representations of each code sample’s descriptive elements to
enhance learning potential.

4. Transformation and Augmentation:

 Split the MG-Verilog dataset into four individual datasets: PoT, High Level, Detailed level
and Block Level.
 For PoT dataset, the MG-Verilog dataset is split into individual entries where each row
corresponds to a specific description level (block-level, detailed-level, or global-level
summaries) and later the combined summaries that is done before is added.
 Augmented the PoT dataset by creating multiple representations of the same code sample
with different levels of granularity.

5. Shuffling and Finalization:

 Shuffled the four datasets to ensure randomness during training, reducing overfitting and
improving generalization.

55
  Saved all four datasets in Arrow file format for efficient storage and processing.

4.3.3 Dataset Splits

The processed dataset was divided into two sets:

● Training Set: Comprising the 95% of samples, used for fine-tuning the LLMs.
● Validation Set: Employed to monitor model performance and prevent overfitting during
training.

Balanced representation across all description types was maintained in these splits.

4.3.4 Challenges in Dataset Preparation

Several challenges were encountered during the dataset preparation process:

● Cleaning and restructuring large-scale textual data while retaining essential information.
● Integrating different description levels without introducing redundancy.
● Ensuring balanced representation across training, and validation sets to reflect dataset
complexity and diversity.

These challenges were addressed through a carefully designed preprocessing pipeline and
iterative testing, ensuring the dataset’s quality and integrity for subsequent analysis.

4.4. Model Fine-Tuning

Fine-tuning is a critical process in adapting pre-trained large language models (LLMs) to domain-
specific tasks, such as Verilog code generation. By leveraging fine-tuning, we refine the models'
capabilities to understand and generate Verilog code based on task-specific datasets and
objectives. This study employs fine-tuning to customize four state-of-the-art LLMs—CodeLlama
7B Instruct, Qwen 2.5 Coder 7B Instruct, CodeGemma 7B Instruct and DeepSeek Coder 6.7B
Instruct—for efficient and accurate Verilog code generation.

The fine-tuning process in this research is specifically designed to explore the effect of dataset
granularity or description levels and Model architecture. Four distinct datasets, derived from the
MG-Verilog dataset, are used for this purpose:

1. High-Level Global Summaries Dataset

2. Detailed Global Summaries Dataset
3. Block-Level Summaries Dataset
4. Verilog Pyramid-of-Thoughts (PoT) Summaries Dataset

The goal of this fine-tuning is twofold:

56
 ● Domain Adaptation: Tailoring the LLMs to the unique characteristics of Verilog, a
hardware description language with specific syntactic and semantic requirements.
● Performance Optimization: Enhancing the models’ ability to generate syntactically
correct and semantically meaningful Verilog code across different levels of abstraction.

To achieve these objectives, the fine-tuning process integrates several state-of-the-art

methodologies and tools:

● QLoRA (Quantized Low-Rank Adaptation) is utilized to enable efficient training by

reducing memory overhead through 4-bit quantization, while still maintaining model
performance.
● Custom hyperparameters are configured for each dataset to optimize training. This
includes adjustments to learning rates, batch sizes, gradient accumulation steps, and
LoRA-specific parameters like lora_r and lora_alpha.
● The training workflow is implemented using Hugging Face Transformers, PEFT
(Parameter-Efficient Fine-Tuning), and Seq2SeqTrainer, enabling scalable and
reproducible experimentation.

The fine-tuning process is systematically designed to adapt the LLMs to handle varying levels of
detail in dataset descriptions. This step is vital for understanding how dataset granularity affects
model performance, providing a foundation for the evaluation and analysis discussed in
subsequent sections. By optimizing the models with domain-specific datasets and advanced
training techniques, this study aims to push the boundaries of LLMs in generating hardware
description language code, specifically Verilog.

4.4.1 Dataset-Specific Training Arguments

Fine-tuning for each dataset was conducted using tailored training arguments or hyperparameters
and training configurations, ensuring the models adapted effectively to the specific granularity of
dataset descriptions. Below is the breakdown of the dataset-specific final hyperparameters that
have been found after lots of tuning.

The following hyperparameters were used for fine-tuning the High-Level Global Summaries,
Detailed Global Summaries, and Block-Level Summaries datasets:

57
 Figure 22: Hyperparameters used for High-Level Global Summaries, Detailed Global
Summaries, and Block-Level Summaries datasets

Detail Description of the Hyperparameters used:

1. Optimizer: paged_adamw_32bit was employed to enhance memory efficiency during

training while maintaining the benefits of the AdamW optimizer.
2. Batch Sizes: Both training and evaluation batch sizes were set to 1 sample per device,
considering the smaller dataset size.
3. LoRA Configuration:
 lora_r: Set to 16 to control the rank of low-rank adaptation.
 lora_alpha: Set to 32 to scale the LoRA layers effectively.
 lora_dropout: Configured at 0.6 to mitigate overfitting.
4. Weight_decay: Set to 0.5 to mitigate overfitting.
5. Learning Rate: Set to 0.0001 with a constant learning rate scheduler for stable training.
6. Precision: BF16 precision was used for numerical stability, and models were quantized to 4
bits for memory efficiency.
7. Epochs: Fine-tuning spanned over 15 epochs to ensure sufficient learning from the dataset.
8. Gradient Accumulation: Steps were set to 8, allowing the effective batch size to increase
without requiring additional memory.

The following hyperparameters were used for fine-tuning the Verilog Pyramid-of-Thoughts
(PoT) Dataset

58
 Figure 23: Hyperparameters used for Verilog Pyramid-of-Thoughts (PoT) Dataset

Detail Description of the Hyperparameters used:

1. Optimizer: paged_adamw_32bit remained the choice for its efficiency in handling larger
datasets.
2. Batch Sizes: Training and evaluation batch sizes were increased to 2 samples per device,
due to larger size of the PoT dataset.
3. LoRA Configuration:
 lora_r: Increased to 64 to accommodate the complexity of combined dataset
descriptions.
 lora_alpha: Set to 128 for scaling, allowing the model to capture nuanced details across
summary levels.
 lora_dropout: Disabled (0.0), as the dataset's size and diversity reduce overfitting
risks.
4. Weight_decay: Also kept Disabled (0.0), as the dataset's size and diversity reduce
overfitting risks.
5. Learning Rate: Elevated to 0.0002 for faster convergence given the larger dataset size.
6. Precision: BF16 precision paired with 4-bit quantization ensured efficient memory use.
7. Epochs: Training was conducted over 15 epochs, consistent with other datasets.
8. Gradient Accumulation: Increased to 16 steps to manage the effective batch size for
computational efficiency.

4.4.2 Fine-Tuning Framework

The fine-tuning framework implemented in this study is designed to adapt pre-trained large
language models (LLMs) to the task of Verilog code generation using our datasets. The
framework employs efficient techniques, including QLoRA (Quantized Low-Rank Adaptation)

59
and LoRA (Low-Rank Adaptation) modules, to fine-tune models while minimizing computational
and memory overhead. Below is a detailed breakdown of the framework, illustrated with relevant
code snippets and descriptions.

1. Model Preparation

The process begins by loading the pre-trained model and tokenizer using Hugging Face's
AutoModelForCausalLM and AutoTokenizer classes. The models are prepared for QLoRA
fine-tuning by applying 4-bit precision quantization.

● Model Loading: The AutoModelForCausalLM function initializes the pre-trained model

from the Hugging Face Hub or a local directory.
● Quantization: The BitsAndBytesConfig module enables the model for QLoRA fine-tuning
by applying 4-bit precision quantization which reduces memory usage and computational
requirements, making it feasible to fine-tune large models on limited hardware.
● Device Mapping: The device_map="auto" argument ensures the model is distributed
across available GPUs or CPUs for optimal performance.
● Precision: The torch_dtype argument ensures the model is configured to use FP16 or BF16
precision, depending on hardware support, enhancing computational efficiency.

2. Tokenizer Initialization

The tokenizer is configured to handle Verilog-specific tokens and special characters effectively.

60
● Tokenizer loading: The AutoTokenizer class initializes a tokenizer tailored for the selected
model.
● Special Token Addition: Custom tokens are added to handle Verilog-specific syntax, such
as <s> (start of sequence), </s> (end of sequence), and [PAD] (padding).
● Embedding Resizing: After adding special tokens, the model's token embeddings are
resized to match the expanded tokenizer vocabulary, ensuring the model can recognize and
utilize the newly added tokens.

3. LoRA Configuration

LoRA modules are integrated into the model to enable efficient parameter tuning.

● r (Rank): Specifies the complexity of modifications applied to the model. A lower rank
represents fewer adjustments, which conserves computational resources and training time.
● lora_alpha (Scaling Factor): Controls the magnitude of adjustments applied to the model.
A higher lora_alpha value allows for significant updates, which can enhance model
performance but may also increase the risk of overfitting.

61
● lora_dropout: A dropout is applied to prevent overfitting by randomly ignoring x% (0.5 for
50%) of connections in the LoRA layers during training, improving the model's ability to
generalize.
● bias: Set to "none", indicating that no additional bias is incorporated into the LoRA layers
during training. While bias can sometimes aid in controlling neuron outputs, it is not utilized in
this configuration.
● target_modules: This argument specifies the linear layers where LoRA is applied. The
find_all_linear_names function identifies linear layers in the model where LoRA will be
applied. These layers are computationally intensive and represent key targets for adaptation.
These layers are determined based on the architecture of the model
● task_type: Defined as "CAUSAL_LM", aligning with the study's goal of fine-tuning models
to understand and generate Verilog code effectively.
● The get_peft_model function integrates LoRA modules into the pre-trained model,
enabling fine-tuning with minimal updates to the model’s parameters. This integration ensures
efficient parameter tuning, conserving both computational resources and time.

4. Gradient Checkpointing

● Gradient Checkpointing: Enabled to save memory by recomputing certain parts of the

computational graph during backpropagation. This feature is especially useful when fine-
tuning large models like the 7B parameter variants.

5. Batch Processing and Tokenization

The DataCollatorForCausalLM class prepares the training data by batching sequences and
padding them to uniform lengths. It ensures that sequences are correctly batched with padding
tokens to handle variations in input length, which is critical for stable training. This setup
guarantees compatibility across all datasets while maintaining alignment with the pre-trained
tokenizer and model configurations.

62
63
● Source and Target Sequences: The sources and targets arrays tokenize the input and
output text respectively, adding the <s> and </s> tokens.
● Padding: Input sequences are padded to the maximum sequence length to ensure uniformity
across batches.
● Batch Preparation: Tokenized and padded sequences are returned as a batch dictionary and
are ready for training.

6. Training Setup

The Seq2SeqTrainer manages the training process, including gradient accumulation,

optimization, and checkpointing.

● Dataset and Data Collator: The training and evaluation datasets, along with the data collator,
are provided to the trainer.
● Training Arguments: Hyperparameters such as batch size, gradient accumulation steps, and
learning rate are passed via the training_args configuration.

7. Trainer

● Training Execution: The [Link]() method initiates the fine-tuning process. If a

previous checkpoint is available, it resumes training from that point using the

64
 resume_from_checkpoint parameter. This ensures no progress is lost in case of
interruptions.
● Metrics Logging: After the training run is complete, performance metrics, such as training
loss and evaluation accuracy, are extracted from the train_result object and logged using
the trainer.log_metrics() method.
● Metrics Saving: The logged metrics are saved to disk with trainer.save_metrics() for
later analysis.
● State Saving: The trainer.save_state() method saves the current state of the model
and training environment, ensuring reproducibility and facilitating debugging or subsequent
training sessions.

4.4.3 Checkpointing and Model Saving

Checkpointing and model saving are integral components of the fine-tuning process, ensuring
that training progress is preserved and that intermediate and final models can be reused for
evaluation or further training. In the implemented fine-tuning pipeline, these processes are
handled systematically, with a focus on saving both the model state and the PEFT (Parameter-
Efficient Fine-Tuning) adapters to optimize storage and reusability.

1. Saving PEFT Checkpoints

This custom callback class, SavePeftModelCallback, is implemented to manage the saving

of fine-tuned models during and after training. It focuses specifically on saving the PEFT adapters,
which are the trainable parts of the model, rather than the entire model. This approach significantly
reduces storage requirements while ensuring that the fine-tuned components are preserved.

65
● Saving During Training (on_save):
○ During training, the model is saved at each 500 steps as defined in the training arguments.
○ The save_model method saves the fine-tuned PEFT adapters to a specified checkpoint
folder.
○ If a best_model_checkpoint exists (indicating the best-performing checkpoint), it is
prioritized for saving. Otherwise, a new checkpoint folder is created based on the current
training step (global_step).

● Final Checkpoint (on_train_end):

○ At the end of training, the method ensures that the final state of the adapters is saved to a
checkpoint.
○ A completed file is created in the output directory to indicate that the training process has
been finalized successfully.

● Optimized Storage:
○ After saving the PEFT adapters, the full model file (pytorch_model.bin) is removed to
save disk space, retaining only the necessary fine-tuned components.

2. Checkpoint Management

This function, get_last_checkpoint, manages the resumption of training from previous

checkpoints and ensures continuity in case of interruptions.

66
● Completed Training Check:
○ Checks if a completed file exists in the checkpoint directory, indicating that the training
has already been finalized.

● Finding the Last Checkpoint:

○ Searches the checkpoint directory for the latest checkpoint by identifying folders named
in the format checkpoint-<step> and extracting the step number.
○ The checkpoint with the highest step number is returned as the most recent state.

● Resumption:
○ If a checkpoint is found, it is loaded, allowing training to resume from the saved state.

4.4.4 Logging and Monitoring

Logging and monitoring are integral components of the fine-tuning process, providing real-time
feedback on model performance and helping track training progress. In this research, Weights &
Biases (wandb) is employed for comprehensive experiment tracking. This section describes how
the code facilitates logging and monitoring during fine-tuning.

● WANDB_PROJECT: Specifies the project name under which all training runs are logged. This
ensures organized storage and visualization of logs for different experiments.

67
● WANDB_LOG_MODEL: Configures wandb to save model checkpoints, enabling easy retrieval
of trained models directly from the wandb platform.
● WANDB_WATCH: Disables real-time logging of model gradients and parameters to improve
logging speed, particularly beneficial for resource-intensive tasks.

4.5 Model Evaluation

The evaluation framework for this research builds upon the VerilogEval benchmarking framework,
which is specifically designed to assess the capability of Large Language Models (LLMs) in
generating functionally correct Verilog code. VerilogEval provides a standardized methodology
for evaluating LLM-generated Verilog modules using functional correctness metrics through
automated simulations. The framework employs testbenches and a sandboxed environment to
ensure reliable and reproducible evaluation results.

Evaluation Process Overview

Figure 24: Overview of Evaluation Process

The evaluation begins with:

1. Generating Verilog code using trained LLMs based on task descriptions.

2. Validating the generated code for functional correctness against predefined test cases
using simulation tools like Icarus Verilog.
3. Aggregating results to compute metrics like pass@k.

4.5.1 Evaluation Datasets

68
 1. Task Description Dataset

The task description dataset is a foundational element of the evaluation framework, providing the
natural language instructions necessary for LLMs to generate Verilog code. This dataset is
organized into three distinct levels of task descriptions, each offering varying degrees of
granularity:

a. High-Level Descriptions:
 Provide an overarching summary of the Verilog module’s functionality.
 Focus on the purpose and general behaviour of the module without delving into
implementation details.
 Example:
"The Verilog module is a multiplexer that selects one of two input signals based on a
control signal and outputs the selected signal."

b. Detailed-Level Descriptions:
 Offer a comprehensive explanation of the module, including its inputs, outputs, and logical
behavior.
 Serve as a bridge between high-level concepts and specific implementation details.
 Example:
"The Verilog module is a 2-to-1 multiplexer with inputs a and b, a select signal sel, and
an output out. When sel is 0, the output is assigned the value of a; when sel is 1, the
output is assigned the value of b."

c. Block-Level Descriptions:
 Provide fine-grained details about individual components or blocks within the Verilog
module.
 Include information about logical operations, specific blocks, and interconnections.
 Example:
"The multiplexer contains two AND gates and one OR gate. Each input signal is ANDed
with the complement of the select signal (sel) and the select signal, respectively. The
outputs of the AND gates are combined using an OR gate to produce the final output
signal."

We will choose any one of the description datasets and proceed with our evaluation. Below is a
sample of a detailed level description dataset:

{
"task_id": "review2015_fsmonehot",
"detail_description": "The Verilog module is a finite state machine (FSM) used for one-hot
encoding. It has five inputs (d, done_counting, ack, state) and seven outputs (B3_next, S_next,
S1_next, Count_next, Wait_next, done, counting, shift_ena)..."
}

69
Where,
● task_id: Unique identifier for the task.
● detail_description: A clear description of the Verilog module, its inputs, outputs, and
functionality.

2. VerilogEval Evaluation Dataset

The dataset provides three essential components for evaluating Verilog code. Module Headers
define the inputs, outputs, and structure of the Verilog module, serving as a blueprint for both the
generated and reference modules. Canonical Solutions act as the reference implementations,
offering a reliable standard for comparison with the generated Verilog code to validate functional
correctness. Finally, Testbenches are designed to simulate the modules under various input
conditions, enabling the systematic testing of functionality and the detection of discrepancies
between the reference and generated modules. These components collectively ensure a
comprehensive and structured evaluation process.

{
"task_id": "mux2to1v",
"prompt": "module top_module (\n\tinput [99:0] a,\n\tinput [99:0] b,\n\tinput sel,\n\toutput [99:0]
out\n);\n",
"canonical_solution": "\n\tassign out = sel ? b : a;\n\t\nendmodule\n",
"test": "`timescale 1 ps/1 ps\n`define OK 12\n`define INCORRECT 13\nmodule
reference_module (...)"
}

Where,
● prompt: Module header specifying the input and output signals.
● canonical_solution: A golden solution for the task.
● test: Verilog testbench for simulation.

4.5.2 Step-by-Step Evaluation Process

1. Preparing Prompts for code generation

Prompts are constructed using task descriptions from the Task Description Dataset and module
headers from the VerilogEval Evaluation Dataset. The VerilogDataset class handles this
process.

70
● prompts: Each task is transformed into a structured input format.
● Dynamic Prompt Construction: Fills placeholders in PROMPT_BASELINE with
description and module_header.

2. Generating Verilog Code

Prompts are fed into the fine-tuned LLM to generate Verilog code.

● [Link]: Generates Verilog code using the LLM.

● Parameters:
○ max_new_tokens: Limits the length of the generated code.
○ Temperature, top_k, and top_p: Control randomness and diversity in generation.
○ Repetition_penalty: Discourages the model from repeating the same token or
phrase excessively

71
3. Compile the Testbench

The generated Verilog code replaces the placeholder module in the testbench. Once the
testbench is complete, it is compiled using Icarus Verilog. The compilation ensures that:

1. Syntax errors in the testbench, reference module, or generated module are identified.
2. A simulation executable (.vvp file) is created for the next step.

Command:

Where

● -Wall: Enables all warnings for better debugging.

● -g2012: Ensures compatibility with SystemVerilog-2012 features.
● -s tb: Specifies the top-level module (testbench) to be simulated.
● -o [Link]: Produces an executable file ([Link]) for simulation.

4: Simulate the Testbench

The compiled testbench is executed using vvp, which runs the simulation and produces output
logs.

Command:

vvp -n [Link]

Where

● -n: Suppresses interactive mode, allowing batch execution.

During simulation:

1. The testbench applies a sequence of inputs to both the reference (out_ref) and
generated (out_dut) modules.

The outputs are compared at each time step using the testbench's verification logic:

assign tb_match = (out_ref === out_dut);

wire tb_mismatch = ~tb_match;

5. Parsing Simulation Results

72
The simulation output is analysed to determine correctness. The simulation produces logs
indicating whether the generated module matches the reference module.

Where

● Extracts the number of mismatches and total test samples.

● A task is marked as passed if there are zero mismatches.

6. Calculate Metrics (Pass@k)

The pass@k metric measures the likelihood that at least one of the top k generated samples is
correct.

Where

○ num_samples: Number of samples tested.

○ num_correct: Number of samples that passed all tests.
○ k: List of k-values for which the metric is calculated (e.g., 1, 5, 10).

73
Pass@k Formula:

○ n: Total samples generated.

○ c: Number of correct samples.
○ k: The number of samples considered.

Output Example:

○ pass@1=1.0\text{pass@1} = 1.0 (100% success for the top-1 sample).

○ pass@5=0.8\text{pass@5} = 0.8 (80% success for the top-5 samples).

4.6 Summary
In this chapter, we outlined the methodology used to explore the impact of dataset descriptions
and LLM architectures on Verilog code generation. The chapter began with an overflow of the
experimental setup, highlighting the independent variables (dataset description types and LLM
models) and dependent variables (code accuracy, functional correctness, and computational
efficiency). The experimental design was aligned with the research objectives to systematically
evaluate the performance of four LLMs—CodeLlama 7B Instruct, Qwen 2.5 7B Coder 7B Instruct,
CodeGemma 7B Instruct and DeepSeek Coder 6.7B Instruct—using four distinct dataset
descriptions: Verilog PoT summaries, global-level summaries, detailed-level summaries, and
block-level summaries.

The dataset preparation process was described, emphasizing the importance of granularity and
diversity in the data. The datasets were pre-processed and categorized to create structured inputs
for fine-tuning, ensuring compatibility with the unique requirements of Verilog. The methodological
section further detailed the fine-tuning process, with a focus on using the QLoRA technique to
achieve parameter-efficient adaptation of LLMs. Hyperparameter tuning was also addressed,
describing the variations applied during training to optimize model performance. Lastly, For
evaluation, the VerilogEval library was identified as a critical tool for assessing the syntax,
functional correctness, and structural quality of the generated Verilog code.

This chapter sets the foundation for the results and analysis presented in the subsequent chapter,
where the outcomes of the experiments are discussed in detail, offering insights into the interplay
between dataset descriptions, training dataset size and LLM architectures in Verilog code
generation.

74
5. Results and Discussion

5.1 Introduction
This chapter presents the results of the experimental evaluation of large language models (LLMs)
for Verilog code generation and discusses their implications in addressing the research questions
and objectives. The primary focus is to analyse the impact of dataset descriptions and model
architectures on the quality and accuracy of the generated Verilog code. By integrating the results
and discussions into a single chapter, the insights derived from this research are thoroughly
contextualized within the broader scope of AI-driven hardware design automation.

The experiments conducted involved four state-of-the-art LLMs: CodeLlama 7B Instruct, Qwen
2.5 Coder Instruct, DeepSeek Coder 6.7B Instruct, and Codegemma 7B Instruct. These
models were fine-tuned using the QLoRA (Quantized Low-Rank Adaptation) method, which
enabled efficient fine-tuning on task-specific datasets while minimizing computational resource
requirements. The evaluation of the generated Verilog code was performed using VerilogEval, a
specialized tool designed to assess the syntactical validity, functional correctness, and overall
quality of Verilog code. This comprehensive evaluation framework ensured a robust comparison
of the models and datasets.

The datasets used for fine-tuning were prepared with four distinct levels of description:

1. Verilog Pyramid of Thoughts (PoT), containing modular and global-level insights.

2. Global-Level Summaries, which provided high-level overviews of the Verilog code
structure and functionality.
3. Detailed-Level Summaries, which included fine-grained descriptions of the code.
4. Block-Level Summaries, focusing on individual blocks and their functionalities in
isolation.

This study aims to explore how these dataset types influence the learning process of LLMs,
especially in understanding the hierarchical and modular nature of Verilog. Additionally, the
research evaluates the differences in performance among the three LLMs, highlighting the impact
of architectural and pretraining differences on their ability to handle domain-specific tasks.

The structure of this chapter is as follows:

● Results: Presenting experimental results found with a focus on the influence of dataset
descriptions and model architecture.
● Discussion: Interpreting the results in light of the research questions and providing
insights into the challenges, limitations, and broader implications of the findings.

This chapter aims to bridge the gap between the experimental outcomes and their practical
applications, contributing to the advancement of LLMs in generating Verilog code and setting a

75
foundation for future work in this domain. By integrating the results and their interpretations, this
section provides a comprehensive understanding of the dynamics between dataset descriptions,
model architectures, and the fine-tuning methodology.

5.2 Influence of Dataset Descriptions on Model Performance

5.2.1 Results

The results presented in the below tables demonstrate how LLM fine-tuned with different levels
of dataset description influences the performance of Verilog code generation across different
evaluation datasets. The LLM (Qwen 2.5 Coder 7B Instruct) finetuned with Verilog PoT
dataset, a highly curated dataset for Verilog code generation, consistently produces the best
results across most evaluation scenarios, establishing itself as the optimal fine-tuning dataset in
this study. The results are discussed below for Pass@1, Pass@5, and Pass@10 metrics.
Performance is represented using a color-coded scheme for better clarity: Green tones (bright
Green and light Green) signify higher performance, while Red tones (bright Red and light Red)
represent lower performance. The gradient progresses from bright green (highest performance)
to light green, light red, and finally bright red (lowest performance). Additionally, the labels H, MH,
ML, and L are used to denote high, medium-high, medium-low, and low performance,
respectively, enhancing visual interpretability.

1. Pass@1 Results

Pass@1

Fine Tune High-Level

Detailed Global Block
Verilog PoT Global
Evaluate
Summaries Summaries
Summaries

High-Level Global
43 H 41.6 ML 42.3 MH 41.0 L
Summaries
Detailed Global
52.7 MH 52.4 ML 53.8 H 50.4 L
Summaries

Block Summaries 53.7 H 26.9 L 39.7 ML 51 MH

Table 10: Pass@1

a) High-Level Global Summaries as Evaluation Dataset:

 Fine-tuning on Detailed Global Summaries resulted in 42.3%, outperforming models fine-
tuned on High-Level Global Summaries (41.6%) and Block Summaries (41.0%).
 Fine-tuning on Verilog PoT achieved 43.0%, the best result in this evaluation.

b) Detailed Global Summaries as Evaluation Dataset:

76
  Fine-tuning on Verilog PoT achieved 52.7%, outperforming models fine-tuned on High-
Level Global Summaries (52.4%) and Block Summaries (50.4%).
 The best result (53.8%) was achieved by fine-tuning Detailed Global Summaries.

c) Block Summaries as Evaluation Dataset:

 Fine-tuning on Verilog PoT achieved 53.7%, outperforming High-Level Global Summaries
(26.9%) and Detailed Global Summaries (39.7%).
 The best result in this category was also observed with Verilog PoT, highlighting its
strength.

2. Pass@5 Results

Pass@5

Fine Tune High-Level

Detailed Global Block
Verilog PoT Global
Evaluate Summaries Summaries
Summaries

High-Level Global
52.1 H 50.0 ML 50.8 MH 48.6 L
Summaries
Detailed Global
62 ML 65.2 H 65 MH 61.7 L
Summaries

Block Summaries 61.4 MH 44.1 L 56.2 ML 61.8 H

Table 11: Pass@5

a) High-Level Global Summaries as Evaluation Dataset:

 Fine-tuning on Detailed Global Summaries achieved 50.8%, surpassing results from High-
Level Global Summaries fine-tuning (50.0%) and Block Summaries fine-tuning (48.6%).
 Fine-tuning on Verilog PoT achieved the best result (52.1%).

b) Detailed Global Summaries as Evaluation Dataset:

 Fine-tuning on Verilog PoT produced 62.0%, second only to Detailed Global Summaries
(65.0%).
 The results confirm the generalizability of Verilog PoT for detailed-level evaluation tasks.

c) Block Summaries as Evaluation Dataset:

 Fine-tuning on Verilog PoT achieved 61.4%, second only to Block Summaries (61.8%),
showing competitive performance even in modular evaluations.

77
 3. Pass@10 Results

Pass@10

Fine Tune High-Level

Detailed Global Block
Verilog PoT Global
Evaluate Summaries Summaries
Summaries

High-Level Global
53.6 H 53.5 MH 52.5 ML 50.7 L
Summaries
Detailed Global
64.3 ML 67.6 H 66.5 MH 63.8 L
Summaries

Block Summaries 63.5 MH 50.5 L 61.3 ML 65.1 H

Table 12: Pass@10

a) High-Level Global Summaries as Evaluation Dataset:

 Fine-tuning on High-Level Global Summaries achieved 53.5%, outperforming Detailed
Global Summaries (52.5%) and Block Summaries (50.7%).
 The best result (53.6%) was observed with the Verilog PoT dataset.

b) Detailed Global Summaries as Evaluation Dataset:

 Fine-tuning on Verilog PoT achieved 64.3%, second only to Detailed Global Summaries
(66.5%).
 These results indicate the ability of Verilog PoT to generalize effectively across evaluation
tasks.

c) Block Summaries as Evaluation Dataset:

 Fine-tuning on Verilog PoT achieved 63.5%, outperforming High-Level Global Summaries
(50.5%) and Detailed Global Summaries (61.3%).

5.2.2 Analysis

Models fine-tuned with the Verilog PoT dataset demonstrate superior performance across all
evaluation scenarios, consistently achieving top two rankings compared to other baselines. This
consistent performance highlights the versatility and generalization capabilities of the Verilog PoT
dataset. In contrast, other fine-tuned models tend to excel only in evaluation scenarios closely
aligned with their dataset characteristics and show significant underperformance in mismatched
scenarios.

Training solely with datasets containing highly detailed or excessively high-level information leads
to diminished performance. This emphasizes the importance of balanced training datasets that

78
provide both global context and detailed precision. Aside from the Verilog PoT dataset, models
trained on detailed global summaries achieve the highest pass rates. These summaries strike
an effective balance between the generality offered by high-level overviews and the precision
inherent in block-level summaries, making them a strong alternative to Verilog PoT.

These results underscore the critical role of dataset structure in fine-tuning LLMs for Verilog code
generation. Balanced datasets like Verilog PoT and detailed global summaries emerge as optimal
for producing robust and generalized model performance.

5.3 Influence of Model Architecture on Code Generation

The performance of four large language models (CodeGemma 7B Instruct, DeepSeek Coder
6.7B Instruct, Qwen-2.5 Coder 7B Instruct, and CodeLlama 7B Instruct) is analyzed based
on their evaluation loss during fine-tuning on two datasets: Verilog PoT Summaries and High-
Level Summaries. Evaluation loss indicates the model's generalization ability to unseen data
during fine-tuning. Lower evaluation loss implies better performance and alignment with the target
dataset.

5.3.1 Evaluation Loss Analysis for Verilog PoT Dataset

Figure 25: Evaluation Loss during Finetuning of Verilog PoT Dataset

This graph shows the evaluation loss trends when models are fine-tuned on the Verilog PoT
dataset.

1. DeepSeek Coder 6.7B Instruct:

79
  Starts with a low initial evaluation loss (~0.25), showing strong alignment with the Verilog
PoT dataset.
 Consistently maintains the lowest evaluation loss throughout training, converging to ~0.05
by 12,000 steps.
 Demonstrates smooth and stable learning with minimal fluctuations.

2. Qwen-2.5 Coder 7B Instruct:

 Starts with a slightly higher evaluation loss compared to DeepSeek but shows steady
improvement.
 Final evaluation loss reaches almost same to DeepSeek Coder 6.7B Instruct.
 Occasional fluctuations indicate slight instability during fine-tuning.

3. CodeLlama 7B Instruct:
 Begins with a higher evaluation loss (~0.37) but reduces steadily.
 Final evaluation loss settles around ~0.07, reflecting reasonable generalization but lagging
behind DeepSeek and Qwen.

4. CodeGemma 7B Instruct:
 Starts with the highest evaluation loss (~0.47), indicating poor initial alignment with Verilog
PoT dataset.
 Despite consistent improvement, it converges to the highest final evaluation loss (~0.1),
suggesting weaker generalization compared to other models.

Key Insights:

 DeepSeek Coder 6.7B Instruct is the best-performing model for Verilog PoT dataset,
demonstrating superior generalization and learning efficiency.
 Qwen-2.5 Coder 7B Instruct also performs well but exhibits minor instability during
training.
 CodeLlama 7B Instruct and CodeGemma 7B Instruct lag behind, with CodeGemma
requiring further optimization for effective fine-tuning on Verilog PoT.

80
5.3.2 Evaluation Loss for High-Level Summaries Dataset

Figure 26: Evaluation Loss during Finetuning of High-Level Summaries Dataset

This graph illustrates the evaluation loss trends when models are fine-tuned on the High-Level
Summaries dataset.

1. DeepSeek Coder 6.7B Instruct:

 Starts with the initial evaluation loss (~0.37), demonstrating strong pretraining alignment
with high-level summaries.
 Converges quickly to the lowest final evaluation loss (~0.25) by 5,000 steps, outperforming
other models.
 Exhibits smooth convergence with negligible fluctuations.

2. Qwen-2.5 Coder 7B Instruct:

 Begins with a slightly lower initial evaluation loss (~0.35) than DeepSeek.
 Achieves significant improvement as finetuning progresses, converging to approximately
0.26 by the end of the process, which is slightly higher than DeepSeek.
 Displays occasional fluctuations, reflecting some instability in learning.

3. CodeLlama 7B Instruct:
 Starts with highest initial loss (~0.47) and shows steady improvement as training steps
increases.
 Final evaluation loss stabilizes around ~0.29, a little bit higher than Qwen and DeepSeek.
 Overall performance is moderate but consistent.

81
4. CodeGemma 7B Instruct:
 Begins with the initial evaluation loss (~0.42) and shows slower improvement.
 Final evaluation loss remains the highest (~0.31), indicating limited ability to generalize to
High-Level Summaries.

Key Insights:

 DeepSeek Coder 6.7B Instruct excels in fine-tuning on High-Level Summaries, achieving

the lowest evaluation loss and demonstrating excellent generalization ability.
 Qwen-2.5 Coder 7B Instruct performs competitively, with slightly higher loss than
DeepSeek but still generalizing effectively.
 CodeLlama 7B Instruct shows moderate performance, trailing behind Qwen and
DeepSeek.
 CodeLlama 7B Instruct underperforms significantly, requiring further optimization for
tasks involving high-level summaries.

5.3.3 Overall Observations

1. DeepSeek Coder's Superior Performance:

 In both datasets, DeepSeek Coder achieves the lowest evaluation loss, highlighting its
strong generalization ability and efficient adaptation to varying dataset types.
 Its stability and smooth convergence make it the most reliable model for Verilog code
generation.

2. Qwen's Competitive Potential:

 Qwen performs consistently well on both datasets, achieving near-optimal evaluation loss.
However, occasional fluctuations indicate minor instability during training.

3. CodeLlama's Moderate Performance:

 CodeLlama exhibits reasonable performance but consistently lags behind DeepSeek and
Qwen. Its slower loss reduction suggests a need for better optimization.

4. CodeGemma's Weak Generalization:

 CodeGemma struggles in both datasets, achieving the highest evaluation loss. Its slow
improvement highlights the need for architectural adjustments or additional fine-tuning
steps.

The evaluation loss trends establish DeepSeek Coder 6.7B Instruct as the best-performing
model for both Verilog PoT and High-Level Summaries datasets. Qwen-2.5 Coder 7B Instruct
also performs well, while CodeLlama 7B Instruct provides moderate results. CodeLlama 7B
Instruct, however, requires further optimization to match the performance of the other models.
These results emphasize the importance of model selection and fine-tuning strategies for Verilog
code generation tasks.

82
5.4 Influence of Training Samples in Fine Tuning

Training Dataset Size Vs Pass@k

60

55 Pass@1
Pass@5
Pass Rates

50 Pass@10

45

40

35
0 2000 4000 6000 8000 10000 12000

Training Samples

Figure 27: The Impact of Training Samples during Finetuning

The graph depicts the relationship between the number of training samples and the Pass@k
metrics (Pass@1, Pass@5, and Pass@10) for the CodeLlama-7B-Instruct model fine-tuned using
detailed level summaries. The dataset sizes analyzed include 2000, 6000, and 11,100 training
samples.

5.4.1 Analysis of the Impact of Increasing Training Samples

1. Pass@1:
 Performance increases significantly with additional training samples, rising from
approximately 39% at 2000 samples to nearly 51% at 11,100 samples.
 The steepest improvement is observed between 2000 and 6000 samples, with smaller
gains beyond 6000.

2. Pass@5:
 Similar to Pass@1, Pass@5 shows noticeable improvement as the number of samples
increases.
 The rate increases from approximately 45% at 2000 samples to around 55% at 11,100
samples, with diminishing performance gains beyond 6000 samples.

3. Pass@10:

83
  Pass@10 consistently achieves the highest rates, starting from approximately 48% at
2000 samples and peaking at ~56% at 11,100 samples.
 The increase between 6000 and 11,100 samples is minimal, indicating that larger datasets
yield diminishing returns for Pass@10.

5.4.2 Diminishing Returns

While increasing training samples improves performance, the rate of improvement diminishes as
the sample size grows, particularly beyond 6000 samples. The marginal improvement from 6000
to 11,100 samples is notably smaller compared to the improvement from 2000 to 6000 samples.
This suggests that the model begins to saturate in its ability to learn effectively from additional
data.

One of the reasons to the diminishing returns could be attributed to limited diversity in the dataset.
If the training samples do not adequately cover a broader range of code patterns or structures,
additional samples may offer less novel information for the model to learn. Another reason can
be that detailed level summaries may provide strong contextual insights but might lack the
variability seen in other dataset descriptions, limiting further performance gains.

Therefore, the analysis shows that increasing the number of training samples improves the
performance of the CodeLlama-7B-Instruct model, with diminishing returns observed as the
sample size exceeds 6000. These trends highlight the importance of balancing dataset size,
diversity in fine-tuning large language models.

84
6. Conclusion
This thesis investigated how dataset descriptions, training dataset size, and LLM architectures
influence the quality of Verilog code generation. Using state-of-the-art models such as DeepSeek
Coder 6.7B Instruct, Qwen 2.5 Coder 7B Instruct, CodeLlama-7B-Instruct, and CodeGemma 7B
Instruct, the study systematically evaluated performance across various training and evaluation
configurations.

Key Findings:

1. Dataset Descriptions:

 The Verilog PoT dataset consistently outperformed other baselines, achieving superior
generalization and high pass rates across evaluation scenarios. Its balanced structure
effectively combines global context and detailed precision.
 Detailed global summaries emerged as a strong alternative, balancing generality and
detail, though they were slightly outperformed by Verilog PoT in overall performance.
 Highly detailed or excessively high-level datasets led to underperformance, emphasizing
the need for balanced dataset structures.

2. LLM Architectures:

 DeepSeek Coder 6.7B Instruct demonstrated the best overall performance, achieving
the lowest evaluation loss and highest pass rates due to its stability and adaptability.
 Qwen 2.5 Coder 7B Instruct was competitive, with effective generalization but occasional
instability during fine-tuning.
 CodeLlama-7B-Instruct and CodeGemma 7B Instruct showed weaker results,
highlighting the importance of selecting and optimizing architectures for Verilog-specific
tasks.

3. Training Dataset Size:

 Increasing training samples improved model performance, with significant gains observed
up to 6000 samples. Beyond this, diminishing returns were evident, suggesting that
dataset diversity and fine-tuning strategies are more critical than sheer size.

This thesis demonstrates that the quality of Verilog code generation is significantly influenced by
dataset descriptions, the size of training datasets, and LLM architectures. The Verilog PoT dataset
emerges as the optimal choice for balanced performance, while DeepSeek Coder 6.7B Instruct
leads in architectural adaptability. The findings underscore the importance of balancing dataset
structure, diversity, and fine-tuning techniques to maximize the potential of LLMs in domain-
specific applications like Verilog code generation. These insights pave the way for future research
in AI-driven hardware design automation, highlighting avenues for improvement in dataset
engineering, model optimization, and scalability.

85
7. Limitations and Future Work
7.1 Limitations
While the Verilog PoT dataset and other dataset descriptions provided valuable insights, the
diversity of training data was limited. The datasets primarily focused on specific code structures
and logic patterns, potentially overlooking edge cases or rare Verilog constructs that could further
test the models’ generalization capabilities.

The evaluation was restricted to lower-sized 7B parameters LLMs. Larger-scale models, which
may perform better for complex logic synthesis, were excluded due to computational and resource
constraints.

The fine-tuning process was constrained by limited access to high-performance GPUs and
computational infrastructure. This limitation restricted the ability to train larger datasets, larger-
sized LLMs, conduct more extensive experiments, or explore more computationally intensive
methods.

7.2 Future Work

Future research should focus on expanding dataset diversity to include a broader range of code
logic, structures, and edge cases. Incorporating rare Verilog constructs, multi-module designs,
and corner cases could enhance the model’s generalization capabilities.

Evaluating larger LLMs, such as models exceeding 14 billion parameters, could improve
performance on complex Verilog code generation tasks. Additionally, developing domain-specific
models optimized for hardware description languages (HDLs) could offer significant advantages.

Exploring other techniques, such as retrieval-augmented generation (RAG) or integrating AI

agents for domain-specific code generation, could reduce computational requirements and
enhance performance. These approaches can augment LLMs with external knowledge bases or
reasoning capabilities, bypassing some limitations of standard fine-tuning.

Extending the methods and findings of this study to other hardware description languages, such
as VHDL, or other domain-specific programming languages could provide further insights. A
comparative analysis across domains may uncover strategies for improving adaptability and
performance in specialized tasks.

Addressing these limitations and pursuing the outlined future directions will enhance the
performance, scalability, and applicability of LLMs for Verilog code generation. This study lays a
strong foundation, and future advancements in computational resources, dataset engineering,
and alternative techniques will continue to drive innovation in AI-driven hardware design
automation.

86
8. References
[1] M. Chen et al., “Evaluating Large Language Models Trained on Code,” Jul. 14, 2021,
arXiv: arXiv:2107.03374. doi: 10.48550/arXiv.2107.03374.
[2] B. Rozière et al., “Code Llama: Open Foundation Models for Code,” Jan. 31, 2024, arXiv:
arXiv:2308.12950. doi: 10.48550/arXiv.2308.12950.
[3] A. Chowdhery et al., “PaLM: Scaling Language Modeling with Pathways,” Oct. 05, 2022,
arXiv: arXiv:2204.02311. doi: 10.48550/arXiv.2204.02311.
[4] T. B. Brown et al., “Language Models are Few-Shot Learners,” Jul. 22, 2020, arXiv:
arXiv:2005.14165. doi: 10.48550/arXiv.2005.14165.
[5] B. Hui et al., “Qwen2.5-Coder Technical Report,” Nov. 12, 2024, arXiv: arXiv:2409.12186.
doi: 10.48550/arXiv.2409.12186.
[6] C. Team et al., “CodeGemma: Open Code Models Based on Gemma,” Jun. 19, 2024,
arXiv: arXiv:2406.11409. doi: 10.48550/arXiv.2406.11409.
[7] D. Guo et al., “DeepSeek-Coder: When the Large Language Model Meets Programming
-- The Rise of Code Intelligence,” Jan. 26, 2024, arXiv: arXiv:2401.14196. doi:
10.48550/arXiv.2401.14196.
[8] A. Vaswani et al., “Attention Is All You Need,” Aug. 02, 2023, arXiv: arXiv:1706.03762.
doi: 10.48550/arXiv.1706.03762.
[9] Y. Zhang, Z. Yu, Y. Fu, C. Wan, and Y. C. Lin, “MG-Verilog: Multi-grained Dataset Towards
Enhanced LLM-assisted Verilog Generation,” in 2024 IEEE LLM Aided Design Workshop (LAD),
Jun. 2024, pp. 1–5. doi: 10.1109/LAD62341.2024.10691738.
[10] M. Liu, N. Pinckney, B. Khailany, and H. Ren, “Invited Paper: VerilogEval: Evaluating
Large Language Models for Verilog Code Generation,” in 2023 IEEE/ACM International
Conference on Computer Aided Design (ICCAD), Oct. 2023, pp. 1–8. doi:
10.1109/ICCAD57390.2023.10323812.
[11] M. Liu et al., “ChipNeMo: Domain-Adapted LLMs for Chip Design,” Apr. 04, 2024, arXiv:
arXiv:2311.00176. doi: 10.48550/arXiv.2311.00176.
[12] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep
Bidirectional Transformers for Language Understanding”.
[13] A. B. V. Kumar, “Fine Tuning LLM: Parameter Efficient Fine Tuning (PEFT) — LoRA &
QLoRA — Part 1,” Medium. Accessed: Jan. 19, 2025. [Online]. Available:
[Link]
part-1-571a472612c4
[14] DhanushKumar, “Fine-tuning with LoRA and QLoRA: Enhancing Efficiency in Neural
Network Adaptation,” Medium. Accessed: Jan. 19, 2025. [Online]. Available:
[Link]
neural-network-adaptation-8b4d1473274b
[15] H. Touvron et al., “Llama 2: Open Foundation and Fine-Tuned Chat Models,” Jul. 19, 2023,
arXiv: arXiv:2307.09288. doi: 10.48550/arXiv.2307.09288.
[16] A. Norouzi, “The Ultimate Guide to LLM Fine Tuning: Best Practices & Tools | Lakera –
Protecting AI teams that disrupt the world.” Accessed: Jan. 19, 2025. [Online]. Available:
[Link]

87
[17] H. Rawat, “LLM finetuning.” Accessed: Jan. 20, 2025. [Online]. Available:
[Link]
[18] D. Karunakaran, “Parameter Efficient Finetuning (PEFT) of LLM,” Intro to Artificial
Intelligence. Accessed: Jan. 20, 2025. [Online]. Available: [Link]
intelligence/parameter-efficient-finetuning-peft-of-llm-710831c0ffb3
[19] B. Banjara, “A Comprehensive Guide to Fine-Tuning Large Language Models,” Analytics
Vidhya. Accessed: Jan. 20, 2025. [Online]. Available:
[Link]
[20] M. Lupu, “Sharpen Your LLMs with Efficient Fine-Tuning Techniques,” Medium. Accessed:
Jan. 20, 2025. [Online]. Available: [Link]
with-efficient-fine-tuning-techniques-1ec7def74916
[21] E. J. Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models,” Oct. 16, 2021,
arXiv: arXiv:2106.09685. doi: 10.48550/arXiv.2106.09685.
[22] C. R. Wolfe, “Easily Train a Specialized LLM: PEFT, LoRA, QLoRA, LLaMA-Adapter, and
More.” Accessed: Jan. 20, 2025. [Online]. Available: [Link]
specialized-llm-peft-lora-qlora-llama-adapter-and-more-aedb5be39244
[23] Y. Bengio, I. Goodfellow, and A. Courville, Deep learning, vol. 1. MIT press Cambridge,
MA, USA, 2017. Accessed: Jan. 20, 2025. [Online]. Available:
[Link]
[24] D. Masters and C. Luschi, “Revisiting Small Batch Training for Deep Neural Networks,”
Apr. 20, 2018, arXiv: arXiv:1804.07612. doi: 10.48550/arXiv.1804.07612.
[25] I. Loshchilov, “Decoupled weight decay regularization,” ArXiv Prepr. ArXiv171105101,
2017.
[26] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a
simple way to prevent neural networks from overfitting,” J. Mach. Learn. Res., vol. 15, no. 1, pp.
1929–1958, 2014.
[27] J. Hoffmann et al., “Training Compute-Optimal Large Language Models,” Mar. 29, 2022,
arXiv: arXiv:2203.15556. doi: 10.48550/arXiv.2203.15556.
[28] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “QLoRA: Efficient Finetuning
of Quantized LLMs,” Adv. Neural Inf. Process. Syst., vol. 36, pp. 10088–10115, Dec. 2023.
[29] P. Vaithilingam, T. Zhang, and E. L. Glassman, “Expectation vs. Experience: Evaluating
the Usability of Code Generation Tools Powered by Large Language Models,” in CHI Conference
on Human Factors in Computing Systems Extended Abstracts, New Orleans LA USA: ACM, Apr.
2022, pp. 1–7. doi: 10.1145/3491101.3519665.
[30] X. Jiang et al., “Self-Planning Code Generation with Large Language Models,” ACM
Trans. Softw. Eng. Methodol., vol. 33, no. 7, pp. 1–30, Sep. 2024, doi: 10.1145/3672456.
[31] L. Zhong and Z. Wang, “Can ChatGPT replace StackOverflow? A Study on Robustness
and Reliability of Large Language Model Code Generation,” Jan. 27, 2024, arXiv:
arXiv:2308.10335. doi: 10.48550/arXiv.2308.10335.
[32] S. Zhang, Z. Chen, Y. Shen, M. Ding, J. B. Tenenbaum, and C. Gan, “Planning with Large
Language Models for Code Generation,” Mar. 09, 2023, arXiv: arXiv:2303.05510. doi:
10.48550/arXiv.2303.05510.
[33] L. Chen et al., “A Survey on Evaluating Large Language Models in Code Generation
Tasks,” Aug. 29, 2024, arXiv: arXiv:2408.16498. doi: 10.48550/arXiv.2408.16498.

88
[34] A. Ni et al., “L2ceval: Evaluating language-to-code generation capabilities of large
language models,” Trans. Assoc. Comput. Linguist., vol. 12, pp. 1311–1329, 2024.
[35] S. Thakur et al., “Benchmarking large language models for automated verilog rtl code
generation,” in 2023 Design, Automation & Test in Europe Conference & Exhibition (DATE), IEEE,
2023, pp. 1–6. Accessed: Jan. 20, 2025. [Online]. Available:
[Link]
[36] S. Thakur et al., “VeriGen: A Large Language Model for Verilog Code Generation,” ACM
Trans. Des. Autom. Electron. Syst., vol. 29, no. 3, pp. 1–31, May 2024, doi: 10.1145/3643681.
[37] N. Wang, B. Yao, J. Zhou, X. Wang, Z. Jiang, and N. Guan, “Large Language Model for
Verilog Generation with Golden Code Feedback,” Aug. 05, 2024, arXiv: arXiv:2407.18271. doi:
10.48550/arXiv.2407.18271.
[38] C.-T. Ho, H. Ren, and B. Khailany, “VerilogCoder: Autonomous Verilog Coding Agents
with Graph-based Planning and Abstract Syntax Tree (AST)-based Waveform Tracing Tool,” Aug.
15, 2024, arXiv: arXiv:2408.08927. doi: 10.48550/arXiv.2408.08927.
[39] F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoorn, “A systematic evaluation of large
language models of code,” in Proceedings of the 6th ACM SIGPLAN International Symposium on
Machine Programming, San Diego CA USA: ACM, Jun. 2022, pp. 1–10. doi:
10.1145/3520312.3534862.
[40] X. Du et al., “Evaluating Large Language Models in Class-Level Code Generation,” in
Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, Lisbon
Portugal: ACM, Apr. 2024, pp. 1–13. doi: 10.1145/3597503.3639219.
[41] J. Liu, S. Xie, J. Wang, Y. Wei, Y. Ding, and L. Zhang, “Evaluating Language Models for
Efficient Code Generation,” Aug. 12, 2024, arXiv: arXiv:2408.06450. doi:
10.48550/arXiv.2408.06450.
[42] N. Tihanyi, T. Bisztray, M. A. Ferrag, R. Jain, and L. C. Cordeiro, “How secure is AI-
generated code: a large-scale comparison of large language models,” Empir. Softw. Eng., vol.
30, no. 2, p. 47, Mar. 2025, doi: 10.1007/s10664-024-10590-1.
[43] M. Weyssow, X. Zhou, K. Kim, D. Lo, and H. Sahraoui, “Exploring Parameter-Efficient
Fine-Tuning Techniques for Code Generation with Large Language Models,” Dec. 27, 2024,
arXiv: arXiv:2308.10462. doi: 10.48550/arXiv.2308.10462.
[44] Z. Ma et al., “LLaMoCo: Instruction Tuning of Large Language Models for Optimization
Code Generation,” Mar. 05, 2024, arXiv: arXiv:2403.01131. doi: 10.48550/arXiv.2403.01131.
[45] J. Li, F. Rabbi, C. Cheng, A. Sangalay, Y. Tian, and J. Yang, “An Exploratory Study on
Fine-Tuning Large Language Models for Secure Code Generation,” Aug. 17, 2024, arXiv:
arXiv:2408.09078. doi: 10.48550/arXiv.2408.09078.
[46] M. A. Haider, A. B. Mostofa, S. S. B. Mosaddek, A. Iqbal, and T. Ahmed, “Prompting and
Fine-tuning Large Language Models for Automated Code Review Comment Generation,” Nov.
15, 2024, arXiv: arXiv:2411.10129. doi: 10.48550/arXiv.2411.10129.
[47] A. Storhaug and J. Li, “Parameter-Efficient Fine-Tuning of Large Language Models for
Unit Test Generation: An Empirical Study,” Nov. 04, 2024, arXiv: arXiv:2411.02462. doi:
10.48550/arXiv.2411.02462.
[48] G. Poesia et al., “Synchromesh: Reliable code generation from pre-trained language
models,” Jan. 26, 2022, arXiv: arXiv:2201.11227. doi: 10.48550/arXiv.2201.11227.

89
90