1 module circuit( 1 module circuit(

2 //... Interface ... 2 //... Interface ...

3 ); 3 );
4 assign parity = ^data_in; 4 always @(posedge clk)
5 always @(posedge clk) 5 begin
6 begin 6 data_reg <= data_in;

Large Language Model for Verilog Generation with

7
8 end
flag <= (data_in==4'd0) ? 1'b1 : 1'b0;

9 assign data_out = data_in;

7 end
8 always @(posedge clk)
9 begin
10 always @(posedge clk) 10 flag <= (data_in==4'd0) ? 1'b1 : 1'b0;
Code-Structure-Guided Reinforcement Learning 11 begin
12 data_reg <= data_in;
11 end
12 assign parity = ^data_in;
13 end 13 assign data_out = data_in;
14 endmodule 14 endmodule
Ning Wang1 Bingkun Yao1 Jie Zhou2 Yuchen Hu2 Xi Wang2 Nan Guan1 Zhe Jiang2
1 2
City University of Hong Kong Southeast University

Abstract—Recent advancements in large language models (LLMs) have 1 module Error_Check_Circuits( 1 module Error_Check_Circuits(
sparked significant interest in the automatic generation of Register Trans- 2 //... Interface ... 2 //... Interface ...
fer Level (RTL) designs, particularly using Verilog. Current research 3 ); 3 );
on this topic primarily focuses on pre-training and instruction tuning, 4 assign parity = ^data_in; 4 always @(posedge clk)
but the effectiveness of these methods is constrained by the limited 5 always @(posedge clk) 5 begin
6 data_reg <= data_in;
availability of training data, as public Verilog code is far less abundant 6 begin
arXiv:2407.18271v4 [[Link]] 19 Apr 2025

7 7 end
than software code. In particular, these methods struggle to effectively flag <= (data_in==4'd0) ? 1'b1 : 1'b0;
8 end 8 always @(posedge clk)
capture Verilog’s parallel code structures, which fundamentally differ
9 assign data_out = data_in; 9 begin
from the imperative, sequential control flow typical in most software flag <= (data_in==4'd0) ? 1'b1 : 1'b0;
programming languages. This paper introduces VeriSeek, an LLM en- 10 always @(posedge clk) 10 end
hanced by reinforcement learning using a limited amount of high-quality 11 begin 11
12 12 assign parity = ^data_in;
data_reg <= data_in;
training data to achieve high Verilog code generation performance. Our 13 end 13 assign data_out = data_in;
reinforcement learning approach employs code structure information
14 endmodule 14 endmodule
as feedback signals to refine the pre-trained model, enabling it to
effectively learn important patterns from Verilog code with parallel Fig. 1: Two functionally equivalent Verilog modules with different
structures. Experiments show that VeriSeek outperforms state-of-the-art
methods across multiple benchmarks. We release VeriSeek’s complete token sequences. The left implementation follows (parity → f lag →
implementation framework, including the dataset, source code, and model dataout → datareg ) sequence, whereas the right one is (datareg →
weights, at [Link] f lag → parity → dataout ). Corresponding colors between the left
and right implementations represent identical code segments.
I. I NTRODUCTION
Large language models (LLMs) have demonstrated promising ca-
pabilities in various software programming tasks, prompting hardware However, applying reinforcement learning to post-train LLMs for
design researchers to explore their applications in hardware design Verilog code generation presents significant challenges. Although
processes. One key application is using LLM for automatic generation reinforcement learning has proven effective for post-training LLMs
of Hardware Description Language (HDL) code, such as Verilog, for software code generation [12], [14], it performs poorly when
from specifications written in natural language. directly applied to Verilog code generation (Section IV provides
The primary challenge in utilizing LLMs for Verilog code gen- detailed experimental results illustrating this). A primary challenge
eration is the scarcity of training data, as the available open-source stems from Verilog’s inherent parallel structures, which contrast
Verilog code is limited in both quantity and quality. Despite recent with the sequential execution typical of most software programming
efforts in data collection and synthesis [31], [33], [16], the volume of languages. For instance, Fig. 1 showcases two Verilog code segments
training data is still inadequate (much fewer than data available for that are functionally identical but exhibit substantial differences if
training LLM to generate code in software programming languages). compared as token sequences.
Moreover, using commercial models like GPT for training data This work introduces VeriSeek, an LLM developed from DeepSeek-
synthesis or augmentation can hinder model performance, as it may Coder [7] and enhanced by reinforcement learning with a novel
introduce biases from the source models, leading to performance reward function to address the aforementioned challenge. Our reward
degradation through recursive training effects [27]. function assesses the generated code by comparing its structural
In this work, we aim to explore effective methods to train LLMs similarities with the reference code, enabling the model to effectively
for Verilog code generation using limited data. Typically, training capture Verilog-specific code patterns. Specifically, we convert the
coding-oriented LLMs involves three stages [4]. The first stage, pre- code into an Abstract Syntax Tree (AST) [1] and develop a similarity
training, utilizes vast corpora of code and documentation to let scoring algorithm to evaluate the structural correspondence. This
the model understand the fundamental programming concepts and reward function is integrated with the Proximal Policy Optimization
syntax. The second stage, instruction tuning, enhances the model’s (PPO) algorithm [25] to post-train the model. VeriSeek outperforms
ability to interpret and execute specific coding tasks. Finally, post- existing state-of-the-art models [33], [6], [31], [17] on Verilog code
training, typically using reinforcement learning, adapts the model to generation benchmarks, RTLLM2.0 [18] and VerilogEval [15].
specific programming paradigms. Comparing with pre-training and While post-training does not rely on a large amount of training
instruction tuning, post-training typically requires much less data as data, it is sensitive to data quality. This is because post-training
it emphasizes on exploring the model’s existing capabilities rather employs more explicit and targeted training objectives, so the
than acquiring new information [11]. Existing research on LLM noise in training data may cause greater disruption to the model.
training for Verilog code generation primarily focuses on the pre- [8]. Therefore, we have curated a dataset VeriCores derived from
training and instruction tuning stages [31], [34], [17]. In contrast, our OpenCores [20], a repository recognized for its high-quality open-
work emphasizes on the post-training stage. Specifically, we apply source hardware designs. Each instance in VeriCores comprises a
reinforcement learning to aggressively explore the parameter space natural-language specification as the model input and a high-quality
and achieve better learning performance with limited training data. reference Verilog code. Both VeriSeek and VeriCores are released at
 Legend Pre-train Corpus Specification 𝐱 Reference code 𝐲 Reward 𝒓 𝐲, 𝐲&
C/C++ Please act as a professional module fedet(
Input
verilog designer. Implement a //... Module Instantiation ...
Output
#include <bits/stdc++.h> 𝐱 module to detect a falling );
using namespace std; edge on its digital input signal always @(posedge clk or posedge rst)
Label int n, m; int a[50]; (`din`). Upon detecting a begin
int main() { falling edge, … Q <= {~Q[0] & Q[1], Q[0], din};
Model cin >> n >> m; Module name: fedet Input end
ports: rst (input): A endmodule
for (int i=0; i<n; i++) { 𝐲 𝟏𝟎 ∗ 𝒔𝒊𝒎𝐀𝐒𝐓 𝐲, 𝐲)
cin >> a[i]; synchronous reset signal. AST-parsable
a[i] %= m; When asserted (1), it
… initializes the internal state of module fedet(
the module. This signal is //... Module Instantiation ...
Verilog );
active high. clk (input): The always @(posedge clk or posedge rst)
module adder_reg( 𝐱 clock signal that drives the begin
input clk, operation of the module. All Q[0] <= din;
input [7:0] x, state transitions occur on the Q[1] <= Q[0];
input [7:0] y, rising edge of this clock. din Q[2] <= !(Q[0]) && Q[1];
output carry_output, (input): The digital input end
output [7:0] sum);
𝐲 signal on which falling edges endmodule
assign carry = sum[8]; are to be detected
… AST-unparsable
…
module fedet(
//... Module Instantiation ...
);
Base Continue Pre-trained Post-trained 𝐲! reg [2:0] Q;
−𝟓
Model 𝝅𝝓 Model 𝝅𝝍 Model 𝝅𝜽 always @(posedge clk or posedge rst)
if(rst) Q <= 3'b011;
Structure-guided …
Continual Pre-training Reinforcement Learning endmodule
2 𝜋- 𝐲|𝐱
& Not valid Verilog code
−0 log 𝜋𝝍 𝑦&/ |𝐱, 𝐲&3/ 𝔼 𝐱~(!"#" ,𝐲~,
* 𝑟 𝐲,
& 𝐲 − 𝛽 log
/01
$ 𝜋. 𝐲|𝐱
& This moule is used to detect the falling
edge. The module should test using, ... −𝟏𝟎

Fig. 2: Overview of VeriSeek’s training pipeline and reward mechanism. Starting from a base model πϕ , the model is trained on Verilog
and C/C++ code to get πψ . In the subsequent reinforcement learning stage, the model πθ learns to generate Verilog code ŷ from natural
language specifications x by optimizing a code-structure-guided reward function r(y, ŷ). This reward function evaluates the similarity
between generated and reference code using AST-based similarity simAST . For unparsable generations, negative rewards (-10 or -5) are
assigned based on the severity of syntax violations, encouraging the model to maintain proper Verilog syntax and semantics.

[Link] generation iteratively. However, post-training LLMs for Verilog code

generation with reinforcement learning remains unexplored.
II. R ELATED W ORK
A. LLM for Verilog Code Generation III. VeriSeek
Many studies have advanced LLM-based Verilog code generation. VeriSeek receives a natural-language specification as its input and
Thakur [30] contributed to Verilog code generation collection through outputs the corresponding Verilog code. As shown in Fig. 2, VeriSeek
synthetic data generation and repository preprocessing. [17] devel- is obtained on the base model through two training steps. The
oped RTLCoder that outperforms GPT-3.5 by training on automat- first step is continual pre-training, which enhances the LLM’s basic
ically generated datasets using GPT. MG-Verilog [33] constructed understanding of Verilog syntax. The second step is reinforcement
a multi-grained dataset that pairs Verilog code with descriptions at learning, which enables the LLM to learn Verilog-specific code pat-
different detail levels to improve the model’s instruction-following terns through iterative feedback and optimization. For reinforcement
capability. BetterV [22] introduced BetterV, which creates training learning, we design a code-structure-guided reward function that
datasets by converting Verilog code to C language, enabling LLMs to evaluates AST similarities between generated and reference code.
leverage their knowledge of general-purpose programming languages. Since this reward mechanism requires reliable reference code, we
Despite these efforts, the available datasets remain insufficient for curate a high-quality dataset named VeriCores and integrate these
comprehensive model training, which requires effective usage of data. components into a PPO-based post-training framework.
B. Post-training LLMs for Coding
A. Continual Pre-training
Recent research has explored reinforcement learning approaches to
improve LLMs’ coding capabilities, specifically focusing on reward We use the public dataset VGen [31] for unsupervised continual
design mechanisms. [4] established the fundamental approach by pre-training. VGen aggregates Verilog repositories from GitHub and
using program outputs and runtime states to create execution-based applies systematic filtering to remove duplicates. VGen also includes
reward signals. Subsequently, [12] developed a hierarchical reward text extracted from 70 Verilog textbooks. In total, VGen dataset
framework that separates code evaluation into structural correctness contains approximately 50 million tokens, with an 8 : 2 ratio between
and functional completion components, thus enabling more specific Verilog code and natural language docstrings and comments.
learning signals. [14] extended this line of work by implementing a Our experiments show that training with C/C++ code helps the
test-based feedback mechanism, where automatically generated test model better understand and generate Verilog code, which aligns
cases function as reward signals for comprehensive code evaluation. with results from previous research [23]. Consequently, we expanded
[13] enhanced the reward signals by integrating static analysis metrics the training data withCodeSearchNet [10], providing approximately
to address both functionality and code quality. Furthermore, [5] im- 10 million tokens with a 9:1 ratio between C/C++ code and their
plemented a compiler-feedback mechanism as reinforcement signals, documentations. The effectiveness of integrating C/C++ code in
allowing the model to learn from syntax errors and improve code continual pre-training is evaluated in Section IV.
 Algorithm 1: simAST : compute structural similarity between (Line #[16]), indicating that at least one tree contains child nodes,
two cleaned ASTs the algorithm calculates the average of the summed similarities
Input: t1 and t2 , the root nodes of two cleaned ASTs (Line #[17]). Alternatively, when both trees reach their leaf nodes
Output: Similarity in [0.0, 1.0] with matching types (Line #[18]), the algorithm returns the maximum
1: if t1 and t2 have the same type then similarity 1.0 (Line #[19]).
2: C1 ← the set of t1 ’s children nodes simAST is used to calculates the structural similarity between
3: C2 ← the set of t2 ’s children nodes the cleaned AST of the generated code and the reference code in
4: (sum, seen) ← (0, ∅)
normal cases. There are also cases where the generated code failed
5: for every c1 in C1 do
6: (best s, best c) ← (0, null)
to be parsed into an AST, for which we give a negative reward
7: for every c2 in C2 \ seen do for punishment. In some cases, the LLM does not generate any
8: if c1 and c2 have the same type then valid Verilog code at all (e.g., the LLM just continues to write
9: s ← simAST (c1 , c2 ) the specification instead of generating code), for which we give an
10: if s > best s then even larger punishment. In our implementation, the reward is finally
11: (best s, best c) ← (s, c2 ) defined as:

10 ∗ simAST (t1 , t2 ), if y is AST-parsable
12: if best c is not null then
r(y, ŷ) = −5.0, if y is valid code but not AST-parsable
13: sum ← sum +S best s −10.0, if y is not valid code

14: seen ← seen {best c}
where y and ŷ represents the generated code by the LLM and the
15: max size ← max (|c1 |, |c2 |)
reference code corresponding to the same specification; t1 and t2
16: if max size > 0 then
represents the root node of y and ŷ, respectively.
17: return sum/max size
2) Proximal Policy Optimization: We incorporate the reward intro-
18: else duced above into Proximal Policy Optimization (PPO) [26], a widely-
19: return 1.0
used reinforcement learning method, to post-train our model.
20: else Here, we represent the LLM as a policy (learned mapping function)
21: return 0.0 πθ , where θ denotes the model parameters. This policy receives a
design specification x and produces a text response ŷ token by token:
Y
πθ (ŷ | x) = πθ (ŷt | x, ŷ<t ), (1)
B. Code-Structure-Guided Reinforcement Learning t

1) Code-Structure-Guided Reward: We first use Pyverilog [29], PPO works by gradually improving the model’s behavior through
an open-source hardware design processing toolkit for Verilog, to an iterative optimization process. During this process, the model
generate AST of the code. We then generate the cleaned AST, by learns from feedback while staying close to its original behavior.
keeping the syntactic structure like operator types, module hierarchy Specifically, the objective function of PPO is defined as:
and statement types while discarding variable names and constant
 
values from the original AST. The award function is calculated using πθ (ŷ | x)
Jr (πθ ) = Ex∼pdata ,ŷ∼πθ r(ŷ, y) − β log . (2)
simAST , which compares the similarity of the cleaned ASTs of the πψ (ŷ | x)
generated code and the reference code, as shown in Alg. 1.
This objective comprises the code-structure-guided reward function
simAST computes the similarity between two cleaned ASTs r(ŷ, y) that evaluates the quality of generated response, and a
through recursive comparison of their nodes and structures. The Kullback-Leibler (KL) divergence term weighted by β. The KL
algorithm receives the root nodes t1 , t2 of the two cleaned ASTs divergence measures how different the updated model is from the
as the input. First, it checks whether t1 and t2 share the same continual pre-trained model πψ (the original continual pre-trained
type (Line #[1]). If yes, their children nodes are put into sets C1 model). The expectation is computed over inputs sampled from the
and C2 respectively (Line #[2-3]). Otherwise the similarity is 0.0 data distribution and outputs from the current policy. Therefore, this
(Line #[20-21]) since these two cleaned ASTs with different types conservative update strategy maintains the LLM’s basic language
of root nodes are substaintially different. capabilities while improving its Verilog generation performance.
Then we iterates through each child node c1 of t1 to find its optimal
match among t2 ’s unmatched children (Line #[5]). For each child C. VeriCores Dataset
c1 , it examines each unmatched child c2 of t2 (Line #[7]). Here, Post-training’s effectiveness depends mainly on data quality rather
seen is the set of nodes in C2 that have been matched. If c1 and c2 than quantity, as it employs targeted training objectives for domain
have the same type, the algorithm computes their similarity through a adaptation. Our dataset VeriCores collects high-quality specification-
recursive call to itself (Line #[9]). If the result s exceeds the current code pairs collected from Opencores [20], an open-source digital
best similarity best s, the algorithm updates both best s and best c hardware development community. After filtering out instances where
accordingly (Line #[10-11]). specifications or output code exceeded 4096 tokens to align with
The algorithm accumulates the similarity scores of matched child LLMs’ context window constraints and maintain consistent training
pairs (Line #[13]) while tracking matched nodes in seen. These quality, and removing instances with reference code failing AST
matched nodes are excluded from consideration for remaining c1 parsing, the final dataset contains approximately 800 instances.
comparisons (Line #[14]) to ensure one-to-one matching. Figure 3 presents statistics of VeriCores, highlighting its diverse
The final similarity score is normalized to ensure the similarity structural characteristics. The instruction lengths range from 1501 to
is in the range of [0.0, 1.0] We first set the maximum number of 3994 tokens (mean: 2761.75). The reference code ranges from 43 to
children nodes to max size (Line #[15]). If it is greater than 0 3903 tokens (mean: 784.61). The AST structures vary in depth from
 Fig. 4: pass@5 performance comparison between VeriSeek and
RTLCoder across different task categories in RTLLM2.0 benchmark.

Fig. 3: Statistics of the VeriCores dataset, showing specification and

code lengths, AST depth, node count, and branching factor metrics.
"
#
n−ci
k
pass@k := Ei 1 − n

(3)
k

5 to 24 levels (mean: 10.68), with node numbers between 8 and 799 where n is the total number of trials for each specification and ci is
(mean: 141.52). The branching factor (the number of children of each the number of correct code generations for task i. We set n = 20
node) ranges from 5.67 to 66.21 (mean: 18.01). in this experiment for comparison with baselines. When any code
within the k trials successfully passes the test, we consider the task
IV. E XPERIMENTS AND P ERFORMANCE E VALUATION addressed. The pass@k metric therefore represents the estimated
A. Training Details percentage of design tasks that can be successfully completed. We
measure syntax and functional pass@1 and pass@5 metrics, where
Based on the base model DeepSeekCoder-6.7B [7], we develop ‘syntax’ means that the code is compiled successfully and ‘functional’
three variants of our model with different training strategies. All three means that the code passes the testbench.
versions has the same model size of 6.7B parameters. In [17], RTLCoder was evaluated with a metric called ‘pass@5’,
• VeriSeekP T : Pre-trained with Verilog code only.
which evaluates whether any test among 5 trials passes the testbench.
• VeriSeekP T wC : Pre-trained with both Verilog and C/C++ code.
This metric differs from the above defined pass@k, k = 5 metric. To
• VeriSeekP T wC+RL : VeriSeekP T wC post-trained by reinforce-
enable direct comparison with RTLCoder while avoiding confusion,
ment learning. we rename this metric as hit@5 and include it in our evaluation.
Experiments are conducted on a server equipped with 8 A800-80G 2) Benchmark: We conduct performance evaluation with two
GPUs. All experiments utilize a cosine learning rate scheduler with Verilog code generation benchmarks: RTLLM2.0 and VerilogEval.
a warmup phase comprising 10% of the total training steps, and an RTLLM2.0 [18] contains 50 design tasks in four categories: Arith-
AdamW optimizer [19] with a weight decay of 0.05. Additionally, metic, Control, Memory and Miscellaneou. VerilogEval [15] is a
we employ deepspeed ZeRO-3 offload [24] for acceleration. comprehensive benchmark with tasks ranging from simple combi-
Following the hyper-parameter settings in the traning of the base national logic to complex state machines. Since we focus on natural
model DeepSeekCoder, we adopt a peak learning rate of 1e−4 language specifications, we exclude hand-written tasks in VerilogEval
and a batch size of 32 for continual pre-training, training for 1 with specifications not in nature languages (e.g., using waveforms to
epoch. For reinforcement learning, we employ Low-rank Adaptation describe the expected output). For both RTLLM2.0 and VerilogEval,
(LoRA) [9] on query and value projection matrices to reduce memory each design task has a specification and corresponding testbench.
usage and training time for PPO’s iterative optimization process. Following the testing methods in [17], we evaluate syntax and
We set a peak learning rate of 1e−5 , a batch size of 8, and train functional pass rate using ModelSim [28]. Syntax pass requires the
for 10 epochs, with maximum sequence length of 2048 tokens generated code to be compiled successfully, while functional pass
and generation parameters of temperature 0.2 and top-p 0.95. The requires the code to succeed simulations with the testbench.
duration of continual pre-training is approximately 1 hour, whereas
the reinforcement learning task requires about 1 day to complete C. Performance Evaluation
training. Reinforcement learning takes considerably longer time to As shown in Table I, our model demonstrates strong capabilities
converge than continual pre-training, primarily because of PPO’s across both RTLLM2.0 and VerilogEval benchmarks. VeriSeekP T and
iterative update mechanism. PPO conducts multiple forward passes VeriSeekP T wC show substantial improvements over the base model
to collect trajectories and performs multiple optimization steps. DeepSeekCoder-6.7B. After reinforcement learning, our final model
VeriSeekP T wC+RL achieves impressive results on both benchmarks.
B. Metric and Benchmark
On RTLLM2.0, VeriSeekP T wC+RL achieves the best performance
1) Metric: We evaluate the models using the widely-adopted among open-source SOTA models Thakur [30], ChipGPT [3] and
pass@k metric for code generation, which is the percentage of RTLCoder [17]. While GPT-4 maintains the overall best performance,
problems solved by using k generated programs per problem [30]: our model surpasses GPT-3.5 across almost all metrics in both syntax
TABLE I: Comparison of model performance on RTLLM2.0 and VerilogEval benchmarks, showing syntax and functional correctness metrics
(pass@1, pass@5, hit@5) for our models against open-source SOTA models, GPT-3.5 and GPT-4. All metrics are in %.

RTLLM2.0 VerilogEval
Syntax Function Syntax Function
Type Model
pass@1 pass@5 hit@5 pass@1 pass@5 hit@5 pass@1 pass@5 hit@5 pass@1 pass@5 hit@5
GPT-3.5 74.8 90.6 98.0 34.4 49.8 52.1 75.4 86.0 87.5 46.7 69.1 71.3
Closed-Source
GPT-4 80.0 89.5 98.9 47.9 58.0 68.9 76.1 86.8 87.4 60.0 70.6 72.8
Goh-7B 62.2 78.4 84.9 19.2 20.1 23.7 56.7 65.1 67.4 40.6 48.4 54.4
Thakur-16B 83.2 91.3 93.4 17.4 24.6 27.8 84.7 87.2 87.6 44.0 52.6 58.3
Open-Source
MG-Verilog-7B 39.1 47.5 50.0 20.4 34.2 39.7 62.9 70.4 71.1 52.7 58.5 60.9
RTLCoder-7B 73.4 89.7 91.3 32.6 48.7 50.8 86.6 97.7 98.9 61.2 76.5 80.4
Base Models DeepSeekCoder-6.7B 72.7 88.1 88.8 26.5 36.3 42.7 73.7 84.5 86.6 54.1 63.8 65.9
VeriSeekP T -6.7B 65.6 89.3 84.1 26.2 48.9 49.2 72.9 84.1 84.9 53.3 63.5 65.2
Ours VeriSeekP T wC -6.7B 72.5 94.2 95.4 30.1 50.7 51.4 76.3 87.4 88.2 58.4 68.5 71.9
VeriSeekP T wC+RL -6.7B 73.5 94.8 96.0 31.9 54.2 52.0 85.1 98.3 99.1 61.6 76.9 81.7
+ Gray background represents the best metric (excluding GPT-4).

and functional evaluations. The radar chart in Fig.4 illustrates the VeriSeekP T wC+RL . As shown in Table II, the model achieves a
pass@5 performance across different categories in RTLLM2.0, where slight improvement in the functional pass@1 metric while performing
each axis represents a specific task. While RTLCoder achieves higher poorly across other evaluation metrics. The degraded performance can
performance on certain individual design tasks, VeriSeekP T wC+RL be attributed to two factors. First, the exposure bias in auto-regressive
demonstrates more consistent performance by successfully handling sequence generation causes model deviations from reference code,
a broader range of tasks, particularly in Miscellaneous-related tasks. as the model depends on its generated tokens rather than reference
On VerilogEval, VeriSeekP T wC+RL achieves the best performance tokens for predictions [17], [2]. Second, the sequential processing of
among all models, including GPT-4. In particular, VeriSeekP T wC+RL auto-regressive generation conflicts with Verilog’s inherent parallel
achieves functional pass rates of 61.6% (pass@1), 76.9% (pass@5) structures, thereby limiting the model’s ability to maintain consistent
and 81.7% (hit@5), which notably exceeds GPT-4’s performance of relationships between concurrent blocks and signals and resulting in
60.0% (pass@1), 70.6% (pass@5) and 72.8% (hit@5). poor generation diversity.
While VeriSeekP T wC+RL outperforms all existing open-source
models and GPT-3.5, it may lag behind GPT-4 in certain design TABLE II: Ablation study on instruction tuning, learned reward by
tasks, potentially due to GPT-4’s substantially larger size (it’s widely paired generations and parallel-unaware reward on RTLLM2.0.
believed that GPT-4 largely exceed GPT-3’s 175 billion parameters).
Syntax Function
The competitive performance of GPT-4 does not diminish the value of Model
pass@1 pass@5 hit@5 pass@1 pass@5 hit@5
VeriSeekP T wC+RL . The hardware design domain is particularly sen-
sitive to intellectual property protection and the security of designs. VeriSeekP T wC 72.5 94.2 95.4 30.1 50.7 51.4
Consequently, hardware design companies may prefer deploying their VeriSeekP T wC+F T 72.3 94.6 93.1 33.1 46.1 48.3
own LLMs rather than relying on closed-source models like GPT-4. VeriSeekP T wC+F T +RL 69.7 89.6 91.2 32.4 45.7 47.5
While the long-term competition between open-source and closed- VeriSeekP T wC+BT 38.4 50.2 54.6 19.5 33.1 38.9
source models for Verilog code generation is likely to continue, our VeriSeekP T wC+SEQ 68.6 91.0 92.2 27.5 48.3 49.7
work advances the state-of-the-art of open-source models.
VeriSeekP T wC+RL 73.5 94.8 96.0 31.9 54.2 52.0
D. Ablation Study + Gray background represents the best metric.
1) Instruction Tuning: First we discuss our attempt of applying
2) Learn Reward from Paired Generations: Now we discuss the
instruction tuning to the training of VeriSeek. Instruction tuning
attempt to apply post-training methods commonly used in natural
is a process where LLMs are trained on datasets comprising in-
language tasks to the training of VeriSeek. In natural language
structions and corresponding responses [32], enhancing their ability
tasks such as question answering, post-training of LLMs typically
to accurately follow human instructions. Instruction tuning employ
employs a learnable reward model r with Bradley-Terry modeling
Maximum Likelihood Estimation (MLE) to find the best parameters: er(x,yw )
er(x,yw ) +er(x,yl )
to capture human preferences between response
T
X pairs [35], [21]. The reward model is trained by minimizing:
Lmle = − log πψ (ŷt | x, ŷ<t ) (4)
er(x,yw )
  
t=1
L = E(x,yw ,yl )∼D − log (5)
which measures how well the model predicts each token ŷt given er(x,yw ) + er(x,yl )
the instruction x and previous tokens ŷ<t . In this objective, the model learns to assign higher scores r to
We instruction-tuned our continual pre-trained model on the winning responses compared to losing ones, where the exponential
Opencores dataset, denoted as VeriSeekP T wC+F T . Then we post- terms are normalized through softmax to obtain probabilities. To
train VeriSeekP T wC+F T +RL using PPO as the same settings with train the reward model, we construct dataset D containing triplets
(x, yw , yl ), where x represents the specification, yw denotes the Warm up Learn Deviate Converge

。 。
2
2
generated code that passes the benchmark, and yl represents the failed
generation. After training the reward model, we apply PPO with the -2
-2
same settings as before and get the model VeriSeekP T wc+BT . -4
-4
The experimental results in Table II show degraded performance -6 -6
compared even to the continual pre-trained baseline, as the Bradley- Reward
Reward
-8 -8 ----
----Smoothed
SmoothedReward
Reward
Terry model optimizes only relative differences between responses 一10 一10 Maximum
Maximumpass@S
pass@S

while ignoring absolute reward values. While this relative preference

approach is suitable for aligning with general human values, it o 50 o 50
becomes problematic in coding tasks where the space of correct
o 45 o 45 ．． RLRL pass@l
pass@l

．．
solutions is substantially smaller than that of incorrect ones. Fine-tunedpass@l
Fine-tuned pass@l

3) Effectiveness of the Code-Structure-Guided Reward: To eval- o 40 o 40 RLpass@S

RLpass@S
Fine-tuned pass@S
Fine-tuned pass@S
uate the effectiveness of code-structure-guided reward which is o 35 o 35

。 。
designed to the capture the parallel structure of Verilog code, we
modify Alg. 1 by implementing sequential node comparison between 10 20 30 40 50 60 70 80 90 100 110 120 130 140
10 20 30 40 50 60 70 80 90 100 110 120 130 140
Trainin g ste p
two ASTs: This modification implements one-by-one comparison Trainin g ste p
Fig. 6: Reward, pass@1 and pass@5 on RTLLM2.0 during training.
Algorithm 2: simAST SEQ : compute sequential structures
similarity between two cleaned ASTs
Input: t1 and t2 , the root nodes of two Cleaned ASTs finetune
Output: Similarity in [0.0, 1.0]
1: ...
2: if t1 and t2 have the same type then
converge
3: sum ← 0 learn
pretrain warm-up
4: for paired (c1 , c2 ) in (C1 , C2 ) do deviate
5: sum ← sum + simAST SEQ (c1 , c2 )

6: max size ← max(|c1 |, |c2 |)

base model
7: ... continual pretrained model
8: ... finetuned model
best PPO model
converged PPO model

between corresponding children nodes between two ASTs. Then we

Fig. 7: Optimization trajectory across different training stages.
use same settings as VeriSeekP T wc+RT to post-train the continued
pre-trained model. We refer to this variant as VeriSeekP T wc+SEQ . As
shown in Table II, this does not improve the performance, indicating Fig.7 illustrates the learning dynamics across various stages. In
the reward sequentially compares ASTs does is ineffective. the warm-up stage (0-20 steps), starting from the bottom-left corner
4) Performance with Different Temperatures: Fig. 5 presents the
with a low pass rate, the model escapes from suboptimal solutions
effect of sampling temperature on VeriSeekP T wC+RL performance is
of the continual pre-trained model. In the learning stage (20-50
evaluated across two benchmarks RTLLM2.0 and VerilogEval, with
steps), the code-structure-guided reward helps the model achieve
temperature ranging from 0.2 to 0.8 at intervals of 0.05. The experi-
better performance. The model walks towards the optimal area, the
mental results show that increasing temperature consistently degrades
yellow region in Fig.7. The deviation stage (50-100 steps), shows
both syntax and functional pass@1. However, the syntax pass@5,
the model deviating from the optimal region due to misalignment
functional pass@5 metrics are stable across different temperatures.
between reward signals and evaluation metrics. During the con-
vergence stage (100-150 steps), the model achieves an instruction-
tuned state, as illustrated by the light red trajectory in Fig. 7. This
convergence occurs because AST comparison, while effective for
parallel structures, cannot fully capture implementation requirements
from specifications due to its dependence on reference code.
These training dynamics demonstrate that well-defined rewards
that can capture the requirement of specification are essential for
stabilizing the reinforcement learning.

V. C ONCLUSION
Fig. 5: Temperature analysis of VeriSeekP T wC+RL .
We presented VeriSeek, a reinforcement learning approach for post-
training LLMs in Verilog code generation using structure-guided
E. Training Dynamics of Reinforcement Learning rewards. By leveraging AST-based structural similarity analysis,
Fig.6 shows the reward and functional pass@1 and pass@5 every VeriSeek effectively addresses the challenge of limited training data
five training steps during reinforcement learning. The optimal model in Verilog generation. Our experimental results show that VeriSeek
performance appears during the early training stages rather than at achieves state-of-the-art performance on standard benchmarks, sur-
convergence. As training progresses, the model converges to a fine- passing GPT-4 on VerilogEval. The approach specifically focus on
tuned state. We can split the training process into four stages. Verilog’s parallel structures, which differ from sequential software
codes. Future work could focus on developing reward functions that
better capture Verilog’s parallel execution patterns and reinforcement
learning strategies tailored for hardware design characteristics.
 R EFERENCES [23] Z. Pei, H.-L. Zhen, M. Yuan, Y. Huang, and B. Yu. Betterv: Con-
trolled verilog generation with discriminative guidance. arXiv preprint
[1] I. D. Baxter, A. Yahin, L. Moura, M. Sant’Anna, and L. Bier. Clone arXiv:2402.03375, 2024.
detection using abstract syntax trees. In Proceedings. International [24] J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He. Deepspeed: System
Conference on Software Maintenance (Cat. No. 98CB36272), pages 368– optimizations enable training deep learning models with over 100 billion
377. IEEE, 1998. parameters. In Proceedings of the 26th ACM SIGKDD International
[2] S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer. Scheduled sampling Conference on Knowledge Discovery & Data Mining, pages 3505–3506,
for sequence prediction with recurrent neural networks. Advances in 2020.
neural information processing systems, 28, 2015. [25] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Prox-
[3] K. Chang, Y. Wang, H. Ren, M. Wang, S. Liang, Y. Han, H. Li, and imal policy optimization algorithms. arXiv preprint arXiv:1707.06347,
X. Li. Chipgpt: How far are we from natural language hardware design. 2017.
arXiv preprint arXiv:2305.14019, 2023. [26] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Prox-
[4] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, imal policy optimization algorithms. arXiv preprint arXiv:1707.06347,
H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large 2017.
language models trained on code. arXiv preprint arXiv:2107.03374, [27] I. Shumailov, Z. Shumaylov, Y. Zhao, N. Papernot, R. Anderson, and
2021. Y. Gal. Ai models collapse when trained on recursively generated data.
[5] S. Dou, Y. Liu, H. Jia, L. Xiong, E. Zhou, W. Shen, J. Shan, Nature, 631(8022):755–759, 2024.
C. Huang, X. Wang, X. Fan, et al. Stepcoder: Improve code generation [28] Siemens Software. Modelsim.
with reinforcement learning from compiler feedback. arXiv preprint [29] S. Takamaeda-Yamazaki. Pyverilog: A python-based hardware design
arXiv:2402.01391, 2024. processing toolkit for verilog hdl. In K. Sano, D. Soudris, M. Hübner,
[6] E. Goh, M. Xiang, I. Wey, T. H. Teo, et al. From english to asic: and P. C. Diniz, editors, Applied Reconfigurable Computing, pages 451–
Hardware implementation with large language model. arXiv preprint 460, Cham, 2015. Springer International Publishing.
arXiv:2403.07039, 2024. [30] S. Thakur, B. Ahmad, Z. Fan, H. Pearce, B. Tan, R. Karri, B. Dolan-
[7] D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, Gavitt, and S. Garg. Benchmarking large language models for automated
X. Bi, Y. Wu, Y. Li, et al. Deepseek-coder: When the large language verilog rtl code generation. In 2023 Design, Automation Test in Europe
model meets programming–the rise of code intelligence. arXiv preprint Conference Exhibition (DATE), pages 1–6, 2023.
arXiv:2401.14196, 2024. [31] S. Thakur, B. Ahmad, H. Pearce, B. Tan, B. Dolan-Gavitt, R. Karri,
[8] S. Gururangan, A. Marasović, S. Swayamdipta, K. Lo, I. Beltagy, and S. Garg. Verigen: A large language model for verilog code
D. Downey, and N. A. Smith. Don’t stop pretraining: Adapt language generation. ACM Transactions on Design Automation of Electronic
models to domains and tasks. arXiv preprint arXiv:2004.10964, 2020. Systems, 29(3):1–31, 2024.
[9] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and [32] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du,
W. Chen. Lora: Low-rank adaptation of large language models. arXiv A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot
preprint arXiv:2106.09685, 2021. learners. arXiv preprint arXiv:2109.01652, 2021.
[10] H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt. [33] Y. Zhang, Z. Yu, Y. Fu, C. Wan, and Y. C. Lin. Mg-verilog: Multi-grained
Codesearchnet challenge: Evaluating the state of semantic code search. dataset towards enhanced llm-assisted verilog generation. In 2024 IEEE
arXiv preprint arXiv:1909.09436, 2019. LLM Aided Design Workshop (LAD), pages 1–5. IEEE, 2024.
[11] J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim. A survey on large [34] Y. Zhao, D. Huang, C. Li, P. Jin, Z. Nan, T. Ma, L. Qi, Y. Pan, Z. Zhang,
language models for code generation. arXiv preprint arXiv:2406.00515, R. Zhang, et al. Codev: Empowering llms for verilog generation through
2024. multi-level summarization. arXiv preprint arXiv:2407.10424, 2024.
[12] H. Le, Y. Wang, A. D. Gotmare, S. Savarese, and S. C. H. Hoi. [35] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei,
Coderl: Mastering code generation through pretrained models and deep P. Christiano, and G. Irving. Fine-tuning language models from human
reinforcement learning. Advances in Neural Information Processing preferences. arXiv preprint arXiv:1909.08593, 2019.
Systems, 35:21314–21328, 2022.
[13] B. Li, Z. Sun, T. Huang, H. Zhang, Y. Wan, G. Li, Z. Jin, and
C. Lyu. Ircoco: Immediate rewards-guided deep reinforcement learning
for code completion. Proceedings of the ACM on Software Engineering,
1(FSE):182–203, 2024.
[14] J. Liu, Y. Zhu, K. Xiao, Q. Fu, X. Han, W. Yang, and D. Ye.
Rltf: Reinforcement learning from unit test feedback. arXiv preprint
arXiv:2307.04349, 2023.
[15] M. Liu, N. Pinckney, B. Khailany, and H. Ren. Verilogeval: Evaluating
large language models for verilog code generation. In 2023 IEEE/ACM
International Conference on Computer Aided Design (ICCAD), pages
1–8. IEEE, 2023.
[16] M. Liu, Y.-D. Tsai, W. Zhou, and H. Ren. Craftrtl: High-quality synthetic
data generation for verilog code models with correct-by-construction
non-textual representations and targeted code repair. arXiv preprint
arXiv:2409.12993, 2024.
[17] S. Liu, W. Fang, Y. Lu, Q. Zhang, H. Zhang, and Z. Xie. Rtlcoder:
Outperforming gpt-3.5 in design rtl generation with our open-source
dataset and lightweight solution. In 2024 IEEE LLM Aided Design
Workshop (LAD), pages 1–5. IEEE, 2024.
[18] S. Liu, Y. Lu, W. Fang, M. Li, and Z. Xie. Openllm-rtl: Open dataset
and benchmark for llm-aided design rtl generation. 2024.
[19] I. Loshchilov and F. Hutter. Decoupled weight decay regularization.
arXiv preprint arXiv:1711.05101, 2017.
[20] OpenCores. Opencores. [Link] 2024. Accessed: 2024-
11-14.
[21] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin,
C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language
models to follow instructions with human feedback. Advances in neural
information processing systems, 35:27730–27744, 2022.
[22] Z. Pei, H.-L. Zhen, M. Yuan, Y. Huang, and B. Yu. Betterv: Con-
trolled verilog generation with discriminative guidance. arXiv preprint
arXiv:2402.03375, 2024.