LASHED: LLMs And Static Hardware Analysis for

Early Detection of RTL Bugs

Baleegh Ahmad Hammond Pearce Ramesh Karri Benjamin Tan
NYU Tandon University of New South Wales NYU Tandon University of Calgary
Brooklyn, USA Sydney, Australia Brooklyn, USA Calgary, Canada
ba1283@[Link] [Link]@[Link] rkarri@[Link] benjamin.tan1@[Link]

Abstract—While static analysis is useful in detecting early- specific heuristic patterns. Formal methods use mathematical
arXiv:2504.21770v1 [[Link]] 30 Apr 2025

stage hardware security bugs, its efficacy is limited because models to analyze and verify a design [20].
it requires information to form checks and is often unable to
explain the security impact of a detected vulnerability. Large Development and use of LLMs [21] has provided a possible
Language Models can be useful in filling these gaps by identifying means to detect bugs in code without the explicit need for
relevant assets, removing false violations flagged by static analysis a fully mature testing framework. LLMs have been used for
tools, and explaining the reported violations. LASHED combines RTL generation [22] and repair [23] with reasonable degrees
the two approaches (LLMs and Static Analysis) to overcome of success, but their ability to detect security bugs has yet to
each other’s limitations for hardware security bug detection.
We investigate our approach on four open-source SoCs for five be proven. In part, this is because LLMs do not verify their
Common Weakness Enumerations (CWEs) and present strategies outputs. Static analysis can address this shortcoming. Prior
for improvement with better prompt engineering. We find that work in the software space has explored the combination of
87.5% of instances flagged by our recommended scheme are LLMs and static analysis; for example, IRIS [24] uses CodeQL
plausible CWEs. In-context learning and asking the model to as the static analysis tool coupled with LLMs to detect the
‘think again’ improves LASHED’s precision.
Index Terms—LLMs, Static Analysis, Security, Bug Detection,
code injection vulnerabilities in Java code. Chapman et al.
CWE present an approach to interleave LLMs with the EESI static
analysis tool to detect the issue of error-specific inference [25].
LLIFT [26] uses the LLM and static analysis combination to
I. I NTRODUCTION detect use before initialization bugs within the Linux kernel.
Taking inspiration from such works, we propose a strat-
Security vulnerabilities in hardware are difficult to de-
egy which uses LLMs and Static Analysis together, i.e.,
tect [1]. It is critical to identify them early on in the system-
LASHED. We use hardware Common Weakness Enumera-
on-chip (SoC) design life-cycle because of the higher costs of
tions (CWEs) [27], which provide examples of vulnerability
fixing issues downstream (pre-silicon) or even recalls (post-
categories to aid the generalizability of our approach. We
silicon) [2]. An exhaustive search for these defects is not
use the LLM for three tasks: i) identifying security relevant
possible because of the high complexity of modern processors
assets, ii) removing false positives from static analysis, and iii)
and SoCs. Therefore, there is a need for innovative solutions
explaining the security issue posed by a reported violation.
that provide early-stage information on potential security
To identify security assets, the LLM uses the RTL source
issues at the Register-Transfer Level (RTL).
code and Hardware CWEs. The hardware static analysis tools
Existing strategies for security verification include simula- formulate checks that could indicate the presence of certain
tion with test benches, formal assertions [3], [4], hardware CWEs. We use either linting or formal property verification
fuzzing [5], [6] and information flow tracking [7], [8]. Security for static analysis, depending on the nature of the CWE.
checks “as-you-go,” while implementation is ongoing, are The linting violations and failing assertions are returned to
more challenging. Recent works have proposed static anal- the LLM to prune out those that do not pose any security
ysis [9], [10] and large language models (LLMs) [11]–[13] threat and provide explanations for the ones that do. Our
for this purpose. Static analysis checks source code without contributions are:
“executing” it, e.g., without simulating the design to perform
directed tests. The code is instead checked against a set of • We present the first framework that combines Static
coding patterns that can indicate undesirable behavior. Linters Analysis and LLMs to detect security issues in RTL code.
[14], [15] and formal verification tools [16], [17] are the two The details of this tool are described in Section II.
most commonly used static analysis [9], [18] methods for RTL. • We validate LASHED on four open source SoCs, de-
Linting is the automated checking of source code for stylistic, scribed in Section III. Results are presented in Section IV.
structural, design and programmatic checks [19], and can • We investigate the impact of in-context learning and
include data-flow analysis and control-flow analysis, as well prompting to insist that the LLM reason through its
as more abstract techniques such as pattern matching for bug- assessments. The outcomes are analyzed in Section IV-A.

1
 II. LASHED is typically present alongside some “structural” or “coding
LASHED takes RTL source and CWE information as inputs style” imperfections, we use lint checks. This is the case
and outputs potential issues pertaining to the CWE. The CWEs for CWEs 1191 and 1300. Conversely, if the CWE requires
covered are detailed in Section III-B. The output contains the verifying whether a signal behaves appropriately depending on
bug code, its explanation, and its location. The framework for the value(s) of other signal(s) or if one signal flows to another,
our approach is shown in Fig. 1. It can be broken down into we use assertions. This applies to CWEs 1231, 1233 and 1244.
three main steps, i.e., Assets Identification, Static Analysis, 1) Linter-based: We select relevant checks from the ∼1000
and Contextualization. We illustrate these steps through two VC SpyGlass Lint [14] tags in the functional lint tag database
motivating examples taken from the Hack@DAC 2021 buggy based on our understanding of the CWE. For some CWEs,
OpenPiton SoC shown in Fig. 2 and Fig. 3. we develop custom lint checks using Verific [28], where, after
The vulnerability in Fig. 2(a) shows an instance of CWE obtaining the Abstract Syntax Tree (AST) of any module, we
1191 where only the least significant 32 bits of the secret traverse it to check for some structural or stylistic element. If
message are used to authenticate the JTAG access control mod- the checks’ results include the assets identified previously, the
ule. This makes the access control susceptible to brute-force results are considered violations. These are sent to the next
attacks. The vulnerability in Fig. 3(a) shows an instance of step, i.e., contextualization.
CWE 1233 where security-sensitive registers are missing lock For CWE-1191, improper access control for debug occurs
bit protection in the Direct Memory Access (DMA) wrapper when a signal containing the user input (password) is not prop-
module. An adversary can modify them from software. erly assigned a value. We check the code for the following:
[Width mismatch, Reverse Connected busses, Improper range
A. Assets Identification (AssetID) index, Concatenation in array assign, Concatenation using
We use an LLM to identify assets relevant to the CWE that unsized numbers, RHS has concatenation] – if the assignment
LASHED is scanning for. We prompt the LLM with a system contains the previously identified access control signals, the
prompt that primes it as a hardware security expert searching signal and assignment are reported as a violation. Fig. 2(c)
for the CWE. This is followed by the description of the shows reported violations for the CWE-1191 example.
CWE from the MITRE website. There are numerous ways to For CWE-1300, we check for design structural elements that
structure the prompt and alter the included information which may cause vulnerability to side-channel attacks: [If without
we investigate (see Section III-C). For instance, for prompt else, Inferred Latches]. If the code has these, we check
strategy v1 the system prompt is followed by an example whether the conditional statement contains any previously
of the CWE in Verilog code derived from MITRE website, identified assets. The presence of these in a conditional state-
including identifying relevant assets and explaining why this ment without an else can result in information leaking.
example poses the CWE. After priming the LLM with the 2) Assertion-based: We develop a custom template for
system prompt, the LLM is given a user prompt that contains System Verilog Assertions (SVAs) for selected CWEs and
the RTL source and instructions on what kinds of signals are to populate the template with information from AssetID. The
be identified. The identified assets are sent for static analysis. formal tool verifies these SVAs for the RTL code, and falsified
The components of the prompt are illustrated below: assertions are sent to the next step, i.e., contextualization. We
s y s t e m _ p r o m p t = " You a r e a h a r d w a r e s e c u r i t y e x p e r t . Your
use VC Formal Property Verification (FPV) [29] for this.
t a s k i s t o a n a l y z e V e r i l o g c o d e f o r p o t e n t i a l CWE−<x> For CWE 1231, we check for a signal containing lock bits
b u g s . CWE−<x> i s < d e s c r i p t i o n o f CWE> . "
that are modifiable when they should not be. From AssetID,
u s e r _ p r o m p t = " What a r e t h e < r e l e v a n t s i g n a l s >? < T y p i c a l we obtain the lock signal and the conditions under which it
n a t u r e o f s u c h s i g n a l s > " + <RTL s o u r c e code > .
should be modified (the negation captures conditions when
i f v a r i a t i o n == ' v1 ' : it should not be modifiable). For each lock signal and the
u s e r _ p r o m p t = " < e x a m p l e o f CWE i n RTL> + < e x p l a n a t i o n
of s e c u r i t y issue > + < a s s e t s i d e n t i f i e d in t h i s
corresponding conditions, we form the following template and
case >" + user_prompt populate it with the appropriate information:
@( [ CLK_SENSE ] [CLK ] )
Asset identification of the motivating examples is shown [ CONDITIONS_FOR_STABLE_LOCK ] | => $ s t a b l e ( [ LOCK_SIGNAL ] ) ;
in Fig. 2(b) and Fig. 3(b). The relevant access control
signals for CWE-1191 example {pass_data,data_d} are For CWE 1233, we check for a security-sensitive signal that
correctly identified along with unrelated signals. For CWE- is missing lock bit protection. From AssetID, we obtain the
1233, the relevant security sensitive signals {start_reg security-sensitive register that should be protected, the lock
... core_lock_reg} and their “expected” lock signals signal that should be protecting it, and the reset conditions
{reglk_ctrl[*]} are identified. The reset conditions are also under which this protection mechanism is not applicable. For
identified for assertion formation in the next step. each lock signal, we form and populate the following template:
@( [ CLK_SENSE ] [CLK ] )
B. Static Analysis d i s a b l e i f f ( [ RESET_CONDITIONS ] ) [ LOCK_SIGNAL ] == ' 1 |= >
$ s t a b l e ( [ SECURITY_SENSITIVE_REGISTER ] ) ;
Depending on the CWE, we decide whether a linting-based
or assertion-based strategy is more appropriate. If the CWE Fig. 3(c) captures an example of this assertion formation.

2
 Form tcl script
2 Lint with relevant Linting
lint checks Linting violations
Tool
Lint checks or Lint violations or
Assertions? Falsified assertions?
Form
Assertions assertion
CWE Info template Formal Falsified
Tool Assertions

1 Assets 3
Relevant to Location of bug
Buggy Verified issues
RTL CWE
pertaining to CWE
LLM Explanation of Bug
LLM

Fig. 1. LASHED framework. 1) Assets Identification (AssetID): LLM identifies assets relevant to a CWE in the RTL. 2) Static Analysis: Depending on
CWE, a linting or assertion-based strategy is used. A linter or formal property verification tool checks for the lint tags identified or assertions formed. 3)
Contextualization: LLM reasons and prunes linting violations or falsified properties. LLM gives an explanation if there is a security issue.

1 i f ( pass_mode ) b e g i n
2 p a s s _ d a t a = { { 6 0 { 8 ' h00 } } , d a t a _ d } ; 1 input l o g i c [7 : 0 ] r e g l k _ c t r l _ i ; / / r e g i s t e r lock values
3 s t a t e _ d = PassChk ; 2 ...
4 pass_mode = 1 ' b0 ; 3 e l s e i f ( en && we )
4 case ( address [ 7 : 3 ] )
5 0 : s t a r t _ r e g <= w d a t a ;
(a) CWE 1191 in JTAG access control mechanism. Only the least 6 ...
significant 32 bits of the secret message are used for authentication. 7 8 : e n d _ r e g <= w d a t a ;

" access_control_related_signals " : [ " pass_data " , (a) CWE 1233 in Direct Memory Access wrapper. Security sensitive
" p a s s _ h a s h " , " exp_hash " , " hmac_ready " , " d a t a _ d " registers are missing lock bit protection.
... ] ,

" relevant −signals " : {

(b) Assets identification using gpt-4o-mini, prompt variation v0. " security_sensitive_signals_info " : [
{" lock_signal " : " reglk_ctrl_i [0] " ,
" security_sensitive_signal " : " start_reg "} ,
" relevant_violations " : [{ ...
" l i n e _ n o " : 158 , {" lock_signal " : " reglk_ctrl_i [7] " ,
" s t a t e m e n t " : " p a s s _ d a t a = { { 6 0 { 8 ' b0 } } , d a t a _ d } ; " , " security_sensitive_signal " : " core_lock_reg "} ,
" lhsexpr " : " pass_data " , " r e s e t _ c o n d i t i o n s " : " ~ ( r s t _ n i && ~ r s t _ 8 ) " , . . . }
" security_sensitive_signal " : " pass_data " }]

(b) Assets identification using gpt-4o-mini, prompt variation v0.

(c) Lint violation using VC SpyGlass Lint and Verific.

@( p o s e d g e c l k _ i )
" insecure " : true , d i s a b l e i f f ( ~ ( r s t _ n i && ~ r s t _ 8 ) ) r e g l k _ c t r l _ i [ 7 ] == ' 1
" e x p l a n a t i o n " : The a s s i g n m e n t t o ' p a s s _ d a t a ' d e f i n e s a 2 |= > $ s t a b l e ( c o r e _ l o c k _ r e g ) ;
56− b i t wide d a t a s i g n a l ( w i t h 6 0 z e r o s p r e p e n d e d t o
' data_d ' ) , . . . I f u n a u t h o r i z e d access i s not
c o n t r o l l e d b e f o r e ' p a s s _ d a t a ' i s e v a l u a t e d o r used , (c) Assertion formation for core_lock_reg signal in DMA wrapper using
an a t t a c k e r c o u l d s e n d m a l i c i o u s d a t a t o t h e d e v i c e
VC Formal Property Verification. This assertion was falsified.
and compromise i n t e n d e d s e c u r i t y o p e r a t i o n s , making
i t a p o t e n t i a l v i o l a t i o n of proper access c o n t r o l
i n l i n e w i t h CWE 1 1 9 1 . " insecure " : true ,
" e x p l a n a t i o n " : The s i g n a l ' c o r e _ l o c k _ r e g ' i s managed
b u t c a n be s e t t o z e r o i n a d v e r t e n t l y , t h e r e b y
(d) Contextualization using gpt-4o-mini variation v0. a l l o w i n g m o d i f i c a t i o n of t h e system ' s i m p o r t a n t
r e g i s t e r s . I t lacks the necessary s t a b i l i t y to
Fig. 2. Motivating example for CWE 1191 to illustrate LASHED flow for prevent unauthorized access .
linter-based strategy.
(d) Contextualization using gpt-4o-mini variation v0.

For CWE 1244, we check for a privilege level signal that Fig. 3. Motivating example for CWE 1233 to illustrate LASHED flow for
assertion based strategy.
is escalated under conditions it should not have been. From
AssetID, we obtain the privilege level signal, the correct
conditions under which it should be escalated, the reset
conditions of the module, the higher privilege level, and the escalation conditions provides the conditions under which the
signal containing the previous privilege level. A negation of privilege signal should not be escalated. For this privilege

3
signal, we form and populate the following assertion template: TABLE II
CWE S COVERED BY LASHED. S ELECTED FROM MITRE’ S LIST OF
@( [ CLK_SENSE ] [CLK ] )
M OST I MPORTANT H ARDWARE CWE S .
d i s a b l e i f f ( [ RESET_CONDITIONS ] )
~ [ CONDITIONS_FOR_PRIVILEGE_ESCALATION ] |= >
( [ PRIVILEGE_SIGNAL ] ! = [ HIGH_PRIVILEGE ] | | CWE Description
[ PRIVILEGE_SIGNAL ] == [ PREVIOUS_PRIVILEGE ] ) ;
1191 On-Chip Debug and Test Interface With Improper Access Control
C. Contextualization 1231 Improper Prevention of Lock Bit Modification

First, the LLM reasons whether the reported linting vio- 1233 Security-Sensitive Hardware Controls with Missing Lock Protection
lations or falsified properties pose a security issue pertinent 1244 Internal Asset Exposed to Unsafe Debug Access Level or State
to the CWE under consideration. If it reasons that there is 1300 Improper Protection of Physical Side Channels
a security issue, the LLM is prompted to explain why. The
explanation and static analysis violation are the final outputs
of LASHED provided to the RTL designer. The components MITRE’s website and were present in the H@DAC-21 SoC to
of the prompt to the LLM are illustrated below: have some ground-truth for validation of initial prototyping.
s y s t e m _ p r o m p t = <same a s A s s e t s I d e n t i f i c a i t o n > The 5 CWEs covered are described in Table II.
u s e r _ p r o m p t = " C o n s i d e r t h e f o l l o w i n g V e r i l o g c o d e : <RTL
s o u r c e code > F o r e a c h o f t h e < s t a t i c a n a l y s i s o u t p u t s > , C. Prompt Variations
determine whether the <output > poses a s e c u r i t y i s s u e
p e r t a i n i n g t o CWE−<x> and p r o v i d e an e x p l a n a t i o n i f The performance of LLMs is dependent on the quality of
t h a t i s the case . I f the v i o l a t i o n does not pose a
s e c u r i t y i s s u e , no e x p l a n a t i o n i s n e e d e d . Here i s t h e
prompts and examples of the correct solutions to the task at
o u t p u t < o u t p u t from S t a t i c A n a l y s i s > . " hand. To study the extent to which in-context learning [34] and
insistence on reasoning helps in LASHED’s performance, we
For experiments where we adopt prompt strategy v2 (details guide the LLM through 4 prompt variations. The 4 variations
in Section III-C), we use the response of the LLM from the are formed using combinations of 2 improvements. The first
first contextualization request and re-prompt the LLM to rea- improvement helps in the Assets Identification and Contex-
son through each of its suggested security issues, categorizing tualization phases by providing a comprehensive example of
the violations as insecure only if very confident. Fig. 2(d) a hardware security bug that captures the CWE. The second
and Fig. 3(d) show examples of contextualization. improvement helps in the Contextualization phase by asking
III. E XPERIMENTAL D ETAILS the LLM to think again about its initial assessments of whether
the reported violation poses a security risk.
A. Dataset
1) Variation v0 (baseline): is the zero-shot implementation
Our dataset consists of 4 open-source RISC-V based SoCs: for LASHED. It contains no in-context learning or request
Hack@DAC 2021’s OpenPiton buggy SoC (H@DAC-21) [30], to re-evaluate outputs and forms the baseline to compare
OpenTitan [31], Hummingbirdv2 E203 (E203) [32] and Veer- improvements in performance with variations v1, v2 and v3.
Wolf [33]. Their details are mentioned in Table I. 2) Variation v1: uses the first improvement only. The exam-
ples and descriptions of bugs for each CWE are taken from the
TABLE I MITRE’s website. Each example appears in the prompt after
DATASET OF OPEN - SOURCE S O C S SCANNED FOR RELEVANT CWE S .
the LLM is given its role and information about the CWE it
SoC Description #Mods #LoCs is going to look for. It consists of the bug in RTL form, the
explanation of the security issues because of the bug and the
OpenPiton SoC (CVA6 core) for relevant security assets. Here is an example for the prompt
H@DAC-21 [30] 63 15k
Hack@DAC 2021 competition
variation v1 appended to the baseline v0 for CWE 1231 (full
OpenTitan [31] Silicon Root of Trust project (Ibex core) 359 171k
example in Appendix Section B):
E203 [32] Hummingbirdv2 E203 core and SoC 76 27k
VeerWolf [33] FuseSoC-based platform for VeeR cores 34 11k Here is an example of CWE-1231 with code from the register
locks module:
always @(posedge clk_i) begin
B. Hardware Common Weakness Enumerations (CWEs) if( (rst_ni && jtag_unlock && rst_9)) begin
Security-related issues that arise because of hardware for (j=0; j < 6; j=j+1) begin
bugs are taxonomized as Common Weakness Enumerations reglk_mem[j] <= ’h0;
(CWEs). MITRE [27] works with academia and industry to <Explanation of the security issue>
develop a list of CWEs that represent categories of vulnerabil- In this example the lock signal is reglk_mem and the cor-
ities. A weakness is an element in a digital product’s software, rect conditions for changing lock signals are (rst_ni &&
firmware, hardware, or service that can be exploited for ma- jtag_unlock).
licious purposes. We develop LASHED for 5 CWEs selected
from the list of Most Important Hardware CWEs published 3) Variation v2: uses the second improvement only. The
by MITRE. We selected the ones that had coded examples on model is “given time to think” so that it can double check

4
its initial assessment. It is asked to use inner monologue to TABLE III
go over its reasoning process. It looks the same for all CWEs R ESULTS S UMMARY. P RECISION = T RUE P OSITIVES (TP S ) / F LAGGED ,
FALSE D ISCOVERY R ATE (FDR) = FALSE P OSITIVES / F LAGGED , A SSETS
and appears in the user prompt for Contextualization. First, = NUMBER OF ASSETS IDENTIFIED , A SSERTIONS = NUMBER OF
the LLM is asked to go over the violations and assess which ASSERTIONS FORMED FROM CUSTOM TEMPLATES FOR EACH CWE. T HE
of them are actually insecure and which are not. This output BEST RESULTS ARE EMBOLDENED .

is then sent back to the LLM to simulate a chat. An example

CWE Variation Flagged TPs Precision FDR Assets Assertions
of the instruction to ‘reason’ for CWE-1231 is shown below:
v0 12 12 1.00 0.00 511 -
v1 11 11 1.00 0.00 580 -
Go over the previously provided response and reason about 1191
v2 13 13 1.00 0.00 511 -
the provided explanation for each falsified property. Only cat- v3 9 9 1.00 0.00 580 -
egorize the falsified property as insecure if you are confident 45 45 1.00 0.00 2182 -
in your assessment. Here is the ‘falsified_properties’ object:
v0 15 3 0.20 0.80 33 33
<string of falsified assertions information> v1 8 6 0.75 0.25 11 11
1231
v2 24 2 0.08 0.92 33 33
4) Variation v3: uses both, first and second improvements. v3 6 4 0.67 0.33 11 11
53 15 0.28 0.72 88 88
D. Large Language Models (LLMs)
v0 143 55 0.38 0.62 669 587
We use 2 OpenAI LLMs, gpt-4o-mini-2024-07-18 and gpt- v1 114 48 0.42 0.58 406 355
4o-2024-08-06 [35], to conduct out experiments and evaluate 1233
v2 68 35 0.51 0.49 669 587
if using a more powerful LLM makes a difference for our v3 89 50 0.56 0.44 406 355
setup. gpt-4o-mini is OpenAI’s most advanced model in the 414 188 0.45 0.55 2150 1884
small models category. gpt-4o is OpenAI’s most advanced v0 5 3 0.60 0.40 24 12
GPT model and is slower and more advanced than gpt-4o. 1244
v1 3 3 1.00 0.00 22 15
Both these models have the ability to give structured outputs v2 1 1 1.00 0.00 24 12
v3 1 1 1.00 0.00 22 15
consistent with the object structure provided.
10 8 0.80 0.20 92 54
IV. R ESULTS
v0 8 7 0.88 0.13 2053 -
We evaluated LASHED on the 4 SoCs for 5 CWEs, 4 1300
v1 9 8 0.89 0.11 1968 -
prompt variations and 2 LLMs. The results are summarized v2 3 3 1.00 0.00 2053 -
v3 3 3 1.00 0.00 1968 -
in Table III. In total, 545 instances are flagged across the
23 21 0.91 0.09 8042 -
160 experiments out of which 51% are potential CWEs.
2026 assertions were formed and 12,554 assets were checked 545 277 0.508 0.49 12554 2026
in the process. Since we do not know the real number of
bugs (we can only confirm or deny a specific bug’s presence
after flagging), it is not possible to calculate a proper Recall A. Analysis
or Accuracy score. Therefore, we rely on Precision i.e. (#
true positives / # predicted positive (flagged)) as the metric 1) Impact of Prompt Variations: Prompt variations were
to evaluate performance. For our recommended combination successful in improving performance. As shown in Fig. 4-
of using gpt-4o with v3, 35 of the 40 flagged instances (Variation), variation v1 which introduces an example of CWE
are plausible CWEs, providing a precision of 87.5%. On to guide the LLM, results in finding less false positives. The
average, for a given SoC being searched for a particular number of true positives decreases slightly from 80 to 76
CWE, choosing the appropriate prompt variation, there are 3.4 while the precision improves from 0.44 to 0.52 in comparison
violations reported. We evaluated the violations manually, with to the baseline v0. Introducing v2, which asks the LLM to
our author-confirmed violations providing the True Positives reason about its initial assessment during contextualization,
count in Table III. shows improvement as well but to a lesser extent. The number
There is significant variation in LASHED’s success based of true positives decreases from 80 to 54 but the precision
on the CWE. It performs the best on CWEs 1191, 1244 and increases from 0.44 to 0.50 in comparison to the baseline v0.
1300 with precisions of 1, 0.8 and 0.91 and worse on CWEs v2 significantly reduced false positives from 103 to 55. The
1231 and 1233 with precisions of 0.28 and 0.45. CWE 1231 best performing variation is v3 with highest precision of 0.62,
was harder to identify because of poor asset identification. It a 42% improvement over the baseline. As v3 is a combination
was difficult for the LLM to identify the lock bit signal that of v1 and v2, it seems that v1 can improve the ‘search’ of
should remain stable and instead kept identifying control and LASHED by identifying assets that are more relevant to the
status registers. CWE 1233 was harder to accurately identify CWE and then v2 prunes out the remaining false positives.
because of the range of possibilities of how a security sensitive This can be seen in Fig. 5-(Variation) with an increase in
signal may be protected in RTL. It may be protected by an if removal of violations in contextualization for v3.
condition or by being ‘anded’ (&) with another signal or by 2) Which SoCs were better analyzed?: LASHED’s perfor-
being assigned a signal which may have some protection. mance significantly varies with the SoC as illustrated in Fig. 4-

5
 True Positives (tp) False Positives (fp)
H@DAC-21 OpenTitan e203 20
Veerwolf All SoCs, LLMs All Vars, LLMsAll Vars, LLMs
60 20 40 300
15 150 200
# flagged

40 15 30
100 200
10 20 10
100
20 5 5 50 100
10
0 0 0 0 0 0 0
v0(m)
v0
v1(m)
v1
v2(m)
v2
v3(m)
v3

v0(m)
v0
v1(m)
v1
v2(m)
v2
v3(m)
v3

v0(m)

v3(m)

H@DAC
v0
v1(m)
v1
v2(m)
v2
v3

v0(m)
v0
v1(m)
v1
v2(m)
v2
v3(m)
v3

v0
v1
v2
v3

OTitan
e203
Vwolf

4o-mini
4o
Variation(LLM) Variation(LLM) Variation(LLM) Variation(LLM) Variation SoC LLM

Fig. 4. Classification of instances flagged by LASHED. Stacked bar shows the number of instances identified by LASHED that pose security issues as true
positives (tp) and those that do not as false positives (fp). The numbers are summed up for all CWEs.

# assets identified # assertions formed # violations removed in contextualization

H@DAC-21 OpenTitan e203 Veerwolf All SoCs, LLMs104 All Vars, LLMs All Vars, LLMs
103 102 102
# instances

103 103
102 102 103
101 101
101 102
102
101 100 100 102
v0(m)

v1(m)

v0(m)

H@DAC
v0

v1
v2(m)
v2
v3(m)
v3

v0
v1(m)
v1
v2(m)
v2
v3(m)
v3
v0(m)
v0
v1(m)
v1
v2(m)
v2
v3(m)
v3
v0(m)
v0
v1(m)
v1
v2(m)
v2
v3(m)
v3
v0
v1
v2
v3

OTitan
e203
Vwolf
4o-mini

4o
Variation(LLM) Variation(LLM) Variation(LLM) Variation(LLM) Variation SoCs LLM

Fig. 5. Intermediary outputs during LASHED’s operation. # assets identified is number of security relevant signals identified during Assets Identification, #
assertions formed is the number of assertions formed during Static Analysis. # violations removed in contextualization is the number of violations that the
LLM reasoned as not posing a security threat. The numbers are summed up for all CWEs.

(SoC). It performed the best on e203 with a precision of 0.61 is no verification. Conversely, detecting vulnerabilities with
and the worst on Veerwolf with a precision of 0.10. The main formal tools requires a lot of expertise and precise information
culprit for poor performance on Veerwolf is the large number which can be hard to obtain. We demonstrate that the limited
of false positives for CWE-1231. LASHED kept misidentify- confidence on the outputs of LLMs can be improved by
ing a lot of the control and status registers as lock registers. verification through static analysis tools and the requirement
Hack@DAC-21 performs the second best with a precision of of specific information can be provided by the LLM to some
0.55, which is expected because the guiding examples we use extent. The precision of 0.51 for LASHED (for all experiments
in v1 are inspired from the Hack@DAC-21 SoC. OpenTitan combined) can be interpreted as a 51% confidence in the
has a precision close to that of Hack@DAC-21, 0.50, which identification of a security issue.
validates our approach – OpenTitan is significantly larger than
Hack@DAC-21 in size and complexity. Linting checks and assertions are very different in their
Another area LASHED had performance issues in with nature. Lint rules are harder to map to security issues and
Veerwolf and OpenTitan came with the forming of meaningful assertions are harder to form correctly. Which of these work
assertions. The ratio of assertion formation to number of assets better in our flow is still an open question. The lint checks
identified for Hack@DAC-21 is significantly larger, as shown that appeared often in flagged instances were related to
in Fig. 5-(SoC). This is perhaps due to the guiding examples improper range indexes, concatenations and ‘if’ statements
being based on Hack@DAC-21. missing ‘else’ statements. While the confidence of reported
3) Which LLM performed better?: LASHED performed violations is high, there are more false positives than would
better with gpt-4o (precision 0.58) compared to gpt-40-mini be ideal. The main culprit is liberal asset identification by
(precision 0.47). gpt-4o flags less violations 204 vs. 341, less the LLM. Providing the CWE information is not enough to
true positives 118 vs. 159, but also, less false positives 86 constrain the number of assets the LLM identifies. Providing
vs. 182, than gpt-4o-mini. This difference is highlighted when information regarding the operation of the specific modules
considering v3 only: gpt-4o detects more true positives 35 vs. and the security objectives could improve results. Another
32 and has a higher precision 0.88 vs. 0.47. reason for false positives is the difficulty in forming correct
assertions. Incorrect conditions identified by the LLM lead to
V. D ISCUSSION failing assertions which flag violations erroneously. Another
Our work shows that a combination of LLMs and static limitation lies in the manual evaluation of flagged violations.
analysis can provide the ‘best of both worlds’, with each tool We assessed all violations through visual inspection which has
helping to overcome the limitations of the other. Detecting a possibility of being incorrect. Each violation, however, is
vulnerabilities with only LLMs leads to a large number of false accompanied by a failing assertion or violation which gives
positives and limited confidence on the outputs because there credence to the classification.

6
 VI. R ELATED P RIOR W ORKS [4] J. He, X. Guo, T. Meade, R. Dutta, Y. Zhao, and Y. Jin, “SoC
interconnection protection through formal verification,” Integration,
Few works have explored using both LLMs and Static vol. 64, pp. 143–151, Jan. 2019. [Online]. Available: [Link]
Analysis components to identify RTL security bugs. So- [Link]/science/article/pii/S016792601830289X
CureLLM [12] is an LLM-driven approach for large-scale [5] J. Hur, S. Song, D. Kwon, E. Baek, J. Kim, and B. Lee, “DifuzzRTL:
Differential Fuzz Testing to Find CPU Bugs,” in 2021 IEEE Symposium
System-on-Chip security verification and policy generation. on Security and Privacy (SP), May 2021, pp. 1286–1303, iSSN: 2375-
While they use LLMs for security policy generation and secu- 1207.
rity policy violation, they do not use static tools for security [6] T. Trippel, K. G. Shin, A. Chernyakhovsky, G. Kelly, D. Rizzo,
and M. Hicks, “Fuzzing Hardware Like Software,” 2022, pp.
violation in their solution. Flag-RTL [11] uses LLMs for bug 3237–3254. [Online]. Available: [Link]
detection in RTL but uses static analysis differently. A front usenixsecurity22/presentation/trippel
end parser localizes the LLM’s search to particular parts of the [7] A. Ardeshiricham, W. Hu, J. Marxen, and R. Kastner, “Register transfer
code, but no static analysis tool is used for verification. Self- level information flow tracking for provably secure hardware design,”
in Design, Automation Test in Europe Conference Exhibition (DATE),
HWDebug [13] automates LLM self-instructing for hardware 2017, Mar. 2017, pp. 1691–1696, iSSN: 1558-1101.
security verification. This work uses known bugs and their [8] W. Hu, A. Ardeshiricham, and R. Kastner, “Hardware Information Flow
CWEs to generate instructions for debugging and repair and Tracking,” ACM Computing Surveys, vol. 54, no. 4, pp. 83:1–83:39,
May 2021. [Online]. Available: [Link]
there is no use of static analysis. While these works are related [9] B. Ahmad, W.-K. Liu, L. Collini, H. Pearce, J. M. Fung, J. Valamehr,
to LASHED, there are significant differences which do not M. Bidmeshki, P. Sapiecha, S. Brown, K. Chakrabarty, R. Karri, and
allow for a direct comparison. None of the works use LLMs B. Tan, “Don’t CWEAT It: Toward CWE Analysis Techniques in Early
Stages of Hardware Design,” in Proceedings of the 41st IEEE/ACM
and Static Analysis together in equal levels of significance and International Conference on Computer-Aided Design, ser. ICCAD ’22.
none explore their tools on unknown bugs. New York, NY, USA: Association for Computing Machinery, Dec. 2022,
pp. 1–9. [Online]. Available: [Link]
VII. C ONCLUSION [10] M. M. Bidmeshki, Y. Zhang, M. Zaman, L. Zhou, and Y. Makris,
This work combines LLMs and Static Analysis for hardware “Hunting Security Bugs in SoC Designs: Lessons Learned,” IEEE
Design & Test, vol. 38, no. 1, pp. 22–29, Feb. 2021.
security bug detection. LASHED, by using LLMs equipped [11] B. Ahmad, B. Tan, R. Karri, and H. Pearce, “FLAG: Finding Line
with in-context learning and requests for ‘thinking again’, Anomalies (in code) with Generative AI,” Jun. 2023, arXiv:2306.12643
has reasonable success in finding unknown bugs with an [cs]. [Online]. Available: [Link]
[12] S. Tarek, D. Saha, S. K. Saha, M. Tehranipoor, and F. Farahmandi,
empirical precision of 0.88. On average, LASHED reported “SoCureLLM: An LLM-driven Approach for Large-Scale System-on-
3.4 violations per CWE for a given SoC, out of which 1.7 were Chip Security Verification and Policy Generation,” 2024, publication
evaluated to be plausible security issues. This is demonstrated info: Preprint. [Online]. Available: [Link]
[13] M. Akyash and H. M. Kamali, “Self-HWDebug: Automation of
over 5 of the most important Hardware CWEs, with the best LLM Self-Instructing for Hardware Security Verification,” May 2024,
performance on CWE 1191 (100% precision) and worst on arXiv:2405.12347. [Online]. Available: [Link]
CWE 1231 (28% precision). We catered the static analysis [14] V. S. Lint, “Synopsys VC SpyGlass Lint,”
2022. [Online]. Available: [Link]
approach to the nature of the CWEs to show 2 techniques static-and-formal-verification/vc-spyglass/[Link]
that do similarly well i.e., linting and assertions. Future work [15] jasperlint, “Jasper Superlint App,” 2022. [Online].
could investigate the application of reasoning models such as Available: [Link]
OpenAI o1 not available during our experimentation, as well system-design-and-verification/formal-and-static-verification/
jasper-gold-verification-platform/[Link]
as examine support for more CWEs. [16] “VC Formal,” 2022. [Online]. Available: [Link]
verification/static-and-formal-verification/[Link]
VIII. ACKNOWLEDGMENTS [17] Cadence, “Jasper RTL Apps | Cadence,” Jul. 2022.
This research work is supported in part by a gift from Intel [Online]. Available: [Link]
tools/system-design-and-verification/formal-and-static-verification/
Corporation. This work does not in any way constitute an [Link]
Intel endorsement of a product or supplier. We thank Verific [18] D. Hansson, “Continuous Linting with Automatic Debug,” in 2014 15th
Design Automation for generously providing academic access International Microprocessor Test and Verification Workshop, Dec. 2014,
to linkable libraries, examples, and documentation for their pp. 70–72, iSSN: 2332-5674.
[19] A. McNutt and G. Kindlmann, “Linting for Visualization: Towards a
RTL parsers. Practical Automated Visualization Guidance System,” 2018.
[20] J. Woodcock, P. G. Larsen, J. Bicarregui, and J. Fitzgerald,
R EFERENCES “Formal methods: Practice and experience,” ACM Comput. Surv.,
[1] G. Dessouky, D. Gens, P. Haney, G. Persyn, A. Kanuparthi, vol. 41, no. 4, pp. 19:1–19:36, Oct. 2009. [Online]. Available:
H. Khattri, J. M. Fung, A.-R. Sadeghi, and J. Rajendran, [Link]
“HardFails: Insights into Software-Exploitable Hardware Bugs,” 2019, [21] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan,
pp. 213–230. [Online]. Available: [Link] H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri,
usenixsecurity19/presentation/dessouky G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan,
[2] S. Mitra, S. A. Seshia, and N. Nicolici, “Post-silicon validation S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian,
opportunities, challenges and recent advances,” in Proceedings of the C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis,
47th Design Automation Conference, ser. DAC ’10. New York, NY, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak,
USA: Association for Computing Machinery, Jun. 2010, pp. 12–17. J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse,
[Online]. Available: [Link] A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford,
[3] S. Ray, N. Ghosh, R. Masti, A. Kanuparthi, and J. Fung, “INVITED: M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew,
Formal Verification of Security Critical Hardware-Firmware Interactions D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, “Evaluating
in Commercial SoCs,” in 2019 56th ACM/IEEE Design Automation Large Language Models Trained on Code,” Jul. 2021, arXiv:2107.03374
Conference (DAC), Jun. 2019, pp. 1–4, iSSN: 0738-100X. [cs]. [Online]. Available: [Link]

7
[22] S. Thakur, B. Ahmad, H. Pearce, B. Tan, B. Dolan-Gavitt, the prompt after the LLM is given its role and information
R. Karri, and S. Garg, “VeriGen: A Large Language Model for about the CWE it is going to look for. It consists of the bug
Verilog Code Generation,” ACM Trans. Des. Autom. Electron. Syst.,
vol. 29, no. 3, pp. 46:1–46:31, Apr. 2024. [Online]. Available: in RTL form, the explanation of the security issues because of
[Link] the bug and the relevant security assets. Here is an example
[23] B. Ahmad, S. Thakur, B. Tan, R. Karri, and H. Pearce, “On Hardware for the prompt variation v1 appended to the baseline v0 for
Security Bug Code Fixes by Prompting Large Language Models,”
IEEE Transactions on Information Forensics and Security, vol. 19, CWE 1231.
pp. 4043–4057, 2024. [Online]. Available: [Link]
abstract/document/10462177 Here is an example of CWE-1231 with code from the register
[24] Z. Li, S. Dutta, and M. Naik, “LLM-Assisted Static Analysis for
Detecting Security Vulnerabilities,” Nov. 2024, arXiv:2405.17238. locks module:
[Online]. Available: [Link] """
[25] P. J. Chapman, C. Rubio-González, and A. V. Thakur, “Interleaving always @(posedge clk_i) begin
Static Analysis and LLM Prompting,” in Proceedings of the 13th ACM
SIGPLAN International Workshop on the State Of the Art in Program if( (rst_ni && jtag_unlock && rst_9)) begin
Analysis. Copenhagen Denmark: ACM, Jun. 2024, pp. 9–17. [Online]. for (j=0; j < 6; j=j+1) begin
Available: [Link] reglk_mem[j] <= ’h0;
[26] H. Li, Y. Hao, Y. Zhai, and Z. Qian, “Enhancing Static Analysis for
Practical Bug Detection: An LLM-Integrated Approach,” Enhancing """
Static Analysis for Practical Bug Detection: An LLM-Integrated Register locks help prevent SoC peripherals’ registers from
Approach (Artifact), vol. 8, no. OOPSLA1, pp. 111:474–111:499, Apr. malicious use of resources. The registers that can potentially
2024. [Online]. Available: [Link]
[27] T. M. C. (MITRE), “CWE - CWE Most Important Hardware leak secret data are locked by register locks. In the vulnerable
Weaknesses,” 2022. [Online]. Available: [Link] code, the reglk_mem is used for locking information. If
lists/2021_CWE_MIHW.html one of its bits toggle to 1, the corresponding peripheral’s
[28] Verific, “Verific Design Automation,” 2022. [Online]. Available:
[Link] registers will be locked. A critical issue arises within the
[29] V. Formal, “VC Formal: Formal Verification Solution | Synopsys,” reset controller module. Specifically, the reset controller can
2024. [Online]. Available: [Link] inadvertently transmit a peripheral reset signal to the register
static-and-formal-verification/[Link]
[30] “HACK-EVENT/hackatdac21,” Apr. 2024, original-date: 2023-07- lock within the user privilege [Link] unintentional
15T20:58:02Z. [Online]. Available: [Link] action can result in the reset of the register locks, potentially
hackatdac21 exposing private data from all other peripherals, rendering
[31] lowRISC contributors, “Open source silicon root of trust (RoT) |
OpenTitan,” 2023. [Online]. Available: [Link] them accessible and readable. In this example the lock signal
[32] N. S. Technology, “Hummingbirdv2 E203 Core and SoC - GitHub,” is reglk_mem and the correct conditions for changing lock
May 2022, original-date: 2020-07-29T06:28:49Z. [Online]. Available: signals are (rst_ni && jtag_unlock).
[Link]
[33] chipsalliance, “VeeRwolf,” Nov. 2024, original-date: 2019-08-
07T15:24:36Z. [Online]. Available: [Link]
VeeRwolf
[34] Y. Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan,
and J. Ba, “Large Language Models Are Human-Level Prompt
Engineers,” Nov. 2022, arXiv:2211.01910 [cs]. [Online]. Available:
[Link]
[35] OpenAI, “GPT-4o,” May 2024. [Online]. Available: [Link]
index/hello-gpt-4o/

A PPENDIX
A. Scalability and Cost
On average, each experiment took 163 seconds to run. This
time includes the complete flow of running the LLM and
static analysis tools from the identification of relevant RTL
to the outputs of location and explanation of bugs. In total,
160 experiments were run in 7.26 hours. There is a relation
between the lines of code and the time taken. The dependency
is linear in the log-log scale and produces follows the relation
time ∝ loc0.53 approximately. This shows that the time taken
grows proportional to the square root of the amount of code
being analyzed, indicating a scalable approach. On average,
each experiment cost $0.055 while using gpt-4o-mini and
$0.379 while using gpt-4o. In total all experiments cost $35
to run, catering to 40.2M input tokens.
B. Propmpt variation v1
The examples and descriptions of bugs for each CWE are
taken from the MITRE’s website. Each example appears in