Expected Exploitability: Predicting the

Development of Functional Vulnerability Exploits

Octavian Suciu, University of Maryland, College Park;
Connor Nelson, Zhuoer Lyu, and Tiffany Bao, Arizona State University;
Tudor Dumitraș, University of Maryland, College Park
[Link]

This paper is included in the Proceedings of the

31st USENIX Security Symposium.
August 10–12, 2022 • Boston, MA, USA
978-1-939133-31-1

Open access to the Proceedings of the

31st USENIX Security Symposium is
sponsored by USENIX.
 Expected Exploitability: Predicting the Development of Functional Vulnerability
Exploits

Octavian Suciu, Connor Nelson† , Zhuoer Lyu† , Tiffany Bao† , Tudor Dumitras,
University of Maryland, College Park
† Arizona State University

Abstract efforts in the industry, to make optimal decisions in the gov-

Assessing the exploitability of software vulnerabilities at ernment’s Vulnerabilities Equities Process [27], and to gain a
the time of disclosure is difficult and error-prone, as features deeper understanding of the research opportunities to prevent
extracted via technical analysis by existing metrics are poor exploitation, we must evaluate each vulnerability’s ease of
predictors for exploit development. Moreover, exploitability exploitation.
assessments suffer from a class bias because “not exploitable” Despite significant advances in defenses [62], exploitability
labels could be inaccurate. assessments remain elusive because we do not know which
To overcome these challenges, we propose a new metric, vulnerability features predict exploit development. For exam-
called Expected Exploitability (EE), which reflects, over time, ple, expert recommendations for prioritizing patches [47, 48]
the likelihood that functional exploits will be developed. Key initially omitted CVE-2017-0144, the vulnerability later ex-
to our solution is a time-varying view of exploitability, a ploited by WannaCry and NotPetya. While one can prove
departure from existing metrics. This allows us to learn EE exploitability by developing an exploit, it is challenging to
using data-driven techniques from artifacts published after establish non-exploitability, as this requires reasoning about
disclosure, such as technical write-ups and proof-of-concept state machines with an unknown state space and emergent
exploits, for which we design novel feature sets. instruction semantics [16]. This results in a class bias of
This view also allows us to investigate the effect of the exploitability assessments, as we cannot be certain that a “not
label biases on the classifiers. We characterize the noise- exploitable” label is accurate.
generating process for exploit prediction, showing that our We address these two challenges through a metric called
problem is subject to the most challenging type of label noise, Expected Exploitability (EE). Instead of deterministically la-
and propose techniques to learn EE in the presence of noise. beling a vulnerability as “exploitable” or “not exploitable”,
On a dataset of 103,137 vulnerabilities, we show that EE our metric continuously estimates over time the likelihood
increases precision from 49% to 86% over existing metrics, that a functional exploit will be developed, based on historical
including two state-of-the-art exploit classifiers, while its patterns for similar vulnerabilities. Functional exploits go
precision substantially improves over time. We also highlight beyond proof-of-concepts (POCs) to achieve the full secu-
the practical utility of EE for predicting imminent exploits and rity impact prescribed by the vulnerability. While functional
prioritizing critical vulnerabilities. exploits are readily available for real-world attacks, we aim
We develop EE into an online platform which is publicly to predict their development and not their use in the wild,
available at [Link] which depends on many other factors besides exploitabil-
ity [30, 55, 73].
Key to our solution is a time-varying view of exploitabilty,
1 Introduction a departure from the existing vulnerability scoring systems
such as CVSS [42], which are not designed to take into ac-
Weaponized exploits have a disproportionate impact on se- count new information (e.g. new exploitation techniques,
curity, as highlighted in 2017 by the WannaCry [63] and leaks of weaponized exploits) that becomes available after the
NotPetya [13] worms that infected millions of computers scores are initially computed [18]. By systematically com-
worldwide. Their notorious success was in part due to the use paring a range of prior and novel features, we observe that
of weaponized exploits. The cyber-insurance industry regards artifacts published after vulnerability disclosure can be good
such contagious malware, which propagates automatically predictors for the development of exploits, but their timeli-
by exploiting software vulnerabilities, as the leading risk for ness and predictive utility varies. This highlights limitations
incurring large losses from cyber attacks [53]. At the same of prior features and a qualitative distinction between pre-
time, the rising bar for developing weaponized exploits [62] dicting functional exploits and related tasks. For example,
pushed black-hat developers to focus on exploiting only 5% prior work uses the existence of public PoCs as an exploit
of the known vulnerabilities [30]. To prioritize mitigation

USENIX Association 31st USENIX Security Symposium 377

predictor [30, 31, 64]. However, PoCs are designed to trigger for them.
the vulnerability by crashing or hanging the target application • We perform two case studies to investigate the practical
and often are not directly weaponizable; we observe that this utility of EE, showing that it can qualitatively improve
leads to many false positives for predicting functional exploits. prioritization strategies based on exploitability.
In contrast, we discover that certain PoC characteristics, such EE is available as an online platform at
as the code complexity, are good predictors, because trig- [Link]
gering a vulnerability is a necessary step for every exploit,
making these features causally connected to the difficulty of
creating functional exploits. We design techniques to extract 2 Problem Overview
features at scale, from PoC code written in 11 programming
languages, which complement and improve upon the preci- We define exploitability as the likelihood that a functional
sion of previously-proposed feature categories. We then learn exploit, which fully achieves the mandated security impact,
EE from the useful features using data-driven methods, which will be developed for a vulnerability. Exploitability reflects
have been successful in predicting other incidents, e.g. vul- the technical difficulty of exploit development, and it does not
nerabilities that are exploited in the wild [30, 55, 73], data capture the feasibility of lunching exploits against targets in
breaches [34] or website compromises [59]. the wild [30, 55, 73], which is influenced by additional factors
However, learning to predict exploitability could be de- (e.g. patching delays, network defenses, attacker choices).
railed by a biased ground truth. Although prior work had While an exploit represents conclusive proof that a vul-
acknowledged this challenge for over a decade [7, 55], no nerability is exploitable if it can be generated, proving non-
attempts were made to address it. This problem, known in the exploitability is significantly more challenging [16]. Instead,
machine-learning literature as label noise, can significantly mitigation efforts are often guided by vulnerability scoring
degrade the performance of a classifier. The time-varying systems, which aim to capture exploitation difficulty, such as:
view of exploitability allows us to uncover the root causes of 1. NVD CVSS [42], a mature scoring system with its Ex-
label noise: exploits could be published only after the data ploitability metrics intended to reflect the ease and tech-
collection period ended, which in practice translates to wrong nical means by which the vulnerability can be exploited.
negative labels. This insight allows us to characterize the The score encodes various vulnerability characteristics,
noise-generating process for exploit prediction and propose a such as the required access control, complexity of the
technique to mitigate the impact of noise when learning EE. attack vector and privilege levels, into a numeric value
In our experiments on 103,137 vulnerabilities, EE signif- between 0 and 4 (0 and 10 for CVSSv2), with 4 reflecting
icantly outperforms static exploitability metrics and prior the highest exploitability.
state-of-the art exploit predictors, increasing the precision 2. Microsoft Exploitability Index [19], a vendor-specific
from 49% to 86% one month after disclosure. Using our score assigned by experts using one of four values to
label noise mitigation technique, the classifier performance communicate to Microsoft customers the likelihood of a
is minimally affected even if evidence about 20% of exploits vulnerability being exploited [18].
is missing. Furthermore, by introducing a metric to capture 3. RedHat Severity [51], similarly encoding the difficulty
vulnerability prioritization efforts, we show that EE requires of exploiting the vulnerability by complementing CVSS
only 10 days from disclosure to approach its peak perfor- with expert assessments based on vulnerability charac-
mance. We show EE has practical utility, by providing timely teristics specific to the RedHat products.
predictions for imminent exploits, even when public PoCs are The estimates provided by these metrics are often inaccu-
unavailable. Moreover, when employed on scoring 15 criti- rate, as highlighted by prior work [2, 3, 18, 52, 56] and by our
cal vulnerabilities, EE places them above 96% of non-critical analysis in Section 5. For example, CVE-2018-8174, an ex-
ones, compared to only 49% for existing metrics. ploitable Internet Explorer vulnerability, recieved a CVSS ex-
In summary, our contributions are as follows: ploitability score of 1.6, placing it below 91% of vulnerability
• We propose a time-varying view of exploitability based scores. Similarly, CVE-2018-8440, an exploited vulnerability
on which we design Expected Exploitability (EE), a met- affecting Windows 7 through 10 was assigned score of 1.8.
ric to learn and continuously estimate the likelihood of To understand why these metrics are poor at reflecting
functional exploits over time. exploitability, we highlight the typical timeline of a vulner-
• We characterize the noise-generating process systemati- ability in Figure 1. The exploitability metrics depend on a
cally affecting exploit prediction, and propose a domain- technical analysis which is performed before the vulnerability
specific technique to learn EE in the presence of label is disclosed publicly, and which considers the vulnerability
noise. statically and in isolation.
• We explore the timeliness and predictive utility of vari- However, we observe that public disclosure is followed
ous artifacts, proposing new and complementary features by the publication of various vulnerability artifacts such as
from PoCs, and developing scalable feature extractors write-ups and PoCs containing code and additional technical

378 31st USENIX Security Symposium USENIX Association

Discovery Disclosure Publication Delay (days)
20 40 60 80 1100 2700
3 Challenges
// // //

Technical We recognize three challenges in utilizing supervised tech-

Exploitability

VD A l
analysis
Metrics niques for learning, evaluating and using EE.
W it
Write-ups Extracting features from PoCs. Prior work investigated the
Tweets existence of PoCs as predictors for exploits, repeatedly show-
T t

PoC Info ing that they lead to a poor precision [2, 31, 64]. However,
P CI f

PoCs are designed to trigger the vulnerability, a step also

P CC d

PoC Code
required in a functional exploit. As a result, the structure
Exploit development
Exploits // //
and complexity of the PoC code can reflect exploitation dif-
ficulty directly: a complex PoC implies that the functional
exploit will also be complex. To fully leverage the predictive
Figure 1: Vulnerability timeline highlighting publication de-
power of PoCs, we need to capture these characteristics. We
lay for different artifacts and the CVSS Exploitability metric.
note that while public PoCs have a lower coverage compared
The box plot delimits the 25th , 50th and 75th percentiles, and
to other artifact types, they are broadly available privately
the whiskers mark 1.5 times the interquartile range.
because they are often mandated when vulnerabilities are
reported [25].
information about the vulnerability, and social media discus- Extracting features using NLP techniques from prior ex-
sions around them. These artifacts often provide meaningful ploit prediction work [7,30,55] is not sufficient, because code
information about the likelihood of exploits. For CVE-2018- semantics differs from that of natural language. Moreover,
8174 it was reported that the publication of technical write-ups PoCs are written in different programming languages and
was a direct cause for exploit development in exploit kits [6], are often malformed programs [2, 40], combining code with
while a PoC for CVE-2018-8440 has been determined to free-form text, which limits the applicability of existing pro-
trigger exploitation in the wild within two days [76]. The ex- gram analysis techniques. PoC feature extraction therefore
amples highlight that existing metrics fail to take into account requires text and code separation, and robust techniques to
useful exploit information available only after disclosure and obtain useful code representations.
they do not update over time. Understanding and mitigating label noise. Prior work
Figure 1 plots the publication delay distribution for differ- found that the labels available for training have biases [7, 55],
ent artifacts released after disclosure, according to our data but to our knowledge no prior attempts were made to link
analysis described in Section 5. Data shows not only that these this issue to the problem of label noise. The literature dis-
artifacts become available soon after disclosure,1 providing tinguishes two models of non-random label noise, according
opportunities for timely assessments, but also that static ex- to the generating distribution: class-dependent and feature-
ploitability metrics, such as CVSS, are often not available at dependent [23]. The former assumes a uniform label flipping
the time of disclosure. probability among all instances of a class, while the latter
Expected Exploitability. The problems mentioned above assumes that noise probability also depends on individual
suggest that the evolution of exploitability over time can be features of instances. If Etrain is affected by label noise, the
described by a stochastic process. At a given point in time, test time performance of the classifier could suffer.
exploitability is a random variable E encoding the probability By viewing exploitability as time-varying, it becomes im-
of observing an exploit. E assigns a probability 0.0 to the mediately clear that exploit evidence datasets are prone to
subset of vulnerabilities that are provably unexploitable, and class-dependent noise. This is because exploits might not
1.0 to vulnerabilities with known exploits. Nevertheless, the yet be developed or be kept secret. Therefore, a subset of
true distribution E generating E is not available at scale, and vulnerabilities believed not to be exploited are in fact wrongly
instead we need to rely on a noisy version Etrain , as we will labeled at any given point in time.
discuss in Section 3. This implies that in practice E has to In addition, prior work noticed that individual vendors pro-
be approximated from the available data, by computing the viding exploit evidence have uneven coverage of the vulnera-
likelihood of exploits, which estimates the expected value of bility space (e.g. an exploit dataset from Symantec would not
exploitability. We call this measure Expected Exploitability contain Linux exploits because the platform is not covered
(EE). EE can be learned from historical data using supervised by the vendor) [55], suggesting that noise probability might
machine learning and can be used to assess the likelihood of be dependent on certain features. The problem of feature-
exploits for new vulnerabilities before functional exploits are dependent noise is much less studied [45], and discovering
developed or discovered. the characteristics of such noise on real-world applications is
considered an open problem in machine learning [23].
1 Interestingly,
some social media posts appear before disclosure. These Exploit prediction therefore requires an empirical under-
"leaks from coordinated disclosure" were investigated by prior work [55]. standing of both the type and effects of label noise, as well as

USENIX Association 31st USENIX Security Symposium 379

the design of learning techniques to address it. information about the vulnerabilities targeted by the PoCs
Evaluating the impact of time-varying exploitability. from the National Vulnerability Database (NVD) [43]. NVD
While some post-disclosure artifacts are likely to improve adds vulnerability information gathered by analysts, including
classification, publication delay might affect their utility as textual descriptions of the issue, product and vulnerability
timely predictions. Our EE evaluation therefore needs to use type information, as well as the CVSS score. Nevertheless,
metrics which highlight potential trade-offs between timeli- NVD only contains high-level descriptions. In order to build a
ness and performance. Moreover, the evaluation needs to test more complete coverage of the technical information available
whether our classifier can capitalize on artifacts with high for each vulnerability, we search for external references in
predictive power available before functional exploits are dis- several public sources. We use the Bugtraq [8] and IBM X-
covered, and whether EE can capture the imminence of certain Force Exchange [28], vulnerability databases which provide
exploits. Finally, we need to demonstrate the practical util- additional textual descriptions for the vulnerabilities. We also
ity of EE over existing static metrics, in real-world scenarios use Vulners [70], a database that collects in real time textual
involving vulnerability prioritization. information from vendor advisories, security bulletins, third-
Goals and Non-Goals. Our goal is to estimate EE for a broad party bug trackers and security databases. We filter out the
range of vulnerabilities, by addressing the challenges listed reports that mention more than one CVEID, as it would be
above. Moreover, we aim to provide estimates that are both challenging to determine which particular one is discussed.
accurate and robust: they should predict the development of In total, our collection contains 278,297 documents from
functional exploits better than the existing scoring systems 76 sources, referencing 102,936 vulnerabilities. We refer
and despite inaccuracies in the ground truth. The closest to these documents as write-ups, which, together with the
work to our goal is DarkEmbed [64], which uses natural NVD textual information and vulnerability details, provide a
language models trained on underground forum discussions broader picture of the technical information publicly available
to predict the availability of exploits. In contrast, we aim to for vulnerabilities.
predict functional exploits from public information, a more Proof of Concepts (PoCs). We collect a dataset of public
difficult task as we lack direct evidence of black-hat exploit PoCs by scraping ExploitDB [20], Bugtraq [8] and Vul-
development. ners [70], three popular vulnerability databases that contain
We do not aim to generate functional exploits automati- exploits aggregated from multiple sources. Because there is
cally. We also do not aim to analyze the code of existing substantial overlap across these sources, but the formatting of
exploits in order to assess how close they are to becoming the PoCs might differ slightly, we remove duplicates using a
functional. Instead, we aim to quantify the exploitability of content hash that is invariant to such minor whitespace differ-
known vulnerabilities objectively, by predicting whether func- ences. We preserve only the 48,709 PoCs that are linked to
tional exploits will be developed for [Link] we aim to CVEIDs, which correspond to 21,849 distinct vulnerabilities.
make our exploit predictor robust to systematic label noise, Social Media Discussions. We also collect social media dis-
we do not attempt to improve the model’s robustness to ad- cussions about vulnerabilities from Twitter, by gathering
versarial examples [55], as techniques for achieving this have tweets mentioning CVE-IDs between January 2014 and De-
been widely studied elsewhere [10]. Finally, we do not aim cember [Link] collected 1.4 million tweets for 52,551 vul-
to predict which vulnerabilities are likely to be exploited in nerabilities by continuously monitored the Twitter Filtered
the wild, in real-world attacks, because this likelihood is in- Stream API [68], using the approach from our prior work [55].
fluenced by additional factors (e.g. attacker choices, patching While the Twitter API does not sample returned tweets, short
delays) that are out of scope for this paper. offline periods for our platform caused some posts to be lost.
By a conservative estimate using the lost tweets which were
later retweeted, our collection contains over 98% of all public
4 Data Collection tweets about these vulnerabilities.
Exploitation Evidence Ground Truth. Because we are not
In this section we describe the methods used to collect the data
aware of any comprehensive dataset of evidence about devel-
used in our paper, as well as the techniques for discovering
oped exploits, we aggregate evidence from multiple public
various timestamps in the lifecycle of vulnerabilities.
sources.
We begin from the Temporal CVSS score, which tracks
4.1 Gathering Technical Information the status of exploits and the confidence in these reports. The
Exploit Code Maturity component has four possible values:
We use the CVEIDs to identify vulnerabilities, because it is "Unproven", "Proof-of-Concept", "Functional" and "High".
one of the most prevalent and cross-referenced public vulner- The first two values indicate that the exploit is not practi-
ability identification systems. Our collection contains vulner- cal or not functional, while the last two values indicate the
abilities published between January 1999 and March 2020. existence of autonomous or functional exploits that work in
Public Vulnerability Information. We begin by collecting

380 31st USENIX Security Symposium USENIX Association

most situations. Because the Temporal score is not updated minimum date among all sources that shared them, and we
in NVD, we collect it from two reputable sources: IBM X- confirm the accuracy of these dates dates by verifying the
Force Exchange [28] threat sharing platform and the Tenable commit history in exploit databases that use version control.
Nessus [66] vulnerability scanner. Both scores are used as To assess whether EE can provide timely warnings, we
inputs to proprietary severity assessment solutions: the former need the dates for the emergence of functional exploits and
is used by IBM in one of their cloud offerings [71], while attacks in the wild. Because the sources of exploit evidence
the latter is used by Tenable as input to commercial vulnera- do not share the dates when exploits were developed, we
bility prioritization solutions [67]. We use the labels "Func- estimate these dates from ancillary data. For the exploit toolk-
tional" and "High" as evidence of exploitation, as defined its, we collect the earliest date when exploits are reported
by the official CVSS Specification [42], obtaining 28,009 in the Metasploit and Canvas platforms. For exploits in the
exploited vulnerabilities. We further collect evidence of 2,547 wild, we use the dates of first recorded attacks, from prior
exploited vulnerabilities available in three commercial ex- work [55]. Across all exploited vulnerabilities, we also crawl
ploitation tools: Metasploit [50], Canvas [29] and D2 [14]. VirusTotal [69], a popular threat sharing platform, for the
We also scrape the Bugtraq [8] exploit pages, and create NLP timestamps when exploit files were first submitted. Finally,
rules to extract evidence for 1,569 functional exploits. Ex- we estimate exploit availability as the earliest date among
amples of indicative phrases are: "A commercial exploit is the different sources, excluding vulnerabilities with zero-day
available.", "A functional exploit was demonstrated by re- exploits. Overall, we discovered this date for 10% (3,119)
searchers.". of the [Link] estimates could result in label noise,
We also collect exploitation evidence that results from ex- because exploits might sometimes be available earlier, e.g.
ploitation in the wild. Starting from a dataset collected from PoCs that are easy to weaponize. In Section 7.3 we measure
Symantec in our prior work [55], we update it by scraping the impact of such label noise on the EE performance.
Symantec’s Attack Signatures [4] and Threat Explorer [61].
We then aggregate labels extracted using NLP rules (matching 4.3 Datasets
e.g. "... was seen in the wild.") from scrapes of Bugtraq [8],
Tenable [65], Skybox [58] and AlienVault OTX [44]. In We create three datasets that we use throughout the paper
addition, we use the Contagio dump [39] which contains a to evaluate EE. DS1 contains all 103,137 vulnerabilities in
curated list of exploits used by exploit kits. These sources our collection that have at least one artifact published within
were reported by prior work as reliable for information about one year after disclosure. We use this to evaluate the time-
exploits in the wild [30, 41, 55]. Overall, 4,084 vulnerabilities liness of various artifacts, compare the performance of EE
are marked as exploited in the wild. with existing baselines, and measure the predictive power
While exact development time for most exploits is not avail- of different categories of features. The second dataset, DS2,
able, we drop evidence if we cannot confirm they were pub- contains 21,849 vulnerabilities that have artifacts across all
lished within one year after vulnerability disclosure, simulat- different categories within one year. This is used to compare
ing a historical setting. Our ground truth, consisting of 32,093 the predictive power of various feature categories, observe
vulnerabilities known to have functional exploits, therefore their improved utility over time, and to test their robustness to
reflects a lower bound for the number of exploits available, label noise. The third one, DS3 contains 924 out of the 3,119
which translates to class-dependent label noise in classifica- vulnerabilities for which we estimated the exploit emergence
tion, issue that we evaluate in Section 7. date, and which are disclosed during our classifier deployment
period described in Section 6.3. These are used to evaluate
the ability of EE to distinguish imminent exploit.
4.2 Estimating Lifecycle Timestamps
Vulnerabilities are often published in NVD at a later date than 5 Empirical Observations
their public disclosure [5, 33]. We estimate the public disclo-
sure dates for the vulnerabilities in our dataset by selecting the We start our analysis with three empirical observations on
minimum date among all write-ups in our collection and the DS1, which guide the design of our system for computing EE.
publication date in NVD, in line with prior research [33, 57]. Existing scores are poor predictors. First, we investigate
This represents the earliest date when expected exploitability the effectiveness of three vulnerability scoring systems, de-
can be evaluated. We validate our estimates for the disclo- scribed in Section 2, for predicting exploitability. Because
sure dates by comparing them to two independent prior esti- these scores are widely used, we will utilize them as base-
mates [33, 57], on the 67% of vulnerabilities which are also lines for our prediction performance; our goal for EE is to
found in the other datasets. We find that the median date dif- improve this performance substantially. As the three scores
ference between the two estimates is 0 days, and our estimates do not change over time, we utilize a threshold-based deci-
are an average of 8.5 days earlier than prior assessments. Sim- sion rule, which predicts that all vulnerabilities with scores
ilarly, we estimate the time when PoCs are published as the greater or equal than the threshold are exploitable. By varying

USENIX Association 31st USENIX Security Symposium 381

 1 1 1
1 1
P P P
R R R
0.8 0.8 0.8
0.8 0.8
Metric Value

0.6 0.6 0.6

0.6 0.6

CDF

CDF
0.4 0.4 0.4
0.4 0.4

0.2 0.2 0.2

NVD Publication NVD Publication
CVSS Publication CVSS Publication
0.2 0.2
Write-Ups Write-Ups
0 0 0 Tweets Tweets
0 1 2 3 4 3 2 1 3 2 1 0 PoC Info PoC Info
MS Exploitability Index RedHat Severity 0 PoC Code 0 PoC Code
CVSSv3 Exploitability
0 0.5 1 1.5 2 2.5 -4 -3 -2 -1 0 1 2
Date First Artifact - Date Disclosure [log10(# days)] Date First Artifact - Date Exploit Available [log10(# days)]
Figure 2: Performance of existing severity scores at capturing
(a) (b)
exploitability. We report both precision (P) and recall (R).
The numerical score values are ordered by increasing severity. Figure 3: (a) Number of days after disclosure when vulnera-
bility artifacts are first published. (b) Difference between the
availability of exploits and availability of other artifacts. The
the threshold across the entire score range, and using all the day differences are in logarithmic scale.
vulnerabilities in our dataset, we evaluate their precision (P):
the fraction of predicted vulnerabilities that have functional
exploits within one year from disclosure, and recall (R): the published, they are added to NVD, their CVSS and technical
fraction of exploited vulnerabilities that are identified within analysis are published in NVD, their first PoCs are released
one year. and when they are first mentioned on Twitter. The publica-
Figure 2 reports these performance metrics. It is possible tion delay distribution for all collected artifacts is available in
to obtain R = 1 by marking all vulnerabilities as exploitable, Figure 1.
but this affects P because many predictions would be false Write-ups are the most widely available ones at the time
positives. For this reason, for all the scores, R decreases as we of disclosure, suggesting that vendors prefer to disclose vul-
raise the severity threshold for prediction. However, obtaining nerabilities through either advisories or third party databases.
a high P is more difficult. For CVSSv3 Exploitability, P does However, many PoCs are also published early: 71% of vul-
not exceed 0.19, regardless of the detection threshold, and nerabilities have a PoC on the day of disclosure. In contrast,
some vulnerabilities do not have scores assigned to them. In only 26% of vulnerabilities in our dataset are added to NVD
the technical report [60] we evaluate CVSSv2, which yields on the day of disclosure, and surprisingly, only 9% of the
similar results. CVSS scores are published at disclosure. This result suggests
When evaluating the Microsoft Exploitability Index on the that timely exploitability assessments require looking beyond
1,100 vulnerabilities for Microsoft products in our dataset NVD, using additional sources of technical vulnerability in-
disclosed since the score inception in 2008, we observe that formation, such as the write-ups and PoCs. This observation
the maximum precision achievable is 0.45. The recall is drives our feature engineering from Section 6.1.
also lower because the score is only computed on a subset of Figure 3(b) highlights the day difference between the dates
vulnerabilities [18]. when the exploits become available and the availability of
On the 3,030 vulnerabilities affecting RedHat products, we the artifacts from public vulnerability disclosure. For more
observe a similar trend for the proprietary severity metric, than 92% of vulnerabilities, write-ups are available before
where precision does not exceed 0.45. the exploits become available. We also find that the 62%
These results suggest that the three existing scores predict of PoCs are available before this date, while 64% of CVSS
exploitability with > 50% false positives. This is compounded assessments are added to NVD before. Overall, the avail-
by the facts that (1) some scores are not computed for all vul- ability of exploits is highly correlated with the emergence
nerabilities, owing to the manual effort required, which intro- of other artifacts, indicating an opportunity to infer the exis-
duces false negative predictions; (2) the scores do not change, tence of functional exploits as soon as, or before, they become
even if new information becomes available; and (3) not all the available.
scores are available at the time of disclosure, meaning that Exploit prediction is subject to feature-dependent label
the recall observed operationally soon after disclosure will be noise. Good predictions also require a judicious solution to
lower, as highlighted in the next section. the label noise challenge discussed in Section 3. The time-
Artifacts provide early prediction opportunities. To assess varying view of exploitability revealed that our problem is
the opportunities for early prediction, we look at the publica- subject to class-dependent noise. However, because we ag-
tion timing for certain artifacts from the vulnerability lifecycle. gregate evidence about exploits from multiple sources, their
In Figure 3(a), we plot, across all vulnerabilities, the earli- individual biases could also affect our ground truth. To test
est point in time after disclosure when the first write-ups are for such individual biases, we investigate the dependence

382 31st USENIX Security Symposium USENIX Association

 Functional Exploits Exploits in the Wild
Tenable X-Force Metasploit Canvas Bugtraq D2 Symantec Contagio Alienvault Bugtraq Skybox Tenable
CWE-79 X X X X X X X X X X X X(0.006)

CWE-94 X X X X X X X X X X X X (1.000)

CWE-89 X X X X X X (1.000) X X X X X X (0.284)

CWE-119 X X X X X X X X X X X X(0.001)

CWE-20 X X X X X X X X (1.000) X X(0.002) X X (1.000)

CWE-22 X X (0.211) X X (1.000) X (1.000) X X (1.000) X (1.000) X (0.852) X (1.000) X (1.000) X (1.000)

Windows X X X X X X (0.012) X X X X X X

Linux X X X X X X (1.000) X X X X X X

Table 1: Evidence of feature-dependent label noise. A Xindicates that we can reject the null hypothesis H0 that evidence of
exploits within a source is independent of the feature. Cells with no p-value are < 0.001.

between all sources of exploit evidence and various vulner- which encodes structural characteristics of the code. From
ability characteristics. For each source and feature pair, we the ASTs we extract complexity features such as statistics
perform a Chi-squared test for independence, aiming to ob- of the node types, structural features of the tree, as well as
serve whether the we can reject the null hypothesis H0 that statistics of control statements within the program and the
the presence of an exploit within the source is independent relationship between them. Additionally, we extract features
of the presence of the feature for the vulnerabilities. Table 1 for the function calls within the PoCs towards external library
lists the results for all 12 sources of ground truth, across the functions, which in some cases may be the means through
most prevalent vulnerability types and affected products in which the exploit interacts with the vulnerability and thereby
our dataset. We utilize the Bonferroni correction for multiple reflect the relationship between the PoC and its vulnerability.
tests [17] and a 0.01 significance level. We observe that the Therefore, the library functions themselves, as well as the
null hypothesis can be rejected for at least 4 features for each patterns in calls to these functions, can reveal information
source. We observe that the null hypothesis can be rejected about the complexity of the vulnerability, which might in turn
for at least 4 features for each source, indicating that all the express the difficulty of creating a functional exploit. We
sources for ground truth include biases caused by individual also extract the cyclomatic complexity from the AST [32],
vulnerability features. These biases could be reflected in the a software engineering metric which encodes the number of
aggregate ground truth, suggesting that exploit prediction is independent code paths in the program. Finally, we encode
subject to class- and feature-dependent label noise. features of the PoC programming language, in the form of
statistics over the file size and the distribution of language
reserved keywords.
6 Computing Expected Exploitability We also observe that the lexical characteristics of the PoC
code provide insights into the complexity of the PoC. For
In this section we describe the system for computing EE,
example, a variable named shellcode in a PoC might sug-
starting from the design and implementation of our feature
gest that the exploit is in an advanced stage of development.
extractor, and presenting the classifier choice.
In order to capture such characteristics, we extract the code
tokens from the entire program, capturing literals, identifiers
6.1 Feature Engineering and reserved keywords, in a set of binary unigram features.
Such specific information allows us to capture the stylistic
EE uses features extracted from all vulnerability and PoC
characteristics of the exploit, the names of the library calls
artifacts in our datasets, which are summarized in Table 2.
used, as well as more latent indicators, such as artifacts in-
Novel: PoC Code. Intuitively, one of the leading indicators dicating exploit authorship [9], which might provide utility
for the complexity of functional exploits is the complexity of towards predicting exploitability. Before training the classi-
PoCs. This is because if triggering the vulnerability requires fier, we filter out lexicon features that appear in less than 10
a complex PoC, an exploit would also have to be complex. training-time PoCs, which helps prevent overfitting.
Conversely, complex PoCs could already implement func-
Novel: PoC Info. Because a large fraction of PoCs contain
tionality beneficial towards the development of functional
only textual descriptors for triggering the vulnerabilities with-
exploits. We use this intuition to extract features that reflect
out actual code, we also extract features that aim to encode the
the complexity of PoC code, by means of intermediate rep-
technical information conveyed by the authors in the non-code
resentations that can capture it. We transform the code into
PoCs, as well as comments in code PoCs. We encode these
Abstract Syntax Trees (ASTs), a low-overhead representation

USENIX Association 31st USENIX Security Symposium 383

 Type Description #
PoCs Write-ups NVD
PoC Code (Novel)
Length # characters, loc, sloc 33
Language Programming language label 11 Code / Text
Program NLP
Keywords count Count for reserved keywords 820 Analysis Separation
Tokens Unigrams from code 92,485
#_nodes # nodes in the AST tree 4
PoC Code PoC Info Write-Up NVD Info
#_internal_nodes # of internal AST tree nodes 4 Features Features Features Features
#_leaf_nodes # of leaves of AST tree 4
#_identifiers # of distinct identifiers 4
#_ext_fun # of external functions called 4 Figure 4: Diagram of the EE feature extraction system.
#_ext_fun_calls # of calls to external functions 4
#_udf # user-defined functions 4
#_udf_calls # calls to user-defined functions 4 tiveness of various feature sets, we also extract 2 categories
#_operators # operators used 4 proposed in prior predictors of exploitation in the wild. The
cyclomatic compl cyclomatic complexity 4 Exploit Prediction Scoring System (EPSS) [31] proposes 53
nodes_count_* # of AST nodes for each node type 316 features manually selected by experts as good indicators for
ctrl_nodes_count_* # of AST nodes for each control statement type 29 exploitation in the wild. This set of handcrafted features con-
literal_types_count_* # of AST nodes for each literal type 6
tains tags reflecting vulnerability types, products and vendors,
nodes_depth_* Stats depth in tree for each AST node type 916
branching_factor Stats # of children across AST 12
as well as binary indicators of whether PoC or weaponized
branching_factor_ctrl Stats # of children within the Control AST 12 exploit code has been published for a vulnerability. Second,
nodes_depth_ctrl_* Stats depth in tree for each Control AST node type 116 from our collection of tweets, we extract social media features
operator_count_* Usage count for each operator 135 introduced in prior work [55], which reflect the textual de-
#_params_udf Stats # of parameters for user-defined functions 12 scription of the discourse on Twitter, as well as characteristics
PoC Info (Novel) of the user base and tweeting volume for each vulnerability.
PoC unigrams PoCs text and comments 289,755
Unlike the the original work, we do not perform feature se-
Write-ups (Prior Work)
Write-up unigrams Write-ups text 488,490
lection on the unigram features from tweets because we want
Vulnerability Info (Prior Work) to directly compare the utility of Twitter discussions to these
NVD unigrams NVD descriptions 103,793 from other artifacts. None of the two categories will be used
CVSS CVSSv2 & CVSSv3 components 40 in the final EE model because of their limited predictive utility.
CWE Weakness type 154
CPE Name of affected product 10
In-the-Wild Predictors (Prior Work) 6.2 Feature Extraction System
EPSS Handcrafted 53
Social Media Twitter content and statistics 898,795 Below we describe the components our our feature extraction
system, illustrated in Figure 4, and discuss how we address
the challenges identified in Section 3.
Table 2: Description of features used. Unigram features are Code/Text Separation. Only 64% of the PoCs in our dataset
counted before frequency-based pruning. contain any file extension that would allow us to identify the
programming language. Moreover, 5% of them have conflict-
ing information from different sources, and we observe that
features as binary unigrams. Unigrams provide a clear base- many PoCs are first posted online as freeform text without
line for the performance achievable using NLP. Nevertheless, explicit language information. Therefore, a central challenge
in our technical report [60] we investigate the performance is to accurately identify their programming languages and
of EE with embeddings, showing that there are additional whether they contain any code. We use GitHub Linguist [24],
challenges in designing semantic NLP features for exploit to extract the most likely programming languages used in
prediction, which we leave for future work. each PoC. Linguist combines heuristics with a Bayesian clas-
Prior Work: Vulnerability Info and Write-ups. To capture sifier to identify the most prevalent language within a file.
the technical information shared through natural language Nevertheless, the model obtains an accuracy of 0.2 on classi-
in artifacts, we extract unigram features from all the write- fying the PoCs, due to the prevalence of natural language text
ups discussing each vulnerability and the NVD descriptions in PoCs. After modifying the heuristics and retraining the
of the vulnerability. Finally, we extract the structured data classifier on 42,195 PoCs from ExploitDB that contain file
within NVD that encodes vulnerability characteristics: the extensions, we boost the accuracy to 0.95. The main cause of
most prevalent list of products affected by the vulnerability, errors is text files with code file extensions, yet these errors
the vulnerability types (CWEID [12]), and all the CVSS Base have limited impact because of the NLP features extracted
Score sub-components, using one-hot encoding. from files.
Prior Work: In-the-Wild Predictors. To compare the effec- Table 3 lists the number of PoCs in our dataset for each

384 31st USENIX Security Symposium USENIX Association

 Language # PoCs # CVEs (% exploited) On Perl, by modifying the existing Compiler::Parser [38]
Text 27743 14325 (47%) tool to improve its robustness, and employing heuristics to
correct malformed PoC files, we improve the parsing success
Ruby 4848 1988 (92%) rate from 37% to 83%.
C 4512 2034 (30%) For Python, we implement a feature extractor based on
Perl 3110 1827 (54%) the ast parsing library [49], achieving a success rate of 67%.
We observe that this lower parsing success rate is due to
Python 2590 1476 (49%)
the reliance of the language on strict indentation, which is
JavaScript 1806 1056 (59%) often distorted or completely lost when code gets distributed
PHP 1040 708 (55%) through Webpages.
HTML 1031 686 (56%) Ruby provides an interesting case study because, despite
being the most prevalent language among PoCs, it is also
Shell 619 304 (29%) the most indicative of exploitation. We observe that this
VisualBasic 397 215 (41%) is because our dataset contains functional exploits from the
None 367 325 (43%) Metasploit framework, which are written in Ruby. We extract
AST features for the language using the Ripper library [54].
C++ 314 196 (34%)
Our implementation is able to successfully parse 96% of the
Java 119 59 (32%) files.
Overall, we successfully parse 13,704 PoCs associated with
Table 3: Breakdown of the PoCs in our dataset according to 78% of the CVEs that have PoCs with code. Each vulnerabil-
programming language. ity aggregates only the code complexity features of the most
complex PoC (in source lines of code) across each of the four
identified language label (the None label represents the cases languages, while the remaining code features are collected
which our classifier could not identify any language, including from all PoCs available.
less prevalent programming languages not in our label set). Unigram Features. We extract the textual features using
We observe that 58% of PoCs are identified as text, while a standard NLP pipeline which involves tokenizing the
the remaining ones are written in a variety of programming text from the PoCs or vulnerability reports, removing non-
languages. Based on this separation, we develop regular alphanumeric characters, filtering out English stopwords and
expressions to extract the comments from all code files. After representing them as unigrams. For each vulnerability, the
separating the comments, we process them along with the text PoC unigrams are aggregated across all PoCs, and separately
files using NLP, to obtain PoC Info features, while the PoC across all write-ups collected within the observation period.
Code features are obtained using NLP and program analysis. When training a classifier, we discard unigrams which occur
Code Features. Performing program analysis on the PoCs less than 100 times across the training set, because they are
poses a challenge because many of them do not have a valid unlikely to generalize over time and we did not observe any
syntax or have missing dependencies that hinders compilation noticeable performance boost when including them.
or interpretation [2, 40]. We are not aware of any unified
and robust solution to simultaneously obtain ASTs from code 6.3 Exploit Predictor Design
written in different languages. We address this challenge by
employing heuristics to correct malformed PoCs and parsing The predictor concatenates all the extracted features, and uses
them into intermediate representations using techniques that the ground truth about exploit evidence, to train a classifier
provide robustness to errors. which outputs the EE score. The classifier uses a feedforward
Based on Table 3, we observe that some languages are neural network having 2 hidden layers of size 500 and 100
likely to have a more significant impact on the prediction per- respectively, with ReLU activation functions. This choice was
formance, based on prevalence and frequency of functional dictated by two main characteristics of our domain: feature
exploits among the targeted vulnerabilities. Given this obser- dimmensionality and concept drift. First, as we have many
vation, we focus our implementation on Ruby, C/C++, Perl potentially useful features, but with limited coverage, linear
and Python. Note that this choice does not impact the ex- models, such as SVM, which tend to emphasize few important
traction of lexical features from code PoCs written in other features [36], would perform worse. Second, deep learning
languages. models are believed to be more robust to concept drift and the
For C/C++ we use the Joern fuzzy parser for program anal- shifting utility of features [46], which is a prevalent issue in
ysis, previously proposed for bug discovery [75]. The tool the exploit prediction task [55]. The architecture was chosen
provides robustness to parsing errors through the use of island empirically by measuring performance for various settings.
grammars and allows us to successfully parse 98% of the Classifier training. To address the second challenge iden-
files. tified in Section 3, we incorporate noise robustness in our

USENIX Association 31st USENIX Security Symposium 385

system by exploring several loss functions for the classifier. higher score for predictions which might have noisy labels.
Our design choices are driven by two main requirements: (i) This prevents the classifier from fitting of instances with po-
providing robustness to both class- and feature- dependent tentially noisy labels. We analyze the loss in more detail in
noise, and (ii) providing minimal performance degradation the technical report [60]. FC partially addresses requirement
when noise specification is not available. (i), being explicitly designed only for class-dependent noise.
BCE: The binary cross-entropy is the standard, noise- However, unlike LR, it naturally addresses (ii) because it is
agnostic loss for training binary classifiers. For a set of N equivalent to BCE if pe = 0.
examples xi with labels yi ∈ {0, 1}, the loss is computed as: FFC: To fully address (i), we modify FC to account for
1 N feature-dependent noise, a loss function we denote Feature
LBCE = − ∑ [yi log(pθ (xi )) + (1 − yi )log(1 − pθ (xi )] Forward Correction (FFC). We observe that for exploit pre-
N i=1
diction, feature-dependent noise occurs within the same label
where pθ (xi ) corresponds to the output probability predicted flipping template as class-dependent noise. We use this ob-
by our classifier. BCE does not explicitly address requirement servation to expand the noise transition matrix with instance
(i), but can be used to benchmark noise-aware losses that aim specific priors: Ti j (x) = p(e y j |x, yi ). In this case the transition
to address (ii). matrix becomes:
LR: The Label Regularization, initially proposed as a semi- !
1 0
supervised loss to learn from unlabeled data [35], has been T (x) =
shown to address class-dependent label noise in malware pe(x) 1 − pe(x)
classification [15] using a logistic regression classifier. Assuming that we only possess priors for instances that have
1 N certain features f , the instance prior can be encoded as a
LLR = − ∑ [yi log(pθ (xi ))] − λKL( pe|| p̂θ ) lookup-table:
N i=1 (
The loss function complements the log-likelihood loss over pef if y = 0 and x has f
pe(x, y) =
the positive examples with a label regularizer, which is the 0 otherwise
KL divergence between a noise prior pe and the classifier’s While feature-dependent noise might cause the classifier to
output distribution over the negative examples p̂θ : learn a spurious correlation between certain features and the
1 N wrong negative label, this formulation mitigates the issue
p̂θ = ∑ [(1 − yi )log(1 − pθ (xi ))]
N i=1 by reducing the loss only on the instances that posses these
Intuitively, the label regularizer aims to push the classifier features. In Section 7 we show that feature-specific prior
predictions on the noisy class towards the the expected noise estimates are achievable from a small set of instances, and use
prior pe, while the λ hyper-parameter controls the regulariza- this observation to compare the utility of class- and feature-
tion strength. We use this loss to observe the extent to which specific noise priors in addressing label noise. When training
existing noise correction approaches for related security tasks the classifier, we discovered optimal performance when using
apply to our problem. However, this function was not de- an ADAM optimizer for 20 epochs and a batch size of 128,
signed to address (ii) and, as our results will reveal, yields using a learning rate of 5e-6.
poor performance in our problem. Classifier deployment. We evaluate the historic performance
FC: The Forward Correction loss has been shown to sig- of our classifier by partitioning the dataset into temporal splits,
nificantly improve robustness to class-dependent label noise assuming that the classifier is re-trained periodically, on all
in various computer vision tasks [45]. The loss requires a the historical data available at that time. At the time the
pre-defined noise transition matrix T ∈ [0, 1]2x2 , where each classifier is trained, we do not include the vulnerabilities dis-
element represents the probability of observing a noisy label closed within the last year because the positive labels from
y j |yi ). For an instance xi , the log-
yej for a ture label yi : Ti j = p(e exploitation evidence might not be available until later on. We
likelihood is then defined as lc (xi ) = −log(T0c (1 − pθ (xi )) + discovered that the classifier needs to be retrained every six
T1c pθ (xi )) for each class c ∈ {0, 1}. In our case, because we months, as less frequent retraining would affect performance
assume that the probability of falsely labeling non-exploited due to a larger time delay between the disclosure of training
vulnerabilities as exploited is negligible, the noise matrix can and testing instances. During testing, the system operates in
!
1 0 a streaming environment in which it continuously collects
be defined as: T = , and the loss reduces to: the data published about vulnerabilities then recomputes their
pe 1 − pe
feature vectors over time and predicts their updated EE score.
1 N The prediction for each test-time instance is performed with
LFC = − ∑ [yi log((1 − pe)pθ (xi ))+
N i=1 the most recently trained classifier. To observe how our clas-
+(1 − yi )log(1 − (1 − pe)pθ (xi ))] sifier performs over time, we train the classifier using the
various loss functions and test its performance on all vulner-
On the negative class, the loss reduces the penalty for con- abilities disclosed between January 2010, when 65% of our
fident positive predictions, allowing the classifier to output a

386 31st USENIX Security Symposium USENIX Association

 Feature % Noise Actual Prior Est. Prior # Inst to Est. of the Vulnerability Info category, that we analyzed in Sec-
CWE-79 14% 0.93 0.90 29 tion 5: the six most prevalent vulnerability types, reflected
CWE-94 7% 0.36 0.20 5
through the CWE-IDs, as well as the two most popular prod-
CWE-89 20% 0.95 0.95 22
ucts: linux and windows. Mislabeling instances with these
CWE-119 14% 0.44 0.57 51
CWE-20 6% 0.39 0.58 26
features results in a wide range of noise: between 5-20% of
CWE-22 8% 0.39 0.80 15
negative labels become noisy during training.
Windows 8% 0.35 0.87 15 All techniques require priors about the probability of noise.
Linux 5% 0.32 0.50 4 The LR and FC approaches require a prior pe over the entire
negative class. To evaluate an upper bound of their capabil-
Table 4: Noise simulation setup. We report the % of negative ities, we assume perfect prior and set pe to match the frac-
instances that are noisy, the actual and estimated noise prior, tion of training-time instances that are mislabeled. The FFC
and the # of instances used to estimate the prior. approach assumes knowledge of the noisy feature f . This
assumption is realistic, as it is often possible to enumerate the
BCE LR FC FFC
features that are most likely noisy (e.g. prior work identified
Feature P AUC P AUC P AUC P AUC linux as a noise-inducing feature due to the fact that the
CWE-79 0.58 0.80 0.67 0.79 0.58 0.81 0.75 0.87 vendor collecting exploit evidence does not have a product
CWE-94 0.81 0.89 0.71 0.81 0.81 0.89 0.82 0.89 for the platform [55]). Besides, FFC requires estimates of the
CWE-89 0.61 0.82 0.57 0.74 0.61 0.82 0.81 0.89 feature-specific priors pef . We assume an operational scenario
CWE-119 0.78 0.88 0.75 0.83 0.78 0.87 0.81 0.89 were pef is estimated once, by manually labeling a subset of
instances collected after training. We use the vulnerabilities
CWE-20 0.81 0.89 0.72 0.82 0.80 0.88 0.82 0.90
disclosed in the first 6 months after training for estimating
CWE-22 0.81 0.89 0.69 0.80 0.81 0.89 0.83 0.90
pef and require that these vulnerabilities are correctly labeled.
Windows 0.80 0.88 0.71 0.81 0.80 0.88 0.83 0.90 Table 4 shows the actual and the estimated priors pef , as well
Linux 0.81 0.89 0.71 0.81 0.81 0.89 0.82 0.90 as the number of instances used for the estimation. We ob-
serve that the number of instances required for estimation
Table 5: Noise simulation results. We report the precision at is small, ranging from 5 to 51 across all features f , which
a 0.8 recall (P) and the precision-recall AUC. The pristine demonstrates that setting feature-based priors is feasible in
BCE classifier performance is 0.83 and 0.90 respectively. practice. Nevertheless, we observe that the estimated priors
are not always accurate approximations of the actual ones,
dataset was available for training, and March 2020. which might negatively impact FFC’s ability to address the
effect of noise.
In Table 5 we list the results of our experiment. For each
7 Evaluation classifier, we report the the precision achievable at a recall of
0.8, as well as the precision-recall AUC. Our first observation
We evaluate our approach of predicting expected exploitabil-
is that the performance of the vanilla BCE classifier is not
ity by testing EE on real-world vulnerabilities and answering
equally affected by noise across different features. Interest-
the following questions, which are designed to address the
ingly, we observe that the performance drop does not appear
third challenge identified in Section 3: How effective is EE at
to be linearly dependent on the amount of noise: both CWE-79
addressing label noise? How well does EE perform compared
and CWE-119 result in 14% of the instances being poisoned,
to baselines? How well do various artifacts predict exploitabil-
yet only the former inflicts a substantial performance drop
ity? How does EE performance evolve over time? Can EE
on the classifier. Overall, we observe that the majority of
anticipate imminent exploits? Does EE have practicality for
the features do not result in significant performance drops,
vulnerability prioritization?
suggesting that BCE offers a certain amount of built-in robust-
ness to feature-dependent noise, possibly due to redundancies
7.1 Feature-dependent Noise Remediation in the feature space which cancel out the effect of the noise.
To observe the potential effect of feature-dependent label For LR, after performing a grid search for the optimal
noise on our classifier, we simulate a worst-case scenario in λ parameter which we set to 1, we were unable to match
which our training-time ground truth is missing all the ex- the BCE performance on the pristine classifier. Indeed, we
ploits for certain features. The simulation involves training observe that the loss is unable to correct the effect of noise
the classifier on dataset DS2, on a ground truth where all the on any of the features, suggesting that it is not a suitable
vulnerabilities with a specific feature f are considered not choice for our classifier as it does not address any of the two
exploited. At testing time, we evaluate the classifier on the requirements of our classifier.
original ground truth labels. Table 4 describes the setup for On features where BCE is not substantially affected by
our experiments. We investigate 8 vulnerability features, part noise, we observe that FC performs similarly [Link],

USENIX Association 31st USENIX Security Symposium 387

 1 1
disclosure of test-time instances. We observe that none of
0.8 0.8
the static exploitability metrics exceed 0.5 precision, while
EE significantly outperforms all the baselines. The perfor-
0.6 0.6
mance gap is especially apparent for the 60% of exploited
Precision

Precision
vulnerabilities, where EE achieves 86% precision, whereas
0.4 0.4 the SMC, the second-best performing classifier, obtains only
EE (AUC = 0.73)
Baseline: EPSS (AUC = 0.45)
NVD Info (AUC = 0.62)
Handcrafted (EPSS) (AUC = 0.47)
49%. We also observe that for around 10% of vulnerabilities,
0.2 Baseline: Social Media Clf (AUC = 0.59)
Baseline: MS Exploitability
0.2 Social Media (AUC = 0.56)
Write-Ups (AUC = 0.72)
the artifacts available within 30 days have limited predictive
Baseline: RedHat Severity
Baseline: CVSSv3 Exploitability
PoC Info (AUC = 0.87)
PoC Code (AUC = 0.86)
utility, which affects the performance of these classifiers.
0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 EE uses the most informative features. To understand why
Recall Recall
EE is able to outperform these baselines, in Figure 5b we plot
(a) (b) the performance of EE trained and evaluated on individual
Figure 5: Performance, evaluated 30 days after disclosure, categories of features (i.e. only considering instances which
of (a) EE compared to baselines, (b) individual feature cate- have artifacts within these categories). We observe that the
gories. We report the Area under the Curve (AUC) and list handcrafted features are the worst performing category, per-
the corresponding TPR/FPR curves in Appendix A.1. haps due to the fact that the 53 features are not sufficient to
capture the large diversity of vulnerabilities in our dataset.
These features encode the existence of public PoCs, which
on CWE-79 and CWE-89, the two features which inflict the is often used by practitioners as a heuristic rule for deter-
most performance drop, FC is not able to correct the noise mining which vulnerabilities must be patched urgently. Our
even with perfect priors, highlighting the inability of the ex- results suggest that this heuristic provides a weak signal for
isting technique to capture feature-dependent noise. In con- the emergence of functional exploits, in line with prior work
trast, we observe that FFC provides a significant performance predicting exploits [2, 31, 64], which concluded that PoCs
improvement. Even for the feature inducing the most degra- "are not a reliable source of information for exploits in the
dation, CWE-79, the FFC AUC is restored within 0.03 points wild" [2]. Nevertheless, we can achieve a much higher preci-
of the pristine classifier, although suffering a slight precision sion at predicting exploitability by extracting deeper features
drop. On most features, FCC approaches the performance of from the PoCs. The PoC Code features provide a 0.93 preci-
the pristine classifier, in spite of being based on inaccurate sion for half of the exploited vulnerabilities, outperforming
prior estimates. all other categories. This suggests that code complexity can
Our result highlights the overall benefits of identifying be a good indicator for the likelihood of functional exploits,
potential sources of feature-dependent noise, as well as the although not on all instances, as indicated by the sharp drop
need for noise correction techniques tailored to our problem. in precision beyond the 0.5 recall. A major reason for this
In the remainder of this section we will use the FFC with drop is the existence of post-exploit mitigation techniques:
pef = 0 (which is equivalent to BCE), in order to observe how even if a PoC is complex and contains advanced functionality,
the classifier performs in absence of any noise priors. defenses might impede successful exploitation beyond denial
of service. This highlights how our feature extractor is able
to represent PoC descriptions and code characteristics which
7.2 Effectiveness of Exploitability Prediction reflect exploitation efforts. Both the PoC and Write-up fea-
Next, we evaluate the effectiveness of our system compared to tures, which EE capitalizes on, perform significantly better
the three static metrics described in Section 5, as well as two than other categories.
state-of-the-art classifiers from prior work. These two predic- Surprisingly, we observe that social media features, are
tors, EPSS [31], and the Social Media Classifier (SMC) [55], not as useful for predicting functional exploits as they are
were proposed for exploits in the wild and we re-implement for exploits in the wild [55], a finding reinforced by our ad-
and re-train them for our task. EPSS trains an ElasticNet ditional experiments from the technical report [60], which
regression model on the set of 53 hand-crafted features ex- show that they do not improve upon other categories. This
tracted from vulnerability descriptors. SMC combines the is because tweets tend to only summarize and repeat infor-
social media features with vulnerability information features mation from write-ups, and often do not contain sufficient
from NVD to learn a linear SVM classifier. For both baselines, technical information to predict exploit development. Besides,
we perform hyper-parameter tunning and report the highest they often incur an additional publication delay over the orig-
performance across all experiments, obtained using λ = 0.001 inal write-ups they quote. Overall, our evaluation highlights
for EPSS and C = 0.0001 for SMC. SMC is trained starting a qualitative distinction between the problem of predicting
from 2015, as our tweets collection does not begin earlier. functional exploits and that of predicting exploits in the wild.
In Figure 5a we plot the precision-recall trade-off of the EE improves when combining artifacts. Next we look at
classifiers trained on dataset DS1, evaluated 30 days after the the interaction among features on dataset DS2. In Figure 6a

388 31st USENIX Security Symposium USENIX Association

 1 0.5
Exploited P CV SS P EPSS P EE (δ) P EE (δ + 10) P EE (δ + 30) P EE (2020-12-07)
Non-Exploited
Mean 0.51 0.36 0.31 0.25 0.22 0.04
0.9 0.4
Std 0.24 0.28 0.33 0.25 0.27 0.11
Median 0.35 0.40 0.22 0.12 0.10 0.00
0.8 0.3
Precision

Mean ε
Table 6: Performance of EE and baselines at prioritizing crit-
0.7 0.2
ical vulnerabilities. P captures the fraction of recent non-
0.6
exploited vulnerabilities scored higher than critical ones.
0.1
Component: NVD & Write-Ups (AUC = 0.85)
Component: Poc Info & Code (AUC = 0.88)
All: EE (AUC = 0.92)
0.5
0 0.2 0.4 0.6 0.8 1
0
0 5 10 15 20 25 30
bilities: on average, only 10% of the non-exploited vulnera-
Recall EE Prediction Delay bilities disclosed on the same day will be scored higher than
(a) (b) an exploited one. However, the score tends to overestimate
Figure 6: (a) Performance of EE compared to constituent the exploitability of non-exploited vulnerabilities, resulting
subsets of features. (b) P evaluated at different points in time. in many false positives. This is in line with prior observa-
tions that static exploitability estimates available at disclosure
we compare the performance of EE trained on all feature sets, have low precision [55]. By following the two curves along
with that trained on PoCs and vulnerability features alone. the X-axis, we observe the benefits of time-varying features.
We observe that PoC features outperform these from vul- Over time, the errors made on non-exploited vulnerabilities
nerabilities, while their combination results in a significant decrease substantially: while such a vulnerability is expected
performance improvement. The result highlights the two cat- to be ranked above 44% exploited ones on the day of dis-
egories complement each other and confirm that PoC features closure, it will be placed above 14% such vulnerabilities 10
provide additional utility for predicting exploitability. On the days later. The plot also shows that this sharp performance
other hand, as described in detail in the technical report [60], boost for the non-exploited vulnerabilities incurs a smaller
we observe no added benefit when incorporating social media increase in error rates for the exploited class. We do not ob-
features into EE. We therefore exclude them from our final EE serve great performance improvements after 10 days from
feature set. disclosure. Overall, we observe that time-varying exploitabil-
EE performance improves over time. In order to evaluate ity contributes to a substantial decrease in the number of false
the benefits of time-varying exploitability, the precision-recall positives, therefore improving the precision our estimates. To
curves are not sufficient, because they only capture a snapshot complement our evaluation, the precision-recall trade-offs at
of the scores in time. In practice, the EE score would be various points in time is reported in Appendix A.1.
compared to that of other vulnerabilities disclosed within a
short time, based on their most recent scores. Therefore, we
7.3 Case Studies
introduce a metric P to compute the performance of EE in
terms of the expected probability of error over time. In this section we investigate the practical utility of EE through
For a given vulnerability i, its score EEi (z) computed on two case studies.
date z and its label Di (Di = 1 if i is exploited and 0 other- EE for critical vulnerabilities. To understand how well EE
wise), the error P EE (z, i, S) w.r.t. a set of vulnerabilities S is distinguishes important vulnerabilities, we measure its per-
computed as: formance on a list of recent ones flagged for prioritized re-

 ||{D j =0∧EE j (z)≥EEi (z)| j∈S}|| if Di = 1 mediation by FireEye [21]. The list was published on De-
||S||
P EE (z, i, S) = ||{D j =1∧EE j (z)≤EE i (z)| j∈S}||
cember 8 2020, after the corresponding functional exploits

||S|| if Di = 0 were stolen [22]. Our dataset contains 15 of the 16 critical
If i is exploited, the metric reflects the number of vulnerabil- vulnerabilities.
ities in S which are not exploited but are scored higher than We measure how well our classifier prioritizes these vul-
i on date z. Conversely, if i is not exploited, P computes nerabilities compared to static baselines, using the P prioriti-
the fraction of exploited vulnerabilities in S which are scored zation metric defined in the previous section, which computes
lower than it. The metric captures the amount of effort spent the fraction of non exploited vulnerabilities from a set S that
prioritizing vulnerabilities with no known exploits. For both are scored higher than the critical ones. For each of the 15 vul-
cases, a perfect score would be 0.0. nerabilities, we set S to contain all others disclosed within 30
For each vulnerability, we set S to include all other vulner- days from it, which represent the most frequent alternatives
abilities disclosed within t days after its disclosure. Figure 6b for prioritization decisions. Table 6 compares the statistics
plots the mean P over the entire dataset, when varying t for the baselines, and for P EE computed on the date criti-
between 0 and 30, for both exploited and non-exploited vul- cal vulnerabilities were disclosed δ, 10 and 30 days later, as
nerabilities. We observe that on the day of disclosure, EE well as one day before the prioritization recommendation was
already provides a high performance for exploited vulnera- published. CVSS scores are published a median of 18 days

USENIX Association 31st USENIX Security Symposium 389

 1 1
EE at disclosure EE w 0% PoCs functional We further consider the possibility that the timestamps in
EE after 10 days EE w 10% PoCs functional
0.9 EE after 20 days 0.9 EE w 30% PoCs functional DS3 may be affected by label noise. We evaluate the potential
Time-varying AUC Score

Time-varying AUC Score

CVSS Exploitability EE w 70% PoCs functional
0.8 EPSS 0.8 EE w 90% PoCs functional
impact of this noise with an approach similar to the one in
Section 7.1. We simulate scenarios where we assume that a
0.7 0.7 percentage of PoCs are already functional, which means that
0.6 0.6
their later exploit-availability dates in DS3 are incorrect. For
those vulnerabilities, we update the exploit availability date
0.5 0.5
to reflect the publication date of these PoCs. This provides a
0.4 0.4 conservative estimate, because the mislabeled PoCs could be
5 10 15 20 25 30 5 10 15 20 25 30
Exploit Publ. Delay (# days after disclosure) Exploit Publ. Delay (# days after disclosure) in an advanced state of development, but not yet fully func-
(a) (b) tional, and the exploit-availability dates could also be set too
early. We simulate percentages of late timestamps ranging
Figure 7: Time-varying AUC when distinguishing exploits from 10–90%. Figure 7b plots the performance of EE(δ) in
published within t days from disclosure (a) for EE and base- this scenario, averaged over 5 repetitions. We observe that
lines, (b) simulating earlier exploit availability. even if 70% of PoCs are considered functional, the classi-
fier outperforms the baselines and maintains an AUC above
after disclosure, and we observe that EE already outperforms 0.58, Interestingly, performance drops after disclosure and
static baselines based only on the features available at dis- is affected the most on predicting exploits published within
closure, while time-varying features improve performance 12 days. Therefore, the classifier based on disclosure-time
significantly. Overall, one day before the prioritization recom- artifacts learns features of easily exploitable vulnerabilities,
mendation is issued, our classifier scores the critical vulner- which get published immediately, but does not fully capture
abilities below only 4% of these with no known exploit. In the risk of functional PoC that are published early. We miti-
our technical report [60] we list the individual vulnerabilities, gate this effect by updating EE with new artifacts daily, after
their scores, and analyze the factors impacting performance disclosure. Overall, the result suggests that EE may be use-
for various examples. Our results indicate that EE is a valu- ful in emergency response scenarios, where it is critical to
able input to patching prioritization frameworks, because it urgently patch the vulnerabilities that are about to receive
outperforms existing metrics and improves over time. functional exploits.
EE for emergency response. Next, we investigate how well
our classifier can predict exploits published shortly after dis-
closure. To this end, we look at the 924 vulnerabilities in DS3 8 Related Work
for which we obtained exploit publication estimates. In the
technical report [60] we perform a statistical test and conclude Predicting exploits in the wild. Most of the prior exploit
that DS3 is a representative sample of all other exploits in our prediction work has been towards the tangential task of pre-
dataset. dicting exploits in the wild. This has been investigated in our
We measure the performance of EE at predicting vulnerabil- prior study [55] and Chen et al. [11] by monitoring Twitter
ities exploited within t days from disclosure. For a given vul- for vulnerability discussions, and Xiao et al. [73] by using
nerability i and EEi (z) computed on date z, we can compute post-disclosure field data about exploitation. Jacobs et al. [30]
the time-varying sensitivity Se = P(EEi (z) > c|Di (t) = 1) and used vulnerability prevalence data to improve prediction. Ja-
specificity Sp = P(EEi (z) ≤ c|Di (t) = 0) [26], where Di (t) cobs et al. [31] proposed EPSS, a scoring system for exploits.
indicates whether the vulnerability was already exploited by Allodi [1] calculated the likelihood of observing exploits in
time t. By varying the detection threshold c, we obtain the the wild after they are traded in underground forums.
time-varying AUC of the classifier which reflects how well Vulnerability Exploitability. Allodi and Massacci [2] inves-
the classifier separates exploits happening within t days from tigated the utility of the CVSS scores for capturing the likeli-
these happening later on. In Figure 7a we plot the AUC for hood of attacks in the wild. Prior work by Bozorgi et al. [7]
our classifier evaluated on the day of disclosure δ, as well as formulated exploitability estimation as the problem of predict-
10 and 20 days later, for the exploits published within 30 days. ing the existence of PoCs based on vulnerability characteris-
While the CVSS Exploitability remains below 0.5, EE(δ) tics. Allodi and Massaci [3] concluded that the publication of
constantly achieves an AUC above [Link] suggests that a PoC in ExploitDB is not a good indicator for exploits in the
the classifiers implicitly learns to assign higher scores to vul- wild. Our work shows that, while their presence might not
nerabilities that are exploited sooner than to these exploited be a sufficiently accurate indicator, the features within these
later. For EE(δ + 10) and EE(δ + 20), in addition to similar PoCs are useful for predicting functional exploits. DarkEm-
trends over time, we also observe the benefits of additional bed [64] uses natural language models trained on private data
features collected in the days after disclosure, which shift the from underground forum discussions to predict the availabil-
overall prediction performance upward. ity of exploits, but such artifacts are generally published with

390 31st USENIX Security Symposium USENIX Association

a delay [1]. Instead, EE uses only publicly available artifacts References
for predicting exploits soon after disclosure; we were unable
to obtain these artifacts for comparison upon contacting the [1] L. Allodi. Economic factors of vulnerability trade and
authors. exploitation. In Proceedings of the 2017 ACM SIGSAC
PoCs. PoCs were also investigated in measurements on vul- Conference on Computer and Communications Security,
nerability lifecycles. Shahzad et al [57] performed a measure- pages 1483–1499. ACM, 2017.
ment of the vulnerability lifecycle, discovering that PoCs are
[2] L. Allodi and F. Massacci. A preliminary analysis
generally released at the time of disclosure. Mu. et al [40]
of vulnerability scores for attacks in wild. In CCS
manually curated and utilized PoCs to trigger the vulnerabil-
BADGERS Workshop, Raleigh, NC, Oct 2012.
ities. FUZE [72] used PoCs to aid exploit generation for 5
functional exploits. [3] L. Allodi and F. Massacci. Comparing vulnerability
Label Noise. The problem of label noise has been studied severity and exploits using case-control studies. ACM
extensively in machine learning [23], primarily focusing on Transactions on Information and System Security (TIS-
random and class-dependent noise. Limited work focuses on SEC), 17(1):1, 2014.
feature-dependent noise, which requires either strong theoret-
ical guarantees about the sample space [37] or depends on a [4] Symantec attack signatures. [Link]
large dataset of clean labels to learn noise probabilities [74]. .com/security_response/attacksignatures/.
In security, the closest to our work is the study by Deloach
et al [15] which models noise in malware classification as [5] L. Bilge and T. Dumitras, . Before we knew it: an
class-dependent. empirical study of zero-day attacks in the real world.
In ACM Conference on Computer and Communications
Security, pages 833–844, 2012.

[6] BleepingComputer. Ie zero-day adopted by rig exploit

kit after publication of poc code. [Link]
9 Conclusions [Link]/news/security/ie-zero-day
-adopted-by-rig-exploit-kit-after-publicat
ion-of-poc-code/, 2018.
By investigating exploitability as a time-varying process, we
discover that it can be learned using supervised classifica- [7] M. Bozorgi, L. K. Saul, S. Savage, and G. M. Voelker.
tion techniques and updated continuously. We discover three Beyond heuristics: learning to classify vulnerabilities
challenges associated with exploitability prediction. First, it and predict exploits. In KDD, Washington, DC, Jul
is prone to feature-dependent label noise, a type considered 2010.
by the machine learning community as the most challenging.
Second, it needs new categories of features, as it differs qual- [8] Bugtraq. Securityfocus. [Link]
itatively from the related task of predicting exploits in the [Link]/, 2019.
wild. Third, it requires new metrics for performance evalua-
tion, designed to capture practical vulnerability prioritization [9] A. Caliskan-Islam, R. Harang, A. Liu, A. Narayanan,
considerations. C. Voss, F. Yamaguchi, and R. Greenstadt. De-
We design the EE metric, which, on a dataset of 103,137 anonymizing programmers via code stylometry. In
vulnerabilities, improves precision from 49% to 86% over 24th USENIX Security Symposium (USENIX Security
state-of-the art predictors. EE can learn to mitigate feature- 15), pages 255–270, 2015.
dependent label noise, capitalizes on highly predictive fea-
tures that we extract from PoCs and write-ups, improves over [10] A. Chakraborty, M. Alam, V. Dey, A. Chattopadhyay,
time, and has practical utility in predicting imminent exploits and D. Mukhopadhyay. Adversarial attacks and de-
and prioritizing critical vulnerabilities. fences: A survey. arXiv preprint arXiv:1810.00069,
2018.
Acknowledgments. We thank Vulners and Frank Li for their
data. We also thank the anonymous reviewers, Yigitcan [11] H. Chen, R. Liu, N. Park, and V. Subrahmanian. Using
Kaya and Ben Edwards for feedback. This material was sup- twitter to predict when vulnerabilities will be exploited.
ported by a grant from the Department of Defense, the Army In Proceedings of the 25th ACM SIGKDD International
Research Office (W911NF-17-1-0370), the National Sci- Conference on Knowledge Discovery & Data Mining,
ence Foundation (CNS-2000792), and based upon work sup- pages 3143–3152, 2019.
ported by the Defense Advanced Research Projects Agency
(DARPA) under Agreement No. HR00112190093. Approved [12] T. M. Corporation. Common weaknesses enumeration.
for public release; distribution is unlimited. [Link]

USENIX Association 31st USENIX Security Symposium 391

[13] NotPetya Technical Analysis. [Link] [28] IBM. Ibm x-force exchange. [Link]
[Link]/blog/petrwrap-ransomware-technic [Link]/.
al-analysis-triple-threat-file-encryption-
mft-encryption-credential-theft/. [29] Immunity Inc. Canvas. [Link]
com/products/canvas/, 2019.
[14] D2 Security. D2 exploitation pack. [Link]
[30] J. Jacobs, S. Romanosky, I. Adjerid, and W. Baker. Im-
[Link]/[Link], 2019.
proving vulnerability remediation through better exploit
[15] J. DeLoach, D. Caragea, and X. Ou. Android malware prediction. In The 2019 Workshop on the Economics of
detection with weak ground truth data. In 2016 IEEE Information Security (WEIS), Jun 2019.
International Conference on Big Data (Big Data), pages [31] J. Jacobs, S. Romanosky, B. Edwards, I. Adjerid, and
3457–3464. IEEE, 2016. M. Roytman. Exploit prediction scoring system (epss).
[16] T. F. Dullien. Weird machines, exploitability, and prov- Digital Threats: Research and Practice, 2(3), July 2021.
able unexploitability. IEEE Transactions on Emerging [32] D. Landman, A. Serebrenik, E. Bouwers, and J. J. Vinju.
Topics in Computing, 2017. Empirical analysis of the relationship between cc and
sloc in a large corpus of java methods and c functions.
[17] O. J. Dunn. Multiple comparisons among means. Jour-
Journal of Software: Evolution and Process, 28(7):589–
nal of the American statistical association, 56(293):52–
618, 2016.
64, 1961.
[33] F. Li and V. Paxson. A large-scale empirical study
[18] C. Eiram. Exploitability/priority index rating systems of security patches. In Proceedings of the 2017 ACM
(approaches, value, and limitations), 2013. SIGSAC Conference on Computer and Communications
[19] Microsoft exploitability index. Microsoft, 30 March Security, pages 2201–2215, 2017.
2020. [Link] [34] Y. Liu, A. Sarabi, J. Zhang, P. Naghizadeh, M. Karir,
/exploitability-index. M. Bailey, and M. Liu. Cloudy with a chance of breach:
Forecasting cyber security incidents. In 24th USENIX
[20] ExploitDB. The exploit database. [Link]
Security Symposium (USENIX Security 15), pages 1009–
[Link]/, 2019.
1024, 2015.
[21] FireEye. Fireeye red team tools. [Link] [35] G. S. Mann and A. McCallum. Simple, robust, scalable
om/fireeye/red_team_tool_countermeasures/b semi-supervised learning via expectation regularization.
lob/master/CVEs_red_team_tools.md. In Proceedings of the 24th international conference on
[22] FireEye. Unauthorized access of fireeye red team tools. Machine learning, pages 593–600, 2007.
[Link] [36] M. Melis, D. Maiorca, B. Biggio, G. Giacinto, and
ch/2020/12/unauthorized-access-of-fireeye- F. Roli. Explaining black-box android malware de-
[Link]. tection. In 2018 26th European Signal Processing
Conference (EUSIPCO), pages 524–528. IEEE, 2018.
[23] B. Frénay and M. Verleysen. Classification in the pres-
ence of label noise: a survey. IEEE transactions on [37] A. K. Menon, B. Van Rooyen, and N. Natarajan. Learn-
neural networks and learning systems, 25(5):845–869, ing from binary labels with instance-dependent corrup-
2013. tion. arXiv preprint arXiv:1605.00751, 2016.
[24] GitHub. linguist. [Link] [38] MetaCPAN. Compiler::parser. [Link]
nguist, 2020. rg/pod/Compiler::Parser, 2020.

[25] Hackerone - disclosure guidelines. HackerOne, 30 [39] Mila Parkour. Contagio dump. [Link]
March 2009. [Link] [Link]/2010/06/overview-of-exploi
sure-guidelines. [Link], 2019.

[26] P. J. Heagerty, T. Lumley, and M. S. Pepe. Time- [40] D. Mu, A. Cuevas, L. Yang, H. Hu, X. Xing, B. Mao,
dependent roc curves for censored survival data and a and G. Wang. Understanding the reproducibility
diagnostic marker. Biometrics, 56(2):337–344, 2000. of crowd-reported security vulnerabilities. In 27th
USENIX Security Symposium (USENIX Security 18),
[27] T. W. House. Vulnerabilities equities policy and process pages 919–936, Baltimore, MD, Aug. 2018. USENIX
for the united states government, 2017. Association.

392 31st USENIX Security Symposium USENIX Association

[41] A. Nappa, R. Johnson, L. Bilge, J. Caballero, and [54] Ruby. Ripper. [Link]
T. Dumitras, . The attack of the clones: A study of 5.1/libdoc/ripper/rdoc/[Link], 2020.
the impact of shared code on vulnerability patching. In
S&P, 2015. [55] C. Sabottke, O. Suciu, and T. Dumitras, . Vulnerability
disclosure in the age of social media: Exploiting Twitter
[42] A complete guide to the common vulnerability scoring for predicting real-world exploits. In USENIX Security
system. [Link] Symposium, Washington, DC, Aug 2015.
ification-document.
[56] R. B. Security. Cvssv3: New system, new problems
[43] National vulnerability database. [Link] (file-based attacks). [Link]
ov/. [Link]/2017/01/16/cvssv3-new-system-new-p
roblems-file-based-attacks/.
[44] Alienvault otx. AlienVault, 30 March 2009. https:
//[Link]/. [57] M. Shahzad, M. Z. Shafiq, and A. X. Liu. A large scale
[45] G. Patrini, A. Rozza, A. Krishna Menon, R. Nock, and exploratory analysis of software vulnerability life cycles.
L. Qu. Making deep neural networks robust to label In Proceedings of the 2012 International Conference on
noise: A loss correction approach. In Proceedings of Software Engineering, 2012.
the IEEE Conference on Computer Vision and Pattern [58] SkyBox. Vulnerability center. [Link]
Recognition, pages 1944–1952, 2017. [Link]/#home, 2019.
[46] F. Pendlebury, F. Pierazzi, R. Jordaney, J. Kinder, and
[59] K. Soska and N. Christin. Automatically detecting
L. Cavallaro. {TESSERACT}: Eliminating experimen-
vulnerable websites before they turn malicious. In
tal bias in malware classification across space and time.
Proceedings of the 23rd USENIX Security Symposium,
In 28th {USENIX} Security Symposium ({USENIX} Se-
San Diego, CA, USA, August 20-22, 2014., pages 625–
curity 19), pages 729–746, 2019.
640, 2014.
[47] Massive microsoft patch tuesday security update for
march. Qualys, 30 March 2017. [Link] [60] O. Suciu, C. Nelson, Z. Lyu, T. Bao, and T. Dumitras.
[Link]/laws-of-vulnerabilities/2017/03/14 Expected exploitability: Predicting the development
/massive-security-update-from-microsoft-fo of functional vulnerability exploits. arXiv preprint
r-march. arXiv:2102.07869, 2021.

[48] Microsoft resumes security updates with ’largest’ patch [61] Symantec Corporation. Symantec threat explorer. ht
tuesday release. Redmont Mag, 30 March 2017. https: tps://[Link]/security-center/a-z,
//[Link]/articles/2017/03/14/march 2019.
-[Link].
[62] L. Szekeres, M. Payer, T. Wei, and D. Song. Sok:
[49] Python. ast. [Link] Eternal war in memory. In 2013 IEEE Symposium on
y/[Link], 2020. Security and Privacy, pages 48–62. IEEE, 2013.

[50] Rapid7. The metasploit framework. [Link] [63] Player 3 has entered the game: say hello to ’wannacry’.
[Link]/, 2019. [Link]
/[Link].
[51] Severity ratings. RedHat, 30 March 2009. https:
//[Link]/security/updates/class [64] N. Tavabi, P. Goyal, M. Almukaynizi, P. Shakarian, and
ification/. K. Lerman. Darkembed: Exploit prediction with neural
language models. In AAAI, 2018.
[52] Microsoft correctly predicts reliable exploits just 27
Reuters, 30 March 2009. [Link] [65] Tenable. Tenable research advisories. [Link]
com/article/urnidgns852573c400693880002576 [Link]/security/research, 2019.
630073ead6/microsoft-correctly-predicts-re
liable-exploits-just-27-of-the-time-idUS18 [66] Tenable Network Security. Nessus vulnerability scanner.
6777206820091104. [Link]

[53] First probabilistic cyber risk model launched by RMS. [67] Severity vs. vpr. Tenable, 30 March 2019. https:
[Link] //[Link]/tenablesc/Content/RiskM
istic-cyber-risk-model-launched-by-rms/. [Link].

USENIX Association 31st USENIX Security Symposium 393

 1 1
[68] Twitter. Filtered stream. [Link]
[Link]/en/docs/twitter-api/tweets/filtere 0.8 0.8
d-stream/introduction.

True Positive Rate

True Positive Rate

0.6 0.6
[69] Virustotal. Virus total. [Link].
0.4 0.4
[70] Vulners. Vulners vulnerability database. [Link]
NVD Info (AUC = 0.79)
[Link]/. Handcrafted (EPSS) (AUC = 0.76)
Social Media (AUC = 0.83)
0.2 0.2
EE (AUC = 0.84) Write-Ups (AUC = 0.85)
Baseline: EPSS (AUC = 0.71) PoC Info (AUC = 0.86)
[71] Y. Watanabe. Assessing security risk of your containers Baseline: Social Media Clf (AUC = 0.85) PoC Code (AUC = 0.82)
0 0
with vulnerability advisor. IBM, 30 March 2019. http 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
False Positive Rate False Positive Rate
s://[Link]/ibm-cloud/assessing-securit
y-risk-of-your-containers-with-vulnerabili (a) (b)
ty-advisor-f6e45fff82ef. Figure 8: ROC curves for the corresponding precision-recall
curves in Figure 5.
[72] W. Wu, Y. Chen, J. Xu, X. Xing, X. Gong, and W. Zou.
1
{FUZE}: Towards facilitating exploit generation for
kernel use-after-free vulnerabilities. In 27th USENIX
0.8
Security Symposium (USENIX Security 18), pages 781–
797, 2018.

True Positive Rate

0.6

[73] C. Xiao, A. Sarabi, Y. Liu, B. Li, M. Liu, and T. Dumi-

0.4
tras. From patching delays to infection symptoms: using
risk profiles for an early discovery of vulnerabilities ex- 0.2
ploited in the wild. In 27th USENIX Security Symposium Component: NVD & Write-Ups (AUC = 0.84)
Component: Poc Info & Code (AUC = 0.87)
All: EE (AUC = 0.91)
(USENIX Security 18), pages 903–918, 2018. 0
0 0.2 0.4 0.6 0.8 1
False Positive Rate
[74] T. Xiao, T. Xia, Y. Yang, C. Huang, and X. Wang. Learn-
ing from massive noisy labeled data for image classi- Figure 9: ROC curve for the corresponding precision-recall
fication. In Proceedings of the IEEE conference on curves in Figure 6a.
computer vision and pattern recognition, pages 2691–
2699, 2015.
the performance gains are not as large later on: the AUC at
[75] F. Yamaguchi, N. Golde, D. Arp, and K. Rieck. Model- 30 days being within 0.02 points of that at 365 days. This
ing and discovering vulnerabilities with code property suggests that the artifacts published within the first days af-
graphs. In 2014 IEEE Symposium on Security and ter disclosure have the highest predictive utility, and that the
Privacy, pages 590–604. IEEE, 2014. predictions made by EE close to disclosure can be trusted to
deliver a high performance.
[76] ZDNet. Recent windows alpc zero-day has been
exploited in the wild for almost a week. https:
//[Link]/article/recent-windows-a
lpc-zero-day-has-been-exploited-in-the-wil
d-for-almost-a-week/, 2018. 1 1

0.9 0.8

A Appendix
True Positive Rate

0.8 0.6
Precision

A.1 Evaluation
0.7 0.4

Additional ROC Curves. Figures 8 and 9 highlight the trade-

0.6
offs between true positives and false positives in classification. EE at disclosure (AUC = 0.87)
EE after 10 days (AUC = 0.89)
0.2 EE at disclosure (AUC = 0.85)
EE after 10 days (AUC = 0.88)
EE after 30 days (AUC = 0.90) EE after 30 days (AUC = 0.88)
EE performance improves over time. To observe how our 0.5
EE after 365 days (AUC = 0.92) EE after 365 days (AUC = 0.91)
0
classifier performs over time, in Figure 10 we plot the perfor- 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
Recall False Positive Rate
mance when EE is computed at disclosure, then 10, 30 and
365 days later. We observe that the highest performance boost (a) (b)
happens within the first 10 days after disclosure, where the Figure 10: Performance of EE evaluated at different points in
AUC increses from 0.87 to 0.89. Overall, we observe that time.

394 31st USENIX Security Symposium USENIX Association