This article has been accepted for publication in IEEE Access.

This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3242127

Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000.
Digital Object Identifier 10.1109/[Link]

Operational Design Domain-driven

Coverage for the Safety Argumentation
of Automated Vehicles
PATRICK WEISSENSTEINER1 , GEORG STETTINGER2 , SIDDARTHA KHASTGIR3 , (MEMBER,
IEEE), AND DANIEL WATZENIG1,4 , (Senior Member, IEEE)
1
Virtual Vehicle Research, Inffeldgasse 21A, 8010 Graz, Austria (e-mail: [Link]@[Link])
2
Infineon Technologies AG, 85579 Neubiberg, Germany
3
WMG, The University of Warwick, Coventry CV4 7AL, U.K
4
Graz University of Technology, 8010 Graz, Austria
Corresponding author: Patrick Weissensteiner (e-mail: [Link]@[Link]).
This research received financial support within the COMET K2 Competence Centers for Excellent Technologies from the Austrian Federal
Ministry for Climate Action (BMK), the Austrian Federal Ministry for Digital and Economic Affairs (BMDW), the Province of Styria
(Dept. 12), and the Styrian Business Promotion Agency (SFG). The Austrian Research Promotion Agency (FFG) has been authorised for
the programme management. They would furthermore like to express their thanks to their supporting industrial and scientific project
partners, namely Infineon Technologies Austria AG, Ing. h. c. F. Porsche AG, Volkswagen AG, aiMotive Kft, Joanneum Research,
University of Klagenfurt, Technical University Graz and University of Graz.

ABSTRACT Automated Vehicles aim to increase road safety as automated driving systems (ADS) take
over the human driving task in the operational design domain (ODD), introducing severe challenges for
safety validation. Pure driving over many kilometres to gather enough evidence for a safety argument is not
feasible. Scenario-based testing is an approach to overcome this, but challenges like parameter discretisation
still prevail, hindering safety assurance. This work proposes contributions towards a traceable and efficient
safety argumentation for ADS built upon ODD coverage. First, ODD coverage is thoroughly quantified
across all scenario levels, assuming distribution functions’ availability for the scenario parameters. Secondly,
a sampling method for n-dimensional scenario parameter distributions is proposed. The provided algorithms
adapt an initial k-means clustering using pre-defined boundary conditions requiring significantly fewer
scenarios. Furthermore, a risk metric for urban intersections is presented for scenario evaluation. The risk
metric consists of two parts, scene prediction of traffic participants (TPs) and risk assessment. The scene
prediction uses a manoeuvre-based motion model with a data-driven approach towards trajectory prediction,
increasing the validity. For the risk assessment, a probabilistic risk prediction for the TPs is performed
for each scenario scene. The risk metric shows a reasonable tradeoff between sensitivity and specificity,
outperforming time-to-collision. These contributions are exemplarily applied at an intersection using a
simplified setup for generating TPs and ego vehicle trajectories. The results indicate that an increased safety
argumentation is enabled using the proposed methods alongside a coverage process, facilitating further
research.

INDEX TERMS Automated vehicles, autonomous vehicles, automated driving, validation, coverage,
operational design domain, automotive safety, vehicle safety, safety argumentation

I. INTRODUCTION Automated vehicles (AVs) are cyber-physical systems op-

erating in open context [2]. AVs are equipped with automated
N 2016, more than 1.3 million traffic deaths happened
I worldwide. Compared to the growing global population,
the death rate has, however, at least stagnated. Taking the in-
driving systems (ADS) which are expected to increase the
aforementioned safety benefits of vehicle automation further
while enabling new mobility methods, e.g., robotaxis, in
creasing motorisation in large parts of the world into account complex urban areas. ADS are SAE Level 3 to 5 [3] as
suggests that the rising safety measures in modern vehicles they take over the complete driving task without human
positively contribute to safety. [1]

VOLUME 4, 2016 1

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see [Link]
 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3242127

Weissensteiner et al.: Operational Design Domain-driven Coverage for the Safety Argumentation of Automated Vehicles

supervision. The boundaries of operation for such systems framework is clearly defined, specific details for an actual
are specified by the target operational design domain (ODD). execution still need to be clarified, including methods for
The description of an ODD contains a set of parameters and choosing CS’ and metrics for scenario evaluation. This article
their respective ranges. A respective taxonomy for such an aims to contribute to the resolution of these current issues.
ODD description is given in [4]. The open context brings var- There are also certain approaches in the industry towards
ious difficulties for the safety validation of these systems. In the safety validation of ADS. General views on safety valida-
general, the definition of safety for ADS is a difficult concept tion and certification are provided in [19] and [20]. A safety
which still needs to be clarified. However, certain initiatives argumentation for assessing an automated lane-keeping sys-
exist (e.g. [5]). For example, in [6], specific interpretations tem (ALKS) focused on virtual test methods is provided in
of safety are defined, including the definition of safety as a [21]. However, the respective ADS has a narrow target ODD
process, which influences the possible validation strategies. and severely bounded behaviour competencies, which makes
Overviews of the different validation strategies are given in the proposed methods only conditionally applicable for more
[7] and [8], respectively. complex use cases.

A. CHALLENGES OF SAFETY VALIDATION FOR ADS II. STRUCTURE OF THE ARTICLE

Most of these strategies use the concept of scenarios, intro- The remainder of the article is structured as follows: First,
duced in [9] and [10]. In there, different types of scenarios Sec. III will introduce the proposed methods and key results
are defined. The logical scenario (LS) is a model of time of this article, which all contribute towards an increased
sequences of consecutive scenes with parameters represented safety argumentation. Secondly, in Sec. IV, the methods are
as ranges. The concrete scenario (CS) is an instance of a LS, applied exemplarily. The generated results and their implica-
with concrete values for each LS parameter. The continuous tions are discussed in Sec. V. The article concludes with a
parameter (CP) is part of the LS and can take on every value summary and an outlook in Sec. VI.
between specific pre-defined ranges. In contrast, the discrete
parameter can assume only specific values (e.g. binary val- III. COVERAGE EVALUATION METHOD
ues). However, it must be noted that the discrete parameters As stated in [13], traditional distance-based statistical ap-
can also be used to split up existing LS into individual LS. proaches are insufficient to cover the resulting test space.
These individual LS then contain only CP. Various methods to explore the scenario space exist. In [22],
This leads to various validation strategies using scenario- a method for criticality identification to provide coverage
based testing (SBT) as the basis. An overview is given estimates is proposed. It is uses a criticality indicator for
in [11], [12] and [13]. The general issues regarding those the exploration of the scenario space using a guided search.
strategies include: Another method is to estimate the probability of a random
scenario being critical. The actual probability is unknown for
• Defining the correct scenarios. These can come from a specific ADS, so estimation is necessary [13]. However,
certain databases (with data from traffic analysis, in- deriving confidence statements for such methods can take
cluding accident data) and expert-based approaches. time and effort. They need to be derived for each ADS sepa-
• Providing suitable test methods. Even if the SBT ap- rately and also repeated for every change in the ADS. This
proach offers ways to tackle the open context problem introduces another source of potential error in the overall
of ADS, there is still a tremendous amount of scenarios safety argumentation process.
to be executed, which calls for suitable test methods, However, note that the goal of the safety validation process
mostly virtual testing. lies not in the search for critical scenarios or certain edge
• Evaluating the executed scenarios and deciding on the cases (e.g., [23], [24]). Such a search is essential during the
correct metric to evaluate criticality. development and internal validation of such systems, espe-
• Provide a suitable safety argumentation which includes cially across different design iterations, as it can influence
evidence of the achieved ODD coverage. certain development decisions. Nevertheless, such edge cases
Potential solutions are formulated in various research ini- inherently depend on the performance of the tested ADS
tiatives, e.g. [14] and standardisation activities [15]. The regarding the experienced criticality, based on the pre-defined
question of how safe ADS-equipped AVs need to be for metrics. The safety validation process essentially needs to
operating in real traffic also concerns regulatory bodies on all be seen as a function of the defined target ODD of the
levels. Recently, the United Nations Economic Commission ADS, as this defines the overall search space that needs to
for Europe (UNECE) proposed a regulatory framework for be covered. Additionally it provides the necessary safety
the safety assessment of ADS ( [16], [17] and [18]). The argumentation for the ADS and its respective target ODD.
framework consists of multiple pillars, including a scenario A proposal for such an overall process is shown in [25].
catalogue for conducting SBT, various test methods for sce- To determine the overall coverage, an exploration method
nario execution and assessment procedures. This framework independent of the respective ADS performance is needed.
aims to be repeatable, objective and evidence-based in as- Hence, a deterministic and transparent way to calculate the
sessing ADS safety. While the overall safety assessment overall coverage can only be achieved if it is based on the re-
2 VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see [Link]
 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3242127

Weissensteiner et al.: Operational Design Domain-driven Coverage for the Safety Argumentation of Automated Vehicles

In general, such a process starts by gathering all requirements

Scenario Automated Driving Scenario for the process itself, including the ADS’ target ODD, ma-
(Input parameters) System outcome noeuvers and the overall use case. These requirements are
used for further execution of the process, presented in detail
Scenarios Operational Design Pass/Fail in [25].
Domain Coverage The overall process consists of six steps, depicted in the
left part of Figure 2. In step 1, the target ODD based on the
FIGURE 1. The coverage of the relevant ODD depends on the executed top-level taxonomy from [4] is defined. Next, step 2 defines
scenarios and if these are passed or failed. the occurring disturbances, a concept defined in [14], which
enhances the SBT approach with the underlying physical
principles for each ADS subtask (perception, judgement and
spective input parameters, meaning the executed scenarios. In control in [14]). These disturbances also define the structure
other words, taking an open-loop approach towards coverage of the relevant LS, which is, together with the target ODD
evaluation, as closing the loop and using the feedback of the parameter of step 1, the input for step 3, the LS generation.
experienced criticality in a guided search for the following In step 3 and step 4, the CS’ are created, whereas in step
scenario, would once again incorporate a specific ADS. This 4+ they are executed using specific test method (ranging
is displayed in Figure 1. from real-world proving grounds to virtual testing). Next, the
An overview of different methods for exploring the sce- executed test cases are evaluated using a pre-defined set of
nario space is given in [26]. As mentioned earlier, the focus key performance indicators (KPIs) to determine if the test
is on sampling-based methods, as they provide the possibility was passed or failed (step 5). This is then used as input for
to respect available parameter distributions, enabling the pre- the determination of the achieved coverage (step 6).
viously mentioned advantages. the advantage of which has While the presented process (left part of Figure 2) intro-
also already been explained. Another advantage of sampling duces an overall method together with a stepwise process
methods is that, since the drawn samples are mutually inde- regarding the coverage evaluation of ADS, certain aspects
pendent, the generated test cases can be executed in parallel, still need to be better defined. The aspects concerning the
reducing overall testing time. Sampling methods, in general, core part of the process (Figure 2) include a consistent
tend to be inefficient if uniform distributions are assumed. definition of the coverage itself in the context of SBT, as
Furthermore, the applied discretisation processes are not able introduced in Sec. I, and a respective set of metrics usable
to provide sufficient argumentations for the safety validation for pass/fail determination in an urban use case. Furthermore,
[27]. the exact type of scenarios to include in the overall process,
There are, after all, a few methods to increase efficiency either gathered from scenario databases such as [30] or
for these assumptions (see [28], or [29]). In [29], different derived otherwise (e.g., using expert knowledge), is an open
concepts for the coverage calculation, based on the past question. The first two aspects are being addressed in Sec.
scenario in the overall scenario space, are presented. III-B and Sec. III-D, respectively. However, scenario creation
However, this can be partly addressed using the respec- is an equally important aspect that will be covered in future
tive parameter distributions of the CPs for each LS. These publications.
parameter distributions can e.g. include the likelihood of
exposure to given scenarios. As opposed to using uniform B. DEFINING COVERAGE IN THE EVALUATION
distributions, realistic parameter distributions enable a more PROCESS
efficient design of the sampling process, e.g., by considering As already examined in the beginning of Sec. III, most
that higher probabilities of exposure in the real world are of definitions of coverage in the domain of safety validation
high significance for the safety validation of ADS. A method for ADS are either simplified pass/fail ratios of executed test
for drawing efficient test case samples, while independent of cases or only cover the aspect of coverage partly (e.g., on
ADS implementations, will be shown in Sec. III-C. parameter level) and not in the context of an overall SBT
approach for safety argumentations. Therefore, the aim is to
A. EXPLORING THE PROCESS OF COVERAGE
define coverage in a way that fits SBT, especially considering
EVALUATION
that for most LS descriptions, the associated CPs are defined
as probability density functions (PDFs) using a specific met-
The aim of the coverage evaluation as part of the safety argu- ric. This directly leads to the fact that each CS contributes
mentation for ADS is to systematically ensure safe operation differently to the overall coverage based on the associated
inside a pre-defined ODD. Therefore, a dedicated process metric in the CP PDF. This is in line with the basic idea of
needs to be in place to enable an efficient and traceable way scenarios, where it is acknowledged that not every situation
to cover the target ODD of the ADS. Two crucial aspects of while driving is equally relevant. Furthermore, this means
such a process are: that some information is available regarding the scenarios
• Extendable towards all necessary target ODDs. (based on the utilised metric) which can be utilised for CS
• The concrete ADS implementation is irrelevant. generation. For the remainder of this article, it is assumed
VOLUME 4, 2016 3

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see [Link]
 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3242127

Weissensteiner et al.: Operational Design Domain-driven Coverage for the Safety Argumentation of Automated Vehicles

High-level ODD coverage process Detailed part of the coverage process

1 Target ODD definition Behaviour

1 Scenario creation
competencies
Abstract parameters
Contribution 1.1: Sampling method
Para-
2 Disturbance definition • One-dimensional
meters
• n-dimensional
Scenario structure
Scenario
3 Logical scenario generation database
2 Scenario execution
Concrete scenarios
Parallel
execution
4 Scenario generation 3 Scenario evaluation

Contribution 3.1: Urban risk metric

4+ Scenario execution • Extended for usage in the coverage
process based on current literature

5 Scenario evaluation

Pass/Fail 4 Coverage evaluation

6 ODD coverage evaluation Risk assessment Contribution 4.1:

Feed-
back Detailed coverage process

Contribution 4.2:
Future extension Definition of ODD coverage levels
High-level process step
Contribution 4.3:
Detailed process steps ODD coverage calculation
Key results/ Contributions

FIGURE 2. On the left, the high-level process for the determination of the ODD coverage, based on [25], can be seen. On the right, steps 3 to 6 of the high-level
process are detailed. These four steps are the focus of this article, for which multiple contributions are provided.

that the associated metric for the CP PDF is the probability domain is once again infinite. For each SL,i , there is a number
of occurrence. This means that values of the CP with higher of CS defined, given as SL,i = {SC,1 , SC,2 , . . . , SC,n }. Then
densities are more likely to appear in the real world. How- again, for each SC,i , concrete values are assigned, making it
ever, other metric types can also be defined on the continuous executable as a test case given a chosen test method. This
parameter level and for CS and LS. Other possible metrics is denoted as SC,i = {x1 , x2 , . . . , xn }. At the lowest level,
would be the associated risk of a CS (based on some pre- CPs define the actual values of each SC,i . The CPs for a given
assessment) or specific metadata for a LS (e.g., amount of SL,i are denoted as SP,i = {SCP,1 , SCP,2 , . . . , SCP,n }. The
roads at a junction) or even a combination of those. elements of SP,i are the chosen test case values for each CP,
e.g., SCP,i = {x1 , x2 , . . . , xn }. Essentially, the generated
1) Defining the different levels of coverage CS for each LS can be seen as the cartesian product of each
The previous section already mentioned that the different ele- SCP,i associated with SL,i , as the values and the number of
ments of a scenario (LS, CS, CP) could all have their metrics resulting pairs are equal to the defined CS.
that influence the overall coverage. This fact is combined into Having established the general relationship between the
a general overview, respecting the hierarchy between the dif- different levels of scenario descriptions, a thorough definition
ferent levels and includes statements regarding the test case of coverage is possible. Starting with the overall coverage on
domain and the number of test cases. It can be seen in Figure target ODD level, cODD ∈ [0, 1] is given by
3. Starting from the top, an ADS of SAE Level 5 would
have to cover the entire input parameter space without any |SL |
X
restrictions, leading to infinite test cases. The ODD is limited cODD = cLS,i mLS,i,coverage , (1)
in the case of an ADS of SAE Level 3 or 4 (one level below i=1
in Figure 3). As the amount of test cases is still infinite, using
the SBT approach, the ODD is split into a finite amount of where cLS,i ∈ [0, 1] is the coverage of one specific LS and
LS, denoted as SL = {SL,1 , SL,2 , . . . , SL,n }. However, for a mLS,i,coverage is the value of the specified metric to weigh
respective LS and the associated scenario space, the test case the different LS across each other (e.g., according to the
4 VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see [Link]
 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3242127

Weissensteiner et al.: Operational Design Domain-driven Coverage for the Safety Argumentation of Automated Vehicles

Coverage Test case Amount of

level domain test cases

Complete input
∞ (SAE Level 5) parameter space
(Full ODD)
∞

Coverage
Limited target
∞ (SAE on target
Level 3,4) ODD
ODD (e.g., ∞
urban pilot)
level

Coverage Applicable
N logical
LS LS LS on LS logical
scenarios
level scenarios (LS)

Coverage n test cases

Concrete
CS CS CS on CS
scenarios (CS)
(per logical
level scenario)

CP Coverage continuous n values

CP on CP parameters (per CP)
CP
level (CP)

FIGURE 4. Example PDF of a continuous parameter x as part of a concrete

FIGURE 3. Defining the ODD coverage on different levels, from the target scenario. The area Ai equals the area under the curve between xi , a and
ODD towards the continuous parameters of the concrete scenarios. xi , b and defines the coverage for that paramter range.

probability of occurrence). Over all LS, the sum of this metric It can be observed from Eq. 4 that each generated CS
is defined as contributes differently to the overall coverage, depending on
|SL |
X the covered area of each CP associated with the CS. Note
mLS,i,coverage := 1. (2) that in Eq. 3 it is possible to include another metric (similar
i=1
to mLS,i,coverage on LS level) to adjust the individual weigh-
The individual coverage of a specific LS is ing of each CS. This would further influence how much it
|SL,j | contributes to the overall coverage based on a metric defined
on the CS level. This is omitted for now but will be explored
X
cLS,j = cCS,i , (3)
i=1
in greater detail in future publications.
where cCS,i ∈ [0, 1] is defined as the coverage for a specific
C. PROPOSED SAMPLING METHOD FOR EFFICIENT
CS and is calculated using the following equation:
TEST CASE GENERATION
P|SP,j |
cCP,i After defining a consistent way for ODD coverage calcula-
cCS,i = P|S i=1 , (4)
tion, the parameter discretisation process is examined further

P,j | 1
i=1 |SCP,i | |SL,j | in this section. This process chooses exact values for each CP
where cCP,i ∈ [0, 1] is the coverage on CP level. To make in a LS description to generate CS’. This technically belongs
sure that for a given LS SL,j the calculated coverage in Eq. 3 to step 1, depicted in Figure 2.
does not exceed the defined limit, the individual CS coverage There are two extreme cases to consider when looking at
calculated in Eq. 4 is corrected, using the amount of chosen sampling from a specific PDF for generating exact values of
CP values for each defined CP in the LS description and the CP. On the one hand, choosing only one value and assigning
total amount of CS for the specific LS. At last, cCP,i is given the complete value range of a CP to this value, e.g. using the
as mean, would be possible. How advisable this is, depends on
Z xi,b the shape of the PDF, as a narrow PDF is more suitable than
P r(xi,a ≤ X ≤ xi,b ) = fX (x)dx = cCP,i , (5) a wide one. Therefore, the overall PDF’s variance is essential
xi,a (which is a property influenced by the shape of the PDF).
which equals Ai in Figure 4. This means that for a respective For each area (e.g. Ai in Figure 4), the variance should be as
CP a specific value range will be represented by one concrete slim as possible since, otherwise, the chosen representative
value, which becomes the test case value. This stems from the value can differ a lot from actual values inside the respective
fact that specific values (for given value ranges) are needed value range. On the other hand, using Monte Carlo sampling
to get executable scenarios and to cope with the infinite test with a high number of samples leads to minimal areas and
case domain of each LS. However, the probability that a narrow value ranges for each representative value, reducing
value is occurring based on a given PDF is only defined for the respective variance and approaching zero in case of
parameter ranges. Therefore, an approach combining specific an infinite number of Monte Carlo samples. Even if both
value ranges with representative concrete values is needed. cases are unrealistic, they contribute towards displaying the
This representative value can e.g. be the mean of the value necessity of generating representative samples for given CP
range, as this takes the shape of the associated PDF, for that PDFs in a traceable manner.
specific range, into account. Exploiting the PDF of scenario parameters has already
VOLUME 4, 2016 5

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see [Link]
 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3242127

Weissensteiner et al.: Operational Design Domain-driven Coverage for the Safety Argumentation of Automated Vehicles

been explored in the literature. For example, in [31], the also be the case that parameter densities are already available,
cumulative probability of a CP, based on synthetic data gen- e.g. from extracted LS’ from a scenario database.
erated using traffic simulation, is split into equidistant areas, Step 2: In any case, samples are drawn from the derived
which all encompass the same amount of coverage on the CP PDF in step 1 and scaling is performed. Concretely, we de-
level. In [32], existing parameter distributions are used for note X = {x1 , x2 , . . . , xm } ⊆ [0, 1]d as the representation
conditional sampling based on risk measures. As a concrete of the dataset on which the clustering is performed upon.
method for sampling, an expectation value-based kD-tree Step 3: Performing clustering on X , we first define C =
is used for dividing the parametric space, which can be n- {C1 , C2 , . . . , Ck } as the respective partition of X into k
dimensional [33]. The kD-tree performs a spatial subdivision clusters, with Ci ∩ Cj = ∅, i ̸= j. Additionally, the centroids
based on a pre-defined number of divisions. Each resulting are denoted as Θ = {θ 1 , θ 2 , . . . , θ k }, whereas θ jP∈ [0, 1]d
subspace has an equal amount of points sampled from the is the centroid of Cj . It is expressed as θ j = |C1j | x∈Cj x.
original PDF. It is in principle similar to the method in Now, we define the within cluster distances WCj between
[31], however, extended towards higher dimensions, in case samples x ∈ Ci as
multiple CPs are dependent on each other and are defined X
over a joint PDF, as both methods aim for test cases of equal W (Ci ) = ∥x − θ i ∥22 , (6)
coverage on CP level. x∈Ci
If parameter distributions are used for the CPs of the
defined scenarios (either one-dimensional, meaning one CP, with ∥.∥22 being the squared euclidian distance. This defines
or multi-dimensional, meaning multiple CPs together in a also the variance within each cluster. k-means is then defined
joint PDF), two main aspects need to be considered: as the minimisation of the objective function

• Overall, the chosen test case values need to be as close |C|

X
as possible to the actual occurring values, given a pre- J(C; Θ) = W (Ci ). (7)
defined amount of values. Otherwise, Monte Carlo sam- i=1
pling would be the apparent default solution. The most common algorithm to tackle this problem is Lloyd’s
• The deviation must be limited for each test case value algorithm [36] which uses a greedy strategy to approximate
and must not exceed a certain predefined limit. This is J(C; Θ). As intitialisation of the centroids for the first step,
necessary to ensure that a particular condition is met for the k-means++ intitialisation is used [37]. For the concrete
each test case value, as the first aspect focuses on an implementation, used for generating the benchmark in Sec.
overall condition across all test case values. III-C2, the Scikit-learn [38] package is utilised.
Therefore, in Sec. III-C2, specific metrics will be intro- Step 4: After having completed the initial partition into k
duced to quantify different sampling methods across each clusters, the clusters are now adapted in the following steps.
other, including the proposed sampling method in Sec. First, the probability pCi that x ∈ Ci is calculated using,
III-C1, which covers both mentioned aspects.
|Ci |
pCi = P|C| . (8)
1) Method for generating samples from parameter i=1 |Ci |
distributions of CPs
The two properties of each cluster, stated in Eq. 6 and 8, can
Clustering is an unsupervised learning task already covered
be seen in a qualitative representation in Figure 5 for step 4.
by literature in many aspects [34]. A well-known algorithm
There, the x-axis, denoted as POccurence , represents the indi-
for clustering is called k-means [35], whose objective func-
vidual occurrence probabilities pCi for each cluster, whereas
tion is to minimise the total within-cluster distances (squared
the y-axis, denoted as tr(Σ), is the trace of the covariance
euclidian distances) to the closest centroids. This objective
matrix. This is used as a scalar-valued generalisation for
function directly addresses the first aspect stated at the end
higher dimensions interpreting the deviation in the variance
of Sec. III-C if the centroids from k-means are chosen as test
definition as euclidian distance since in Eq. 6 the squared
case values. To apply clustering, samples must be drawn from
euclidian distance was also interpreted as a variance. If X
the available parameter distributions of the CP. However, the
is a column vector which elements are random variables Xi
second aspect (see the end of Sec. III-C) is still unaddressed.
and Σ is the covariance matrix between each components of
Therefore, the following proposed method will use k-means
X, the scalar valued variance Vars is defined as
as the basis for the first clustering and further adapt this
initial solution to address the second aspect. The steps of Vars (X) = E ∥X − E[X]∥22 .
 
(9)
the sampling method are all depicted in Figure 5 and will
be explained stepwise in the following. After some rewriting, which is omitted, this equals the sum
Step 1: It is assumend that the original data concerning of the individual variances of each Xi , which in turn equals
some CP (in Figure 5 denoted as xi and xj ) is available in tr(Σ). Therefore, in the following, the use of tr(Σ) is equal to
point samples. Using kernel density estimation (KDE) on the W (Ci ). As the overall goal is to bound each within-cluster
data would lead to a respective estimation of the PDF. It could variance W (Ci ), based on some pre-defined condition to
6 VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see [Link]
 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3242127

Weissensteiner et al.: Operational Design Domain-driven Coverage for the Safety Argumentation of Automated Vehicles

1 Derive PDF 2 Resample & Scale 3 Perform clustering

xj xj Point Pj xj Pj from Cluster Ck
1 1

xi

1 xi 1 xi

6 Calculate final 5 Choose cluster 4 Derive boundary

solution adaption strategy condition
Upper/Lower
tr(Σ) boundary line
Boundary line
xj Pi,k Cluster Cs tr(Σ) Boundary line
d Cluster Ck Cluster Ck
Cluster Ck Pi,s
Cluster position at
iteration t

POccurence xi POccurence

FIGURE 5. The six steps of the developed method for efficient sampling from PDFs of continuous parameters are part of logical scenario descriptions. The method
applies to n-dimensional PDFs of n continuous parameters and is pictured for the two-dimensional case.

fulfil the second aspect presented in Sec. III-C, the following Step 5: In this step, a respective adaption strategy needs to
inequality is defined: be specified so that all clusters obey the respective bound-
ary, discussed in step 4. The general idea, as qualitatively
W (Ci ) ≤ g(pCi ) + ϵ ∀ Ci ∈ C, (10) depicted in Figure 5, is to efficiently search for pairs of points
in the sample space that fulfil two criteria:
with ϵ being a threshold value and g being described as
• They belong to different clusters and,
follows:
• the euclidian distance between them falls below a cer-

g : pCi ∈ [0, 1] 7→ k pCi + d ∈ R+ , (11) tain threshold.

These point pairs are potential candidates for exchange
where k and d are parameters to be defined. Eq. 10 deter- between their respective clusters. Because fundamentally,
mines the condition each cluster needs to obey. For a start, the clusters above the boundary need to drop some points to
condition itself, Eq. 11, is expressed as a boundary line and reduce the occurrence probability. As the dropped points, by
reflects the following assumption: For a cluster Ci , where its definition, are rather far off from their respective centroid,
centroid θ i represents the actual test case value, with higher the within-cluster variance is also reduced. Eventually, if
occurrence probability, pCi , there should be a lower within- enough points are dropped, the boundary condition will be
cluster variance WCi . This makes intuitive sense. For a met. This already describes the basics of the developed
cluster representing more of the sampled points than a cluster algorithms. Concretely, two algorithms for cluster reduction
with lower occurrence probability, the imposed condition (in were developed, Alg. 1 for the one-dimensional problem and
this case, the variance) should be more strict. How exactly Alg. 2 as a generalisation for the n-dimensional problem. In
this relationship (↑ pCi ⇒ ↓ WCi ) is characterised, needs the following, both algorithms are explained.
to be explored more in future publications. As stated in Eq. The one-dimensional algorithm, shown in Alg. 1, deals
11, a simple boundary line with two parameters is chosen, with the situation of having clustered samples (re-scaled to
as it reflects the basic intuitive assumption. The parameters [0,1]) from the parameter distribution of a CP, which is
are chosen using linear regression on the points defined by independent of all other CPs defined in a LS. This means
the properties of each cluster, as can be seen in step 4 in that only the direction of the x-axis is left as the degree of
Figure 5. This gives a realistic boundary, as some clusters freedom for dropping or absorbing points in clusters, which
will be above and beyond the line. For future publications, is in maximum two other clusters ("left" and or "right" of the
assuming a predefined boundary condition will be explored cluster in focus). Considering this fact enables an adapted
further. Adjusting the sampling method presented in this arti- algorithm design for the one-dimensional problem. Taking
cle would enable the determination of the required amount of the overall point samples and the clusters and centroids from
test cases, considering that each of these cases needs to obey the k-means clustering as input, and the adapted clusters are
the boundary condition. returned. At first, the clusters are sorted based on the individ-
VOLUME 4, 2016 7

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see [Link]
 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3242127

Weissensteiner et al.: Operational Design Domain-driven Coverage for the Safety Argumentation of Automated Vehicles

ual centroids, from small to large, and the cluster properties, the current cluster itself) are checked for how much they
WCi and pCi , are calculated. Then, until each cluster obeys reduce the defined objective function
the boundary condition, three options are available for each
|S|
cluster. Option 1 is that the cluster is inside the ϵ-threshold, X
which means no further action is taken for this cluster. J(C) = W (Si ) − g(pSi ) − ϵ. (13)
i=1
Option 2 means the cluster is above the boundary line g(pCi ).
Therefore, points are dropped towards the next cluster based Once again, points with less euclidian distance than a pre-
on the sorting. The last option, a cluster being below g(pCi ), defined threshold are considered for the exchange. After hav-
means further points can be added. The euclidian distance ing iterated through every possible combination, the option
threshold between points of different clusters determines which minimises the objective function in Eq. 13 the most, is
how many points are dropped or added. If the threshold is chosen and a new iteration starts.
high, the algorithm needs fewer iterations as more points are
exchanged in each iteration and vice versa. Algorithm 2 N-dimensional algorithm for cluster adaption
Input: X ⊆ [0, 1]d , C, ϵ , threshold
Algorithm 1 One-dimensional algorithm for cluster adaption Output: C
Input: X ⊆ [0, 1], C, Θ, ϵ , threshold, Init: Obtain W (Ci ) & pCi by Eq. 6 & 8 ∀ Ci ∈
number of samples m, number of clusters k C; S from Eq. 12
Output: C repeat
Init: sort(C) after Θ, for Si ∈ S do
Obtain W (Ci ) & pCi by Eq. 6 & 8 ∀ Ci ∈ C for Cj ∈ {C \ Si } do
repeat Di,j = ∥xi − y j ∥2 , x ∈ Ci , y ∈ Si
for i = 1, 2, ..., k do (di,j ) ∈ R|Ci |×|Si |
if W (Ci ) ≤ g(pCi ) + ϵ then for di,j < threshold do
′ ′
continue Ci = Ci ∪ {y j }, Si = {Si \ y j }
else end for
′ ′ ′
for j = 1, 2, ...mi do Ci,j = {C \ {Ci , Si }} ∪ {Ci , Si }
′
dj,l = ∥xj − xl ∥2 ∀ xl ∈ Ci+1 L = L ∪ {Ci,j }
if dj,l < threshold then end for
if W (Ci ) > g(pCi ) then end for
′ ′
Ci = {Ci \ xj }, Ci+1 = Ci+1 ∪ {xj } C = argminC ′ ∈L J(C )
′
else if W (Ci ) < g(pCi ) then update: C = C , S from Eq. 12
Ci+1 = {Ci+1 \ xl }, Ci = Ci ∪ {xl } until S = ∅
end if return C
else
continue Step 6: In this last step, the proposed algorithms, either
end if Alg. 1 or Alg. 2, depending on the dimensionality of the
end for problem, are applied to the original clustering of X using k-
end if means, which leads, qualitatively, to the situation depicted for
end for step 6 in Figure 5, as all clusters, based on their properties,
update: W (Ci ) & pCi using Eq. 6 & 8 ∀ Ci ∈ C obey to the defined boundary condition. Next, the proposed
until W (Ci ) ≤ g(pCi ) + ϵ ∀ Ci ∈ C sampling method (adapted k-means) will be benchmarked
return C against other methods.

The generalisation of the one-dimensional algorithm to- 2) Benchmarking the proposed method for continuous
wards n-dimensions is based on the same principles of ex- parameter sampling
changing points, part of the original dataset X associated to The proposed adapted k-means addresses both aspects de-
different clusters, to obtain a final set C that obeys Eq. 10. fined in Sec. III-C that sampling methods should have. In
At first, the cluster properties WCi and pCi are calculated order to compare this method to other ones and showcase
for each cluster. Additionally, we denote S ⊆ C which is in which aspects the method outperforms others, a short
constructed as: benchmarking is performed for the one-dimensional and two-
dimensional cases. In total, five methods are included in the
S = {Ci ∈ C | W (Ci ) > g(pCi ) + ϵ}. (12) benchmark. The first option is simply sampling from a given
PDF, basically a Monte Carlo approach. As this method was
Now the algorithm repeats the following steps until S is mentioned in Sec. III-C as an extreme case, it was essential to
empty: Iterating through S, all possible other clusters (except include it here. Then, equidistant steps across the value range
8 VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see [Link]
 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3242127

Weissensteiner et al.: Operational Design Domain-driven Coverage for the Safety Argumentation of Automated Vehicles

are used next for the one-dimensional case. For the two- TABLE 1. For test case amounts between 10 and 20 (rows), for each given
method (column) the test case reduction potential in percent when using the
dimensional case, Latin hypercube sampling (LHS), a much adapted k-means, based on the DBM, is given. Positive values refer to
more efficient method based on [39], is used. Both methods reduction potential for the adapted k-means.
discourage the available parameter distribution and assume
an underlying uniform distribution. However, it should be Test case reduction potential (%) compared
to the adapted k-means:
showcased that if a respective PDF is available, it is best to Test cases K-means Equidistant steps Sampling from PDF
utilise it. That is why these methods are included. The last 10 9.1 28.6 65.5
two methods are the k-means and the proposed adapted k- 11 8.3 31.2 65.6
12 7.7 29.4 62.5
means approach, respectively. Next, the two relevant metrics 13 7.1 27.8 63.9
are introduced, followed by the definition of the used param- 14 6.7 26.3 64.1
eter distributions and the actual results of the benchmark. 15 6.2 25.0 61.5
16 5.9 23.8 62.8
The two metrics are: 17 5.6 26.1 61.4
• Distance-based metric (DBM): The idea of this metric 18 5.3 25.0 60.9
19 5.0 24.0 59.6
is that if a large number of samples m is drawn from 20 4.8 23.1 60.8
the parameter distribution, it should be measured how
close the closest test case value is to each of the sample TABLE 2. For test case amounts between 5 and 20 (rows), for each given
points. Concretely this is expressed as method (column) the test case reduction potential in percent compared to the
adapted k-means, based on the DBM, is given. Positive values refer to
m
1 X reduction potential for the adapted k-means.
DBM = min ∥xi − θ∥2 . (14)
m i=1 ∀θ∈Θ
Test case reduction potential (%) compared
to the adapted k-means:
The aim of a sampling method for parameter distribu-
Test cases K-means Sampling from PDF LHS
tions of CP in a SBT approach is therefore to reduce 5 -25 78.3 84.4
DBM as much as possible as it reduces the uncertainty 6 25 76.9 83.3
that is inevitably introduced since not every exact value 7 12.5 73.1 80.6
8 11.1 70.4 81.0
that is possible can also be tested. 9 10 72.7 81.6
• Standard deviation metric (STDM): This metric en- 10 9.1 69.7 80.0
ables measurement of the expected standard deviation 11 8.3 67.6 78.0
12 7.7 66.7 76.0
for a chosen cluster, scaled by the actual occurrence 13 -8.3 65.8 74.0
probability of said cluster. This directly reflects the 14 6.7 66.7 72.0
already mentioned fact that clusters with higher occur- 15 11.8 64.3 70.0
16 -14.3 61.9 68.0
rence probability should have stricter boundaries on the 17 -6.2 62.2 66.0
within-cluster variance. It is defined as 18 14.3 64.0 64.0
p 19 -18.8 57.8 62.0
STDM = W (Ci ) pCi . (15) 20 -5.3 59.2 60.0

Both metrics operate only on clusters and their properties,

which means that they are independent of the dimensions of adapted k-means over the other respective method to achieve
the parameter distributions. the same value for the DBM. Both evaluations are also
To benchmark the methods, two examples, one and two- depicted for one and two dimensions in Figure 6 and Figure
dimensional, have been constructed. For the one-dimensional 7, respectively.
case, the probability density is defined using a mixture of two For the one-dimensional case, the mean of the reduction
Gaussian distributions, potential is approx. 6.5% over the k-means, 26.4% over the
n
X equidistant steps, and 62.6% over sampling from the PDF.
f (x) = wi pi (x), (16) For the two-dimensional case, these values are 2.4% over the
i=1 k-means, 67.3% over sampling from the PDF, and 73.8% over
with n = 2, w1 = 0.55, w2 = 0.45, p1 = N (−2, 1) and using LHS.
p2 = N (2, 0.04). For the two-dimensional case, two one- The achieved results show various things: First, the perfor-
dimensional Gaussian distributions, p1 (x) = N (0.5, 0.25) mance of the adapted k-means in comparison to the k-means
and p2 (x) = N (0.2, 1), are defined. The two-dimensional is not degraded by applying the cluster adaption algorithms,
samples are obtained by adding and subtracting p1 (x) and neither in one nor in two dimensions. Secondly, it can be
p2 (x) for the x and y axis, respectively. Executing all the observed that both clustering techniques outperform all other
methods on this given examples for various amounts of benchmarked methods severely. This indicates, based on the
test cases leads to the results shown in Tab. 1 for the one- applied metrics, that utilising parameter distributions, when
dimensional example, Tab. 2 shows the two-dimensional available, offers a large test case reduction potential on CP
case. For both Tables, the values shown for each column level. This effect is further enhanced on LS level since it is
represent the fractional of the necessary test cases for the typically composed of many individual CP.
VOLUME 4, 2016 9

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see [Link]
 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3242127

Weissensteiner et al.: Operational Design Domain-driven Coverage for the Safety Argumentation of Automated Vehicles

0.016
Sampling from PDF Upper boundary
0.08 Equidistant steps Adapted k-means
0.014
K-means
Adapted k-means
0.07 K-means

Relative standard deviation metric

0.012

Distance-based metric
Distance-based metric

0.06 0.020
0.010

Relative standard deviation metric

0.05 0.015 0.0020
0.008

0.04 0.0015
0.010 0.006
0.0010
0.03
10 20 0.004
Number of test cases 0.0005
0.02
0.002
0.0000
0.00 0.05 0.10
0.01 Probability of occurence
0.000

12 18 24 30 36 42 48 54 60 0.0 0.1 0.2 0.3 0.4 0.5

Number of test cases Probability of occurence

FIGURE 6. The number of test cases over the DBM for the different types of FIGURE 8. For the one-dimensional case, the probability of occurrence for
sampling methods for the one-dimensional case. The number of test cases each cluster is pictured over the STDM. It can be seen that in the case of the
refers to the number of samples drawn from the PDF of the continuous classical k-means approach, one cluster violates the chosen boundary
parameter. condition by being above the upper boundary, depicted in blue.

0.35 Sampling from PDF

LHS
Adapted k-means
corresponds to step 3 (see Figure 2). Evaluating scenarios
0.30 means analysing if a certain amount of criticality did occur
K-means

0.09
during the execution, whereas criticality is defined as ’the
combined risk of the involved actors when the traffic situation
Distance-based metric

0.25
Distance-based metric

0.08
is continued’ [40]. The most obvious way to determine that
0.07 is by checking if an actual collision between entities of a
0.20
0.06 scenario did occur. However, that would disregard certain
0.05 near-misses and would provide no means of indicating the
0.15
0.04
criticality of those scenarios. Therefore, many different criti-
5 10 15 20 cality metrics exist, as they are not only relevant for evaluat-
Number of test cases
0.10 ing scenarios during the safety validation phase using a SBT
approach. There are applications for such metrics along the
0.05 entire V-model [41]. The most well-known metric is the time
to collision (TTC), which calculates the time until a predicted
5 10 15 20 25 30 35 40 45 50
Number of test cases collision between traffic participants [42]. It originates from
traffic conflict research and is, therefore, collision-focused.
FIGURE 7. The number of test cases over the DBM for the different types of
sampling methods for the two-dimensional case. The number of test cases
There exist many adapted and improved versions of TTC. In
refers to the number of samples drawn from the joint PDF of the continuous [43], different accident configurations are considered, taking
parameters. the vehicles’ width and length into account and providing a
more accurate TTC. For a-posteriori traffic data analysis, the
post encroachment time (PET) calculates the time gap be-
At last, for the one-dimensional case, the STDM is ob- tween a traffic participant leaving and another entering a pre-
served. Calculating this metric for each cluster in the given defined area [44]. A comprehensive overview of criticality
example is depicted in Figure 8. The upper boundary line is metrics is given in [45]. The authors of [45] also define a set
created by inserting Eq. 10 into the STDM (Eq. 15). This of properties relevant to such metrics. These properties are
means, that for the adapted k-means the STDM is bounded (excerpt):
based on the given boundary condition. In Figure 8 it can be
observed that one specific cluster of the k-means approach is • Reliability: Describes how close repeated measurements
violating this boundary. All clusters of the adapted k-means are to one another.
are within the given boundary. • Validity: This property is concerned with how close the
measurements are to the actual accident probability.
D. METRICS FOR THE EVALUATION OF SCENARIOS
• Sensitivity: This is the true positive rate (TPR), which is
defined as
This section deals with the evaluation of executed scenarios. TP
Following the main steps of the ODD coverage process, this TPR = , (17)
TP + FN
10 VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see [Link]
 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3242127

Weissensteiner et al.: Operational Design Domain-driven Coverage for the Safety Argumentation of Automated Vehicles

2) Scene prediction
Urban Risk Metric
Many advanced metrics (including the proposed urban risk
Trajectories Scene Prediction Risk Assessment
of Calculation of traffic
metric) calculate the risk for every scenario scene and then
1st Layer:
traffic Scene
participants’
Road network-based
manoeuvre prediction
prediction
participant pairs’
Euclidian distance Risk derive a final risk value for the whole scenario. On that basis,
distribution
the pass/fail criteria are evaluated. To do that, for every scene
2nd Layer:
Traffic participants’
trajectory prediction
Risk calculation in the scenario, a scene prediction needs to be carried out to
calculate criticality based on a predicted future evolvement
of the scenario. This prediction either, for a pre-defined
FIGURE 9. The overall structure of the urban risk metric, based on [51]. prediction horizon, calculates a single evolution or multiple
ones for each traffic participant in the scenario. For the single
evolution, the worst case is mostly chosen, which leads to
whereas TP are the true positives and FN are the false severely reduced specificity. Multiple evolvements can be
negatives. carried out using a probabilistic framework or calculating
• Specificity: Similarily, specificity is the true negative reachable sets using reachability analysis.
rate (TNR), which describes the rate of correctly identi- In [52], motion models for prediction are categorised
fied uncritical situations, expressed as into three levels, with increasing abstraction. Physics-based
models consider the motion of vehicles based on the laws
TN of physics. Manoeuvre-based motion models consider the
TNR = , (18)
TN + FP future motion of the vehicle also based on the potential
with TN and FP being the true negatives and the false driver’s intention. Interaction-aware models are additionally
positives, respectively. taking into account the vehicles’ manoeuvres (e.g., [53],
[54]). Another option is to define potential functions for
In this article, the focus is on the test evaluation of certain
each object type and aggregate these functions to obtain a
scenarios, based on pre-defined pass/fail criteria to determine
manoeuvre model [55].
the overall coverage of the target ODD for a specific ADS.
If a manoeuvre-based motion model is used, the actual
If the focus is on an urban use case, including complex
trajectory of the vehicle needs to be predicted. In [56], the
intersections, TTC is considered to not be an ideal choice,
authors perform trajectory prediction based on observations
since it suffers from reduced validity and sensitivity [45].
using image processing and map-matching techniques. It
Various literature exists that aims to apply distinct risk
is shown that these generated predictions outperform the
metrics towards urban-based use cases. In [46], a probabilis-
usual physics-based motion models in the observed intersec-
tic risk assessment algorithm considering occlusion, focus-
tions. Schreier et al. perform criticality assessment using a
ing on integration into planning algorithms, is proposed. In
manoeuvre-based trajectory prediction utilising a Bayesian
[47], a potential risk assessment for occluded areas in urban
network where each manoeuvre is modelled separately [57].
situations is proposed by threat modelling of the potential
Several other approaches towards trajectory prediction exist,
risk, however, it is designed for real-time purposes. In [48], a
using data-driven machine learning techniques (e.g., [58],
virtual risk assessment, determining the suitability of an au-
[59], [60], [61]).
tomated shuttle deployment in suburban areas, is presented,
The scene prediction for the urban risk metric uses a
taking into account the actual geo-location. Another risk
manoeuvre-based motion model prediction with a data-
metric for urban use cases is presented in [49]. The authors
driven approach towards concrete trajectory prediction. Us-
propose a threat metric in complex urban scenarios using
ing a data-driven approach increases the validity of the ap-
the inD dataset [50]. Moreover, they perform a reachability
plied risk metric, although it is restricted to the locations with
analysis, a technique to compute reachable system states
available data. However, if these locations represent complex
based on parametrised motion models. Also, respective lane
intersections (e.g. included in the inD dataset [50] or the
association is considered. However, this association is static,
INTERACTION dataset [62]), these are vital aspects of the
not considering the typical driver’s intention. This leads to a
defined target ODD, which may justify the additional effort
threat metric which is prone to reduced sensitivity.
associated with gathering the necessary amount of traffic
observations. Concretely, a specific map of the INTERAC-
1) Structure of the developed metric for urban junctions TION dataset is used to develop the scene prediction and
A useful structure of such metric, which also applies towards provide an exemplary application in Sec. IV-A. The dataset
simple ones like TTC, can be seen in Figure 9, which is based provides naturalistic motions of different traffic participants
on [51]. It serves as an overview for the urban risk metric for various complex intersections. One of these intersections
which is proposed to be used to evaluate executed scenarios is chosen to showcase the developed method and can be seen
during the ODD coverage process. The two main parts, the in Figure 10. The junction is provided in the Lanelet2 format
scene prediction and the risk assessment, will be explained in [63]. The chosen map provides an unsignaled intersection
detail in Sec. III-D2 and Sec. III-D3, respectively. and approx. 260 minutes of birds-view video material with
more than 10 thousand vehicles. The dataset can be evaluated
VOLUME 4, 2016 11

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see [Link]
 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3242127

Weissensteiner et al.: Operational Design Domain-driven Coverage for the Safety Argumentation of Automated Vehicles

TABLE 3. Performance comparison of different classifiers used for

Map DR USA Intersection GL from manoeuvre estimation for the manoeuvre decision depicted in Figure 10.
the INTERACTION dataset
First lanelet section Type of classifier Train scores Test scores
1020 Lanelet: Manoeuvre decision Decision Tree Classifier 0.916 0.931
Lanelet: Manoeuvre → Straight Random Forest Classifier 0.944 0.951
Lanelet: Manoeuvre → Turning KNeighbors Classifier 0.933 0.945
Global y-position (m)

1000 SVC 0.908 0.915

Gaussian NB 0.871 0.883
Logistic Regression 0.885 0.895
980

960 where n is the size of the dataset, R3 is the feature space and
L is the label space. Furthermore, yi is the label and xi is the
feature vector of the ith sample, which is expressed as
940
xi = [velocity, global heading, lateral position]T . (20)
920 940 960 980 1000 1020 1040
Global x-position (m) The label space, for the example shown in Figure 10, is
FIGURE 10. One specific map of the INTERACTION dataset [62], provided in
designed to be:
the format Lanelet2 [63].
L = [Turning, Straight] (21)
The goal of the algorithm is to find h : R3 → L so that for
extensively with the provided Lanelet2 map, which provides
new samples (x1 , y1 ) we get h(x) ≈ y.
good starting points for further analysis.
Different classifiers were trained and evaluated for the sit-
Based on the overall structure of the urban risk metric
uation depicted in Figure 10. The results of the test and train
(Figure 9), the scene prediction is divided into two separate
scores (80/20 split) can be seen in Tab. 3. The random forest
layers. The first layer performs a road network-based ma-
classifier (RBF), which fits several decision tree classifiers
noeuvre prediction. In contrast, the second layer performs
on various sub-samples of the dataset [64], performs best
an actual trajectory prediction, providing a respective scene
for the test and the training dataset. In general, it can be
prediction for the second primary step, the risk assessment.
observed that the manoeuvre intention is predictable with
For the road network-based manoeuvre prediction, a clas-
very high accuracy. Therefore, for future trajectories, for
sification problem is formulated, where the goal is to store
example, considering the decision for a right turn or going
pre-trained classifiers for each relevant lanelet section. A
straight in Figure 10, it is not necessary to always chose the
lanelet section is defined by four nodes (two nodes for each
worst case, which would unnecessarily reduce specificity.
side). Doing that enables a better performance of each indi-
Suppose the probability for a particular predicted manoeu-
vidual classifier. All lanelets at respective road forks, and for
vre is below 0.65. In that case, the prediction is disregarded,
which more than one option for future manoeuvres is possi-
all possible manoeuvres are considered future trajectories,
ble, are considered. A respective ground truth (GT) is needed
and a worst-case estimation is performed in the risk assess-
for training the classifiers. The GT is created by analysing the
ment. Future advancements in the urban risk metric regard
provided vehicle tracks in the dataset and matching it with
the investigation of calibrated classifiers. If such classifiers
a pre-defined list of lanelet IDs that belong to a global route
are being used, it would be possible to utilise the different
(e.g. the route from the top left of Figure 10 going right (from
prediction probabilities for each manoeuvre as a weighting
top-view), crossing the red lanelet and taking the immediate
factor for the predicted trajectories, which can be exploited
right turn). Furthermore, certain manoeuvres are connected
for the risk assessment. This requires a classifier with a low
with that global route. Additionally, for these manoeuvres, it
log loss, which is necessary if the probability estimates of the
is defined at which respective lanelet the manoeuvre decision
classifier are utilised directly.
has to be made and which lanelets are the successors. For
As the first layer of the scene prediction provided the
example, for the red lanelet in Figure 10, a decision if turn
manoeuvre estimation, the second layer predicts the actual
right or go straight needs to be made. The features for
future trajectories of the traffic participants. The output is
the classifier include the lateral position (in regards to the
thereby defined by the necessary input to the second major
respective lane in the provided map), the vehicles’ velocity
part of the urban risk metric, which is based on the prob-
and the global heading for each lanelet section. An overview
abilistic evolvement of object positions for the pre-defined
of manoeuvre intention estimation at road intersections and
prediction horizon. As Figure 11 shows, a bivariate Gaussian
its associated features is given in [52].
distribution is constructed for each object and each timestep
In our case, the required supervised learning algorithm acts
∆t. This distribution is described in the respective local
on a dataset D, drawn from an unknown distribution, which
object coordinate system for each prediction timestep. It dis-
is defined as
tinguishes between a longitudinal prediction along the axis in
D = {(x1 , y1 ), . . . , (xn , yn ) ⊆ R3 × L} (19) the driving direction and a lateral prediction perpendicular to
12 VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see [Link]
 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3242127

Weissensteiner et al.: Operational Design Domain-driven Coverage for the Safety Argumentation of Automated Vehicles

5.950
995

N (µ1,1, Σ1,1) N (µ1,2, Σ1,2) 4.463

Global y-position (m)

990

Time horizon (s)

Object 1
985
Lateral prediction 2.975
dE1 980 Adapted center line using
dE2 mean lateral positions from dataset
Center line from lanelet data 1.488
975
Left and right borders from
lanelet data
970 0.000
960 965 970 975 980 985 990
N (µ2,1, Σ2,1) N (µ2,2, Σ2,2) Global x-position (m)
Object 2
FIGURE 12. The trajectory prediction of a traffic participant (Layer 2 of the
time t t1 = t + ∆t t2 = t1 + ∆t scene prediction in the urban risk metric) on the map depicted in Figure 10.

FIGURE 11. The potential evolvement of each object position for a specific
time t in a scenario is modelled as bivariate Gaussian distribution for each
following timestep. vlon,i . These samples are approximated by a Gaussian distri-
bution by calculating the mean and variance of the acquired
acceleration samples, which leads to
this. It is crucial to notice that there is no restriction regarding
the shape of the bivariate Gaussian distribution. 2
alon,i = N (ameanlon,i , σlon,i ). (27)
For the traffic participants’ lateral prediction along the
road, an Ornstein-Uhlenbeck process [65] is modelled based Now, for both directions, longitudinal and lateral, a dis-
on [57]. The process has an exponentially decaying autoco- tribution for predicting the future motion of the respective
variance function and is discretised as traffic participant along the prediction horizon is in place.
The lateral prediction is parametrised by setting α = 0.66.
ylat,i+1 = e−αT ylat,i +(1−e−αT )u+wylat ∀i ∈ {j : j+TP },
The long-term mean u is set to equal the mean of all relevant
(22)
trajectories in the dataset for the specific global route. In
with α being a constant defining the rate of decay, T is the
Figure 12, this is equal to the dashed red line. The original
current time in the prediction horizon, and u is defined as
centre line would be the dashed yellow line derived from
the long-term mean of the process. Furthermore, wylat is the
the respective means of the lanelet boundaries. It can be
process noise scalar, with a variance stated as
observed that directly using an already adapted centre lane
Qi = σy2lat (1 − e−2αT ), (23) as the basis for defining the lateral prediction process is
much more realistic. The limiting variance σy2lat is defined
where σy2lat equals the limiting value of Qi . The notation {j : so that the generated trajectories do not exceed the respective
j + TP } equals the sequence {j, j + 1, . . . , j + TP with TP } lanelets. These parameters have been verified to lead to
being the amount of prediction timesteps, calculated as feasible kinematic predictions using [66].
tpred Observing Figure 12 shows that the lateral prediction
TP = , (24)
∆t converges towards the adapted centre line with ongoing pre-
with tpred being the prediction horizon and ∆t the prediction diction time. For each value, the respective point samples
timestep. This leads to the following Gaussian distribution of for this specific prediction timestep are plotted in the time
the lateral prediction: horizon colour bar of Figure 12. As the prediction progresses,
the variance of the longitudinal prediction grows more sig-
N (e−αT ylat,i + (1 − e−αT ) u, Qi ). (25)
nificant, as the longitudinal position is primarily influenced
The longidutinal prediction for the traffic participant at by potential braking manoeuvres caused by other traffic
each timestep in the prediction horizon is defined as participants. While this is not accounted for in the motion
1 model itself, to a certain degree, this is considered by the
xlon,i+1 = xlon,i +vlon,i ∆t+ alon,i ∆t2 ∀i ∈ {j : j+TP }, growing variance in the prediction, as these situations are part
2
(26) of the dataset samples used for generating the longitudinal
which equals a constant acceleration model, with xlon,i prediction. The variance of the lateral prediction also grows
being the current position along the prediction horizon and more prominent, as modelled with the Ornstein-Uhlenbeck
vlon,i the current velocity. Analysing the INTERACTION process. However, it is outgrown by the longitudinal predic-
dataset at the complex intersection shown in Figure 10 shows tion variance. This makes sense, as the probability that the
that there is a correlation between the current velocity and traffic participant stays inside its respective lane (and deviates
the acceleration of traffic participants. This is utilised by to the calculated mean) is much more probable.
conditionally sampling from the joint PDF of velocity and The generated probability distributions must reflect real-
acceleration, derived for each section of relevant lanelets, istic traffic participant behaviour to serve as valuable scene
to get points samples for accelerations based on the current predictions. To rate the overall scene prediction, a scene
VOLUME 4, 2016 13

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see [Link]
 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3242127

Weissensteiner et al.: Operational Design Domain-driven Coverage for the Safety Argumentation of Automated Vehicles

TABLE 4. Overall accuracy of the scene prediction for the specific example with rindi ∈ [0, 1] and the threshold calculated as
shown in Figure 10.
(
2
0.25 + 0.002 vrel vrel > 0
Sigma interval Miss Rate on 3 second prediction threshold = (29)
1σ 34 % 0.25 vrel ≤ 0.
2σ 8.5 %
3σ 2.6 % The relative velocity between traffic objects vrel is calculated
as a relative velocity vector in the direction of the distance
vector between both points of the forward edge of the objects
prediction for a prediction horizon of 3 seconds is performed as described in [70]. The threshold is increased if vrel is
for each object trajectory which is part of the displayed greater than zero. If dadapted falls beneath the threshold,
first section of the red lanelet in Figure 10. The generated the indicated risk approaches 1. In cases where the thresh-
probability distributions are then compared to the positions old is still higher than dadapted , the risk is increased by a
of all trajectories in the dataset following the same global higher variance σi2 , which is used as a parameter to quantify
route. The 1σ, 2σ and 3σ intervals are constructed based on the uncertainty in the distance estimate dadapted . Higher
the positions after 3 seconds. The resulting overall accuracy uncertainty leads to higher risk estimates in cases above
of the scene prediction based on the specific example shown the threshold. A detailed explanation of how to analytically
in Figure 10 is displayed in Tab. 4. The accuracy for the 3σ compute dadapted and σi2 follows at the end of this section.
interval is already quite good, considering it already includes Next, rindi is used to calculate the event probability
both the manoeuvre prediction at the red lanelet and the  1 − e−rindi (ti ) 
following route prediction. The accuracy could be improved pevent (ti ) = , (30)
further if the trajectories in the dataset are filtered to exclude 1 − e−1
traffic participants performing a braking manoeuvre since with pevent ∈ [0, 1]. Furthermore, the survival probability is
this is not explicitly accounted for in the motion model as calculated as follows:
this would require an interaction-aware model. R ti
psurvival (ti ) = e− 0 pevent (ti )dt
, (31)
3) Risk calculation with psurvival ∈ [0, 1]. The actual collision probability for
Calculating a criticality measure for every scene in a given timestep ti in the prediction horizon is calculated as
scenario, each for a pre-defined prediction horizon, requires a
metric that can be calculated for arbitrary time intervals, also pcoll (ti ) = psurvival (ti ) pevent (ti ), (32)
in the absence of evident object collisions. As the preceding
with pcoll ∈ [0, 1]. The overall collision probability for a
scene prediction outputs respective position probability dis-
scenario is then determined using the maximum value over
tributions, using a probability-based risk calculation is con-
the prediction horizon across all individual scenes.
sistent. The probability distributions of objects in the scene
Observing Figure 11 already indicates that using the eu-
reflect the growing uncertainty moving along the prediction
clidian distance between the means of the probability distri-
horizon and can be accounted for directly in a probabilistic
butions is not sufficient for dadapted , as choosing σi2 would
risk prediction. Eggert et al. [67] use an exponential trans-
still be an open question. The probability distribution of the
form in combination with a survival function to estimate
distance between two bivariate Gaussian probability distri-
future event probabilities. The focus is on collision risk,
butions can be solved analytically. First, the problem needs
which is accounted for by modelling a distance-dependent
to be reformulated. This is achieved by transforming these
risk. Other types of risks could be modelled, e.g. the risk of a
two distributions into a signle one, by applying a coordinate
vehicle losing control in a curve. However, the focus is now
transform so that the mean of one distribution becomes the
on modelling the risk of traffic collisions and potential close
coordinate origin. The final covariance matrix Σd is obtained
calls.
by applying uncertainty propagation as follows:
The future collision event probability is derived by com-  1 0 
bining a survival probability, designed as a survival function T 0 1
Σd = J Σ1,2 J , J = −1 0 , (33)
that decays in proportion to the collision event probabilities. 0 −1
Earlier events significantly influence the predictive risk cal-
with Σ1,2 being the matrix of the combined individual co-
culations as potential risks that occur before others are con-
variance matrices defined as
sidered more important. The distance-dependent risk directly  
influences the collision event probability. More detailed ex- Σ1 0
Σ1,2 = . (34)
planations are in [68], and [69], respectively. 0 Σ2
Next, the concrete implementation of the risk calculation Now the problem is reduced to finding the distribution
is explained. At first, the distance-dependent risk is defined of the distance vector from the coordinate origin to the
as individual points sampled from the combined bivarate Gaus-
1   threshold − dadapted   sian distribution. This problem, however, has been addressed
rindi (ti ) = erf + 1 , (28)
in other domains before. In case the bivarate Gaussian is
p
2 2σi2
14 VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see [Link]
 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3242127

Weissensteiner et al.: Operational Design Domain-driven Coverage for the Safety Argumentation of Automated Vehicles

circularly-symmetric, the actual distance distribution is de- TABLE 5. Three different methods are compared for computing the distance
vector distribution for bivarate Gaussian distributions.
scribed by the Rice distribution [71]. However, since using
a Rice distribution in our case would mean a restriction, Type of method Simulation time (s) Mean (m) Std (m)
as it was already shown in Sec. III-D2 that a circularly- Monte Carlo 4.4223 6.2957 1.3445
symmetric assumption for the individual Gaussians would be Rice distribution 0.0014 6.0907 2.0621
Beckmann distribution 0.0296 6.2833 1.3544
unwanted simplification. However, the Beckmann distribu-
tion describes the distribution for the general case, with no
assumptions [72]. From the Beckmann distribution, various
others, including the Rice distribution can be derived.
Next, a simple example is constructed to showcase the dif- 0.30 Monte Carlo
ferent distributions and their effect on the distance estimate, Rice
also concerning the needed values, mean and variance of the 0.25 distribution
distance distribution, for the distance-dependent risk in Eq. Beckmann
0.20 distribution
28. The position of the first traffic object, in a global coordi-

Density
nate system, is described by a bivariate Gaussian distribution
0.15
of a random vector X 1 = (X, Y )T ∈ R2 with
X 1 ∼ N (µ1 , Σ1 ), (35) 0.10

with µ1 and Σ1 being defined as 0.05

   
4 0.35 0.3 0.00
µ1 = , Σ1 = , (36) 0 5 10 15
4 0.3 1
Euclidian distance (m)
and similar for the second object with the random vector
FIGURE 13. The PDFs of the euclidian distance between two traffic
X 2 = (X, Y )T ∈ R2 with participants (dE in Figure 11) assuming a Rice distribution (red line) and a
Beckmann distribution (orange line) compared to the Monte Carlo evaluation
X 2 ∼ N (µ2 , Σ2 ), (37) (dashed blue line).

with µ2 and Σ2 being expressed as

   
0 3 −3.5 resulting object shapes into account. These ellipses act as
µ2 = , Σ2 = . (38) a safety envelope, similar to the concepts presented in [75]
0 −3.5 5
and [76]. Next, the shortest distance between these two
Three different methods are investigated to construct the
resulting ellipses is determined by sampling, using uniform
resulting distance vector distribution. The already mentioned
distributed ellipse angles as shown in Figure 14. To account
Rice and Beckmann distributions together with the Monte
for the already discussed fact that the distance vector between
Carlo approach. Starting with the latter, samples from both
those two objects, using bivariate Gaussians, actually follows
individual Gaussians are drawn and the euclidian distance
a parameter distribution, the mean of the chosen distribu-
between these points is calculated and stored. Performing
tion (e.g. the presented Beckmann distribution) dmean,dist
this calculation for 10k samples each leads to the distribution
is substracted from the simple euclidian distance deuclidian
depicted in Figure 13 as a dashed blue line. Next, the Rice
between the objects to account for the shifted mean and
and Beckmann distributions are calculated using R, a free
added to the shortest distance between the objects at each
software environment for statistical computing [73] with an
positional mean dshort,ellipses , concretely
additional package [74] for calculating the specific distribu-
tions. In the case of the Rice distribution it can be observed dadapted = dshort,ellipses + (dmean,dist − deuclidian ). (39)
that, while the mean deviates not too much from the Monte
Carlo approach, the variance is much greater. The Beckmann Doing that, the final distance value includes the probabilis-
distribution is solved by numerical integration, which leads tic approach regarding the objects’ position as well as the
to small deviations compared to the Monte Carlo simulation object’s global heading and also the actual dimensions of
around the mean. However, overall, the computed Beckmann each respective object. The variance σi2 for Eq. 28 is directly
distribution is very close to the actual solution (derived using derived from the calculated distance vector distribution.
Monte Carlo simulation). The individual simulation time, the
mean and the standard deviation for all methods can be seen IV. RESULTS
in Tab.5. This section is structured into three main parts. First, the steps
To calculate the actual dadapted for Eq. 28, the shape of the that have been taken to construct an application example of
traffic participants and uncertainty regarding the orientation the coverage process are explained. Secondly, the produced
of the object need to be considered. Based on the objects’ results are presented in detail. Lastly, a detailed evaluation of
current orientation, ± 3° of angular deviation is considered, the urban risk metric is included.
for which an enveloping ellipse is constructed taking the
VOLUME 4, 2016 15

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see [Link]
 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3242127

Weissensteiner et al.: Operational Design Domain-driven Coverage for the Safety Argumentation of Automated Vehicles

0.4
10
Enveloping ellipse
Traffic participant 0.2
8
Global y-position (m)

0.0
6 −25.0 −22.5 −20.0 −17.5 −15.0 −12.5 −10.0
Global heading (Degrees)
4

0.1
2

0 0.0
0.0 2.5 5.0 7.5 10.0 12.5 15.0
Vehicle velocity ( ms )
−2.5 0.0 2.5 5.0 7.5 10.0 12.5 1.0
Global x-position (m)

FIGURE 14. Calculating the shortest distance between two ellipses by 0.5
sampling using uniformly distributed ellipse angles.

0.0
−1.0 −0.5 0.0 0.5 1.0 1.5
Lateral position (m)
Derive PDFs for
1 Generate logical
continuous 2 FIGURE 16. These are the generated parameter distributions for the
scenarios (LS)
parameters of LS application example. They are extracted for each chosen vehicle feature for
the first lanelet section displayed in Figure 10.
Use proposed sampling method for
Use scene prediction from urban
efficient generation of traffic
4 risk metric to
participant (TP) start values for 3 TABLE 6. The parameters of the LS used in the application example.
generate TP trajectories
concrete scenarios (CS)

Parameter Range Distribution

Evaluate each CS using the urban
Generate example risk metric for pass/fail and ego start position (x, y) (992, 970) fixed value
5 trajectory for ego vehicle calculate the achieved 6 ego target position (x, y) (1022, 982) fixed value
overall coverage
ego start velocity ( ms
) 7 fixed value
TP start velocity ( ms
) [0.5, 15] from Figure 16 (middle)
FIGURE 15. These six steps were taken in the execution of the application TP start lateral pos. (m) [-0.85, 1.75] from Figure 16 (bottom)
example. TP start heading (°) [-25, -8] from Figure 16 (top)

A. APPLICATION OF THE COVERAGE PROCESS the CPs are defined to be the vehicle’s lateral position, the
Two methods that advance the main parts of the coverage global heading and the velocity. The longidutinal starting
process (see Figure 2) have been presented. First, the creation position is set to be at the beginning of the lanelet section
of CS’ by applying the proposed sampling method using an depicted in Figure 10. Based on the available data in the
adapted k-means is tackled. Additionally, a respective metric INTERACTION dataset, the values for these three CPs for
for the evaluation of the created scenarios is presented. It is all available vehicle trajectories in the dataset in this specific
based on metrics already existing in the literature and has lanelet section are extracted. Using that, the respective PDFs,
been further developed for use in specific urban areas and for each CP individually, assuming that these parameters are
called the urban risk metric. To apply these two contributions independent, are derived. They can be observed in Figure 16.
to the coverage process, various steps have been taken, which Step 3: With the individual PDFs for each CP of the LS
can be seen in Figure 15 and are discussed next. in place, the proposed sampling method from Sec. III-C1
Step 1: In the first step, the LS need to be designed. For is used to efficiently choose values from the CP PDFs to
this application example, no LS from any scenario databases generate the actual CS’. Concretely, the Alg.1 is applied
are used, but self-constructed ones. The LS is located at the to the parameter distributions in Figure 16. In case of the
exact complex intersection pictured in Figure 10. One traffic vehicles’ global velocity distribution, the generated test case
participant is approaching from the top left (from a bird’s values are shown in Figure 17 as individual black dots.
view) and is either turning right or going straight at the road Furthermore, the boundaries of the resulting coverage areas
fork and turning right one road fork later. This is equivalent to (value under the parameter distribution) for each test case
the manoeuvre decision in Figure 10. The other traffic object value, are displayed. All parameters that define the LS and
is meant to be representing the ego vehicle, which tries to turn are used to construct the respective CS are shown in Tab. 6.
right at the intersection. It is represented as the green vehicle Step 4: Now, the scene prediction presented in Sec. III-D2
in Figure 10. is used to generate the actual trajectories of the traffic partici-
Step 2: The CPs for this LS define the vehicle configuration pant for each CS. Another option would have been to choose
of the traffic participant at the start of the scenario. Therefore the closest matching trajectory available in the dataset for the
16 VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see [Link]
 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3242127

Weissensteiner et al.: Operational Design Domain-driven Coverage for the Safety Argumentation of Automated Vehicles

KDE 1010
Test case value Traffic participant at t=2.4
0.15
from Alg. 1 Ego vehicle at t=2.4
Boundary for 1000 Lanelet network
coverage area

Global y-position (m)

Traffic participant trajectories
Density

0.10 Global vehicle

velocity density 990

0.05
980

0.00
0.0 2.5 5.0 7.5 10.0 12.5 15.0 970
Vehicle velocity ( ms )
FIGURE 17. The generated test case values using Alg. 1 for the extracted
parameter distribution of the global vehicle velocity in the first lanelet section 960
960 970 980 990 1000 1010 1020
displayed in Figure 10.
Global x-position (m)

FIGURE 19. The generated TP trajectories can be seen in blue. All TP’s start
at the beginning of the red lanelet depicted in Figure 10 and either turn right at
the first or second possibility. The generated ego vehicle trajectory can be
seen in green. For a specific pair, consisting of a TP trajectory and the
Cluster Ci trajectory of the ego vehicle, the time and object positions of the collision are
0.0012 Cluster Ci at iteration t depicted.
Boundary
Upper/lower boundary
0.0010
vehicle trajectory, these two object trajectories are used as
input to the urban risk metric. Based on the output of the
0.0008 risk metric, each CS is either categorised as passed or failed.
W (Ci)

This information is then used to compute the overall coverage

value cODD .
0.0006

V. DISCUSSION
0.0004
All of the generated TP trajectories can be observed in
Figure 19. For a specific pair, consisting of a TP trajectory
and the trajectory of the ego vehicle, the time and object
0.0002 positions of the collision are depicted. Approx. 30% of the
0.025 0.050 0.075 0.100 0.125 0.150 generated trajectories lead to an actual collision, with many
pCi near-misses.
For evaluating each CS, three different types of metrics are
FIGURE 18. The Alg. 1 is applied to the parameter distribution for the global
vehicle velocity (shown in the middle of Figure 16) to generate the required applied. First, the urban risk metric with the Rice distribution
test cases values. These values are shown in Figure 17. The iterative steps of is used for the distance vector between the objects’ positional
the algorithm for each generated clusters using the proposed adapted
k-means is shown here.
distribution. Second, the urban risk metric with the Beck-
mann distribution and, as a third method, the well-known
TTC calculation, using the improved calculation method of
traffic participant. However, since the scene prediction has [43]. For each metric, the TPR and the TNR using Eq. 17
been shown to produce meaningful results, it was also applied and Eq. 18 is calculated and shown in Tab. 7 for all metrics,
for trajectory generation in this application example. whereas the positive case is referred to an actual collision,
Step 5: For the ego vehicle, one trajectory was generated and vice versa. This can be done since this is an a-posteriori
that resembles a typical turn manoeuver at the intersection. analysis of trajectory pairs. It can be observed that the urban
Here, the underlying assumption is that the behaviour of risk metric (for both implemented cases) can observe nearly
the ego vehicles ADS is always the same for each CS. every collision. The risk assessment part of the urban risk
Note that steps 4 and 5 tackle the scenario execution (step metric is parametrised so that for every scenario where the
4+ of the high-level ODD coverage process, see Figure 2) calculated risk pcoll exceeds a value of 0.7 for at least one
in a simplified manner. Since the aim of this application future scene evolvement, the scenario is flagged as failed.
example is to show the proposed methods from Sec. III in an Based on how the actual risk is calculated, using an event-
overall example coverage process, this is sufficient but will based distance-dependent risk, even in actual collision cases,
be expanded in future publications. the calculated risk is slightly below 0.7. This, however, can
Step 6: As the last step, the CS’ are evaluated. Since further be tuned based on the threshold parameter in Eq. 29.
each CS consists of a pair of one TP trajectory and the ego However, in any case, each CS will be additionally collision-
VOLUME 4, 2016 17

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see [Link]
 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3242127

Weissensteiner et al.: Operational Design Domain-driven Coverage for the Safety Argumentation of Automated Vehicles

TABLE 7. Three types of risk metrics are used to evaluate the generated CS’

Offset from actual time to collision (s)

in the application example
0.7 Urban risk metric
TTC calculation
Type of risk metric True positive rate True negative rate 0.6
Urban risk metric (Rice) 0.96 0.011
Urban risk metric (Beckmann) 0.96 0.61 0.5
TTC calculation 1.0 0.0 0.4
0.3
0.2
checked with the actual trajectories.
It can be observed that the urban risk metric with the im- 0.1

plemented Beckmann distribution provides the best tradeoff 0.0

Type of Risk Metric
of sensitivity, with a high TPR, but also reasonable specificity
with flagging near-miss cases as being critical. In the case FIGURE 20. The distribution of the offset (in seconds) from the actual time to
of the urban risk metric with the Rice distribution, however, collision, for all generated scenarios with a collision, is shown for the TTC
calculation and the urban risk metric. The latter has a smaller offset and also
the TNR is severely reduced. This is a direct consequence less variance.
of the greater variance that this distribution assumption has,
which overstates the distance-depending risk in Eq. 28. Note
that such a metric does not need to achieve a TNR of 1 since
this would mean that no other than actual collision scenarios 2.5
Exact time until collision

Predicted time until collision (s)

are flagged as critical. However, a simple collision check TTC calculation
2.0
of both trajectories is enough to achieve the same. In an Urban risk metric
application with an actual ADS, it is much more essential to 1.5
unveil potentially critical evolvements of an already executed
scenario (since this is a-posteriori analysis) which can be 1.0
further evaluated. The urban risk metric provides precisely
that, as it generates realistic scene predictions, trained using 0.5
actual traffic data, next to the reliability for the specific
geolocation the metric is applied at. Using the urban risk 0.0
0.0 0.5 1.0 1.5 2.0 2.5
metric with the Beckmann distribution for the overall ODD Time (s)
coverage evaluation leads to an achieved coverage value of
FIGURE 21. For a specific pair, consisting of a TP trajectory and the trajectory
55.96%. of the ego vehicle, also depicted as an example in Figure 19, the time (in
For the TTC calculation, the situation is different. As seconds) which is left until a predicted collision is shown. The urban risk metric
is more reliable in its predictions than the TTC calculation across the whole
displayed in Tab. 7, the TTC metric categorised every sce- time horizon.
nario as critical. For the TTC metric to flag a scenario as
critical, the TTC value needs to equal zero for at least one
scene evolution. Consequently, this means that using the TTC displayed in blue. For this specific trajectory pair, the time
metric for evaluating CS’ in complex urban intersections (in seconds) which is left until a predicted collision is shown
turns out to be worthless, as no new information can be for the urban risk metric, the calculated TTC and the actual
gathered by applying this metric. A fact discussed as well time until the collision is added as well (see Figure 21). It
in [45]. shows that for the first timestep, the prediction of the TTC
Next, the validity of the urban risk metric (with the Beck- is worse, and also, until the actual collision, the urban risk
mann distribution) and the TTC calculation are evaluated. metric proves to be more reliable.
For that, all executed CS’ with collisions are analysed. For These results show that using a manoeuvre-based motion
these scenarios, the actual time to collision is known in this model as part of the scene prediction used in the urban risk
a-posteriori analysis. Then, for each metric, the time offset metric is suitable since the traffic participant has the right
is calculated from the actual time to collision in the first time of way in this example. Otherwise, the motion model needs
step of the scenario. The distribution of the offset (in seconds) to be enhanced towards interaction awareness to provide
from the actual time to collision is shown in Figure 20. reasonable estimates. Using the scene prediction from Sec.
The urban risk metric has a smaller offset and less variance III-D2 for the generation of TP trajectories also, in turn,
than the TTC calculation. This shows the urban risk metric’s betters the performance of the urban risk metric, as the
greater validity than the TTC calculation, as it already has identical scene prediction is used to calculate potential future
quite a low error in predicting the collision time, even at the evolvements for each scene. However, the TTC calculation
first scenario timestep, which has the largest prediction time also partly benefits from this since if the generated TP trajec-
until the actual collision. tories would include interaction-aware breaking or acceler-
Now, a specific CS with a collision is analysed in Figure ating, the performance of this metric would also be greatly
19, with the TP trajectory in orange and the ego vehicle deterred. Furthermore, the application example enabled to
18 VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see [Link]
 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3242127

Weissensteiner et al.: Operational Design Domain-driven Coverage for the Safety Argumentation of Automated Vehicles

analyse the urban risk metric based on the properties of method for the respective CS, as CS, which contributes
validity, sensitivity and specificity. Additionally, analysing strongly to the overall coverage, need to be executed
the derived PDFs from the INTERACTION dataset for the with a test method of high validity. Considering that
specific lanelet section (see Figure 10) shows a strong corre- most test methods have certain limitations (e.g., virtual
lation between the global heading angle of the vehicle and its testing regarding weather and the respective effects),
lateral position. This can be utilised by applying Alg. 2 to the this can be utilised when deciding the test method for
two-dimensional PDF of these two features to generate test the CS’. Furthermore, as the impact on the overall
cases. achievable coverage can be stated explicitly for each
CS, test method efforts can be guided more efficiently.
VI. CONCLUSION AND OUTLOOK This article does not cover this topic, but the proposed
This work proposes multiple extensions for a pre-existing coverage calculation provides a good starting point for
high-level coverage process for the safety validation of ADS. future research.
The contributions are across a particular part of the coverage • Since the proposed sampling method for concrete CS
process, detailed in the right part of Figure 2, and enable generation is variance-bounded based on the individual
more traceable and efficient safety argumentations for ADS’. coverage contribution, the number of necessary test
Concretely, these contributions are (the first number of each cases for each CP can be calculated inversely given the
contribution refers to the respective step of the right process respective boundary condition. That way, the bound-
in Figure 2): aries can be adjusted for each CP individually, and
the required amount of test cases (and their concrete
• Contribution 1.1: A n-dimensional sampling method
values) can be automatically generated, enabling more
for the scenario discretisation process is presented,
automation in the coverage process.
based on k-means clustering. It enables the definition
• Currently, the proposed generation of CS using the
of relevancy-dependent variance boundaries for the in-
adapted k-mean sampling is deterministic, as, for each
dividual test case values.
generated cluster, the centroid is chosen as the actual test
• Contribution 3.1: A risk metric for the use in complex
case value. However, every other rule of determining the
urban intersections is extended based on current liter-
actual values based on the generated clusters is possible.
ature. A significantly reduced specificity compared to
Suppose the generated CS should not be deterministic,
time-to-collision is shown by using an exact mathemati-
e.g., because the whole process is repeated multiple
cal formulation for the distribution of the distance vector
times during an ADS development cycle. The actual
between the positional distributions of traffic objects.
test case value could again be a drawn sample from the
Additionally, using actual traffic data from the respec-
respective cluster.
tive geolocation increased the validity of the metric.
• The urban risk metric can also be extended in mul-
• Contribution 4.1: The coverage process is advanced by
tiple ways. This concerns primarily scene predictions
the individual contributions and the application example
as this determines how valid the actual risk estimates
in Sec. IV.
are. Currently, the proposed scene prediction depends
• Contribution 4.2: Different levels of ODD coverage are
on the availability of enough traffic data to train the
introduced corresponding to the scenario ontology (LS,
individual manoeuvre prediction classifiers. Also, for
CS and CP).
route prediction, this data is needed. However, the scene
• Contribution 4.3: The ODD coverage is thoroughly
prediction could also be carried out with more generally
quantified across all scenario levels, assuming distribu-
applicable driver models.
tion functions’ availability for the scenario parameters.
• For this article, no concrete ADS was used in the ap-
In summation, a transparent process to evaluate the plication example, as the coverage calculation, the pro-
achieved coverage for the target ODD, independent of the posed sampling method and the urban risk metric can be
ADS implementation, is needed. Such a process is extended exemplarily shown without a concrete ADS implemen-
in multiple ways in this article, which in turn enables further tation. However, using a concrete ADS offers specific
extensions. Potential future work includes: other usages. Concretely, the sensor setup requirements
• The proposed coverage calculation can also be extended for a specific ADS with a pre-defined target ODD can
to include a metric to individually weigh each executed be recursively determined using the urban risk metric
CS, e.g., taking into account the experienced criticality. in combination with a value on the necessary coverage.
Also, other types of metrics on the scenario level are The urban risk metric can be applied to determine if the
possible, e.g., if the focus of the coverage evaluation ADS detected particular other traffic participants that
needs to be steered towards assessing the perception could lead to dangerous situations eventually.
layer.
• The coverage evaluation assigns every CS a certain ACKNOWLEDGMENT
value that contributes to the overall coverage. That in- This research received financial support within the COMET
formation can be useful when deciding the specific test K2 Competence Centers for Excellent Technologies from
VOLUME 4, 2016 19

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see [Link]
 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3242127

Weissensteiner et al.: Operational Design Domain-driven Coverage for the Safety Argumentation of Automated Vehicles

the Austrian Federal Ministry for Climate Action (BMK), [Online]. Available: [Link]
the Austrian Federal Ministry for Digital and Economic ASAM-Test-Specification-Study-Group-Report-2022_V1-[Link]
[16] United Nations Economic Commission for Europe, “New Assessment/Test
Affairs (BMDW), the Province of Styria (Dept. 12), and the Method for Automated Driving (NATM),” Geneva, Switzerland, Submit-
Styrian Business Promotion Agency (SFG). The Austrian ted by the Working Party on Automated/Autonomous and Connected
Research Promotion Agency (FFG) has been authorised for Vehicles ECE/TRANS/WP.29/2021/61, 2021. [Online]. Available: https://
[Link]/sites/default/files/2021-04/[Link]
the programme management. They would furthermore like to [17] B. Ciuffo, K. Mattas, and M. C. Galassi, “Safety assurance of automated
express their thanks to their supporting industrial and scien- driving systems - Raising the level of ambition,” European Commission -
tific project partners, namely Infineon Technologies Austria Joint Research Center, Nov. 2020.
[18] R. Donà, B. Ciuffo, A. Tsakalidis, L. Di Cesare, C. Sollima, M. Sangiorgi,
AG, Ing. h. c. F. Porsche AG, Volkswagen AG, aiMotive
and M. C. Galassi, “Recent Advancements in Automated Vehicle
Kft, Joanneum Research, University of Klagenfurt,Technical Certification: How the Experience from the Nuclear Sector Contributed to
University Graz and University of Graz. Making Them a Reality,” Energies, vol. 15, no. 20, p. 7704, Jan. 2022,
number: 20 Publisher: Multidisciplinary Digital Publishing Institute.
[Online]. Available: [Link]
REFERENCES [19] “Building Safe Autonomous Systems - Handbook for the verification
[1] World Health Organization, Global status report on road and validation (V&V) of autonomous systems,” Applied Intuition —
safety 2018: Summary, Geneva, Switzerland, 2018. [On- Accelerating Safety, Handbook, Jul. 2022.
line]. Available: [Link] [20] FIVE AI, “Certification of Highly Automated Vehicles for Use on UK
[Link]?ua=1 Roads - Creating An Industry-Wide Framework for Safety,” Tech. Rep.,
[2] A. Poddey, T. Brade, J. E. Stellet, and W. Branz, “On the validation of com- 2019.
plex systems operating in open contexts,” Jan. 2019, arXiv:1902.10517 [21] C. Miethaner and J.-E. Stavesand, “Virtual homologation of an ALKS
[cs]. [Online]. Available: [Link] according to UNECE R157,” TÜV SÜD & dSPACE GmbH, Westendstr.
[3] SAE International, SAE J 3016 - Taxonomy and Definitions for Terms 199, Munich, Germany, White paper, 2022.
Related to On-Road Motor Vehicle Automated Driving Systems. SAE In- [22] H. Hungar, “A concept of scenario space exploration with criticality
ternational, On-Road Automated Driving (ORAD) committee, Apr. 2021. coverage guarantees, extended abstract,” in Leveraging Applications
[4] “PAS 1883:2020,” Aug. 2020. [Online]. Available: [Link] of Formal Methods, Verification and Validation: Applications, 9th
com/en-GB/CAV/pas-1883/ International Symposium on Leveraging Applications of Formal Methods,
[5] SaFAD, “Safety First for Automated Driving,” 2019. ISoLA 2020, Rhodes, Greece, October 20-30, 2020, Proceedings, Part
[Online]. Available: [Link] III, ser. Lecture Notes in Computer Science. Springer International
framework-for-safe-automated-driving-systems/ Publishing, 2020, vol. 12478, ch. chapter 19, pp. 293–306. [Online].
[6] M. S. Blumenthal, L. Fraade-Blanar, R. Best, and J. L. Irwin, “Safe Available: [Link]
Enough: Approaches to Assessing Acceptable Safety for Automated [23] B. Gangopadhyay, S. Khastgir, S. Dey, P. Dasgupta, G. Montana, and
Vehicles,” RAND Corporation, Tech. Rep., Oct. 2020. [Online]. P. Jennings, “Identification of test cases for automated driving systems
Available: [Link] using bayesian optimization.” IEEE, Oct 2019, pp. 1961–1967. [Online].
[7] F. Batsch, S. Kanarachos, M. Cheah, R. Ponticelli, and M. Blundell, “A Available: [Link]
taxonomy of validation strategies to ensure the safe operation of highly [24] S. Khastgir, S. Brewerton, J. Thomas, and P. Jennings, “Systems
automated vehicles,” Journal of Intelligent Transportation Systems, pp. approach to creating test scenarios for automated driving systems,”
1–20, Mar. 2020. [Online]. Available: [Link] Reliability Engineering & System Safety, vol. 215, p. 107610, Nov
full/10.1080/15472450.2020.1738231 2021. [Online]. Available: [Link]
[8] A. Corso, R. Moss, M. Koren, R. Lee, and M. Kochenderfer, “A survey S0951832021001551
of algorithms for black-box safety validation of cyber-physical systems,”
[25] P. Weissensteiner, G. Stettinger, S. Genser, and D. Watzenig, “Operational
Journal of Artificial Intelligence Research, vol. 72, Oct 2021. [Online].
Design Domain Coverage for the Safety Validation of Automated Driving
Available: [Link]
Systems,” in Driving Simulation Proceedings. Strasbourg, France:
[9] S. Ulbrich, T. Menzel, A. Reschka, F. Schuldt, and M. Maurer,
Driving Simulation Association, 2022.
“Defining and Substantiating the Terms Scene, Situation, and Scenario
[26] X. Zhang, J. Tao, K. Tan, M. Törngren, J. M. G. Sánchez, M. R.
for Automated Driving,” in 2015 IEEE 18th International Conference
Ramli, X. Tao, M. Gyllenhammar, F. Wotawa, N. Mohan, M. Nica, and
on Intelligent Transportation Systems. Gran Canaria, Spain: IEEE,
H. Felbinger, “Finding Critical Scenarios for Automated Driving Systems:
Sep. 2015, pp. 982–988. [Online]. Available: [Link]
A Systematic Literature Review,” arXiv:2110.08664 [cs, eess], Oct. 2021,
document/7313256/
arXiv: 2110.08664. [Online]. Available: [Link]
[10] T. Menzel, G. Bagschik, L. Isensee, A. Schomburg, and M. Maurer,
“From Functional to Logical Scenarios: Detailing a Keyword-Based [27] T. Ken Mori, X. Liang, L. Elster, and S. Peters, “The inadequacy
Scenario Description for Execution in a Simulation Environment,” of discrete scenarios in assessing deep neural networks,” IEEE
arXiv:1905.03989 [cs], May 2019, arXiv: 1905.03989. [Online]. Access, vol. 10, pp. 118 236–118 242, 2022. [Online]. Available:
Available: [Link] [Link]
[11] S. Riedmaier, T. Ponn, D. Ludwig, B. Schick, and F. Diermeyer, [28] “Design of Experiments (DoE),” Technical Statistics, 2010.
“Survey on scenario-based safety assessment of automated vehicles,” [Online]. Available: [Link]
IEEE Access, vol. 8, pp. 87 456–87 477, 2020. [Online]. Available: group/purchasing_and_logistics/information_for_business_partners/
[Link] downloads/quality_docs/general_regulations/bosch_publications/
[12] T. Brade, B. Kramer, and C. Neurohr, “Paradigms in scenario-based booklet-no11-design-of-experiments-doe_EN.pdf
testing for automated driving.” ACM, Feb 2021, pp. 108–114. [Online]. [29] “Scenario-based verification and validation of self-driving vehicles: rele-
Available: [Link] vant safety metrics,” Siemens Digital Industries Software & IVEX NV,
[13] C. Neurohr, L. Westhofen, T. Henning, T. de Graaff, E. Möhlmann, and White paper, 2022.
E. Böde, “Fundamental Considerations around Scenario-Based Testing [30] “Safety pool - powered by deepen ai and wmg university of warwick.”
for Automated Driving,” arXiv:2005.04045 [cs], May 2020, arXiv: [Online]. Available: [Link]
2005.04045. [Online]. Available: [Link] [31] N. Weber, D. Frerichs, and U. Eberle, A simulation-based, statistical
[14] JAMA and SAKURA, “Automated Driving Safety Evalua- approach for the derivation of concrete scenarios for the release of
tion Framework Ver. 1.0 -Guidelines for Safety Evaluation highly automated driving functions, ser. AmE 2020 - Automotive meets
of Automated Driving Technology,” Tech. Rep. [Online]. Electronics; 11th GMM-Symposium, Dortmund, Jan. 2020, no. 11.
Available: [Link] [32] Y. Akagi, R. Kato, S. Kitajima, J. Antona-Makoshi, and N. Uchida, “A
Safety_Evaluation_Framework_Ver1.[Link] risk-index based sampling method to generate scenarios for the evaluation
[15] ASAM e.V., “ASAM Test Specification Study Group Report 2022,” of automated driving vehicle safety.” IEEE, Oct 2019, pp. 667–672.
Hoehenkirchen, Germany, Tech. Rep. Version 1.0.0, Feb. 2022. [Online]. Available: [Link]

20 VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see [Link]
 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3242127

Weissensteiner et al.: Operational Design Domain-driven Coverage for the Safety Argumentation of Automated Vehicles

[33] A. T. Ihler, E. B. Sudderth, W. T. Freeman, and A. S. Willsky, “Efficient [53] K. Kreutz and J. Eggert, “Analysis of the generalized intelligent driver
multiscale sampling from products of gaussian mixtures,” in Proceedings model (gidm) for uncontrolled intersections.” IEEE, Sep 2021, pp. 3223–
of the 16th International Conference on Neural Information Processing 3230. [Online]. Available: [Link]
Systems, ser. NIPS’03. Cambridge, MA, USA: MIT Press, 2003, pp. [54] K. Mattas, G. Albano, R. Donà, M. C. Galassi, R. Suarez-Bertoa, S. Vass,
1–8. and B. Ciuffo, “Driver models for the definition of safety requirements of
[34] R. Xu and D. Wunsch, “Survey of clustering algorithms,” IEEE automated vehicles in international regulations. application to motorway
Transactions on Neural Networks, vol. 16, no. 3, pp. 645–678, May 2005. driving conditions,” Accident Analysis & Prevention, vol. 174, p. 106743,
[Online]. Available: [Link] Sep 2022. [Online]. Available: [Link]
[35] J. B. MacQueen, “Some methods for classification and analysis of multi- S0001457522001798
variate observations,” pp. 281–297, 1967. [55] M. T. Wolf and J. W. Burdick, “Artificial potential functions for highway
[36] S. Lloyd, “Least squares quantization in pcm,” IEEE Trans. Inf. Theory, driving with collision avoidance.” IEEE, May 2008, pp. 3731–3736.
vol. 28, pp. 129–137, Mar 1982. [Online]. Available: [Link]
[37] D. Arthur and S. Vassilvitskii, “k-means++: the advantages of careful [56] J. Quehl, H. Hu, S. Wirges, and M. Lauer, “An Approach to Vehicle
seeding,” in Proceedings of the eighteenth annual ACM-SIAM symposium Trajectory Prediction Using Automatically Generated Traffic Maps,” 2018
on Discrete algorithms, ser. SODA ’07. USA: Society for Industrial and IEEE Intelligent Vehicles Symposium (IV), 2018.
Applied Mathematics, Jan. 2007, pp. 1027–1035. [57] M. Schreier, V. Willert, and J. Adamy, “An integrated approach
[38] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, to maneuver-based trajectory prediction and criticality assessment
O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, in arbitrary road environments,” IEEE Transactions on Intelligent
J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and Transportation Systems, vol. 17, no. 10, pp. 2751–2766, Oct 2016.
E. Duchesnay, “Scikit-learn: Machine Learning in Python,” Journal of [Online]. Available: [Link]
Machine Learning Research, vol. 12, no. 85, pp. 2825–2830, 2011. [58] S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y. Chai,
[Online]. Available: [Link] B. Sapp, C. Qi, Y. Zhou, Z. Yang, A. Chouard, P. Sun, J. Ngiam,
[39] M. D. McKay, W. J. Conover, and R. J. Beckman, “A comparison of three V. Vasudevan, A. McCauley, J. Shlens, and D. Anguelov, “Large scale
methods for selecting values of input variables in the analysis of output interactive motion forecasting for autonomous driving : The waymo open
from a computer code,” Technometrics, pp. 239–245, 1979. motion dataset.” IEEE, Oct 2021, pp. 9690–9699. [Online]. Available:
[40] C. Neurohr, L. Westhofen, M. Butz, M. Bollmann, U. Eberle, and R. Gal- [Link]
bas, “Criticality Analysis for the Verification and Validation of Automated [59] D. Petrich, T. Dang, D. Kasper, G. Breuel, and C. Stiller, “Map-based
Vehicles,” IEEE Access, vol. 9, Jan. 2021. long term motion prediction for vehicles in traffic environments.” IEEE,
[41] ISO 26262-1:2018 - Road vehicles - Functional safety. Oct 2013, pp. 2166–2172. [Online]. Available: [Link]
Geneva, Switzerland: International Organization for Standardization, document/6728549/
2018. [Online]. Available: [Link] [60] S. Biswas, M. Bianchini, R. N. Shaw, and A. Ghosh, “Prediction
4e26ddadc54a4198bed652afe29669fa/[Link] of traffic movement for autonomous vehicles,” vol. 960, pp.
[42] J. C. Hayward, “Near-miss determination through use of a scale of danger,” 153–168, 2021. [Online]. Available: [Link]
Pennsylvania Transportation and Traffic Safety Center, 1972. 978-981-16-0598-7_12
[43] F. Jiménez, J. E. Naranjo, and F. García, “An improved method to calculate [61] V. P. Thazha, D. Filliat, and J. Ibañez-Guzmán, “Applying map-masks to
the time-to-collision of two vehicles,” International Journal of Intelligent Trajectory Prediction for Interacting Traffic-Agents,” IEEE International
Transportation Systems Research, vol. 11, no. 1, pp. 34–42, Jan 2013. Conference on Intelligent Transportation Systems (ITSC’19), 2019.
[Online]. Available: [Link] [62] W. Zhan, L. Sun, D. Wang, H. Shi, A. Clausse, M. Naumann,
[44] B. L. Allen, B. T. Shin, and P. J. Cooper, “Analysis of Traffic Conflicts J. Kummerle, H. Konigshof, C. Stiller, A. de La Fortelle, and
and Collisions,” Transportation Research Record, vol. 667, pp. pp 67–74, M. Tomizuka, “INTERACTION Dataset: An INTERnational, Adversarial
1978. and Cooperative moTION Dataset in Interactive Driving Scenarios with
[45] L. Westhofen, C. Neurohr, T. Koopmann, M. Butz, B. Schütt, F. Utesch, Semantic Maps,” Sep. 2019, arXiv:1910.03088 [cs, eess]. [Online].
B. Neurohr, C. Gutenkunst, and E. Böde, “Criticality Metrics for Available: [Link]
Automated Driving: A Review and Suitability Analysis of the State of [63] F. Poggenhans, J.-H. Pauls, J. Janosovits, S. Orf, M. Naumann,
the Art,” Archives of Computational Methods in Engineering, Aug. 2022. F. Kuhnt, and M. Mayr, “Lanelet2: A high-definition map framework
[Online]. Available: [Link] for the future of automated driving,” in Proc. IEEE Intell. Trans.
[46] M.-Y. Yu, R. Vasudevan, and M. Johnson-Roberson, “Occlusion-Aware Syst. Conf., Hawaii, USA, November 2018. [Online]. Available: http:
Risk Assessment for Autonomous Driving in Urban Environments,” //[Link]/z/publ/download/2018/[Link]
arXiv:1809.04629 [cs], Jul. 2019, arXiv: 1809.04629. [Online]. Available: [64] L. Breiman, “Random Forests,” Machine Learning, vol. 45, no. 1,
[Link] pp. 5–32, Oct. 2001. [Online]. Available: [Link]
[47] D. Wang, W. Fu, Q. Song, and J. Zhou, “Potential risk assessment 1010933404324
for safe driving of autonomous vehicles under occluded vision,” [65] Y. Bar-Shalom, X.-R. Li, and T. Kirubarajan. John Wiley & Sons, Inc.,
Scientific Reports, vol. 12, no. 1, Mar 2022. [Online]. Available: 2001.
[Link] [66] C. Pek, V. Rusinov, S. Manzinger, M. C. Uste, and M. Althoff,
[48] P. Weissensteiner, G. Stettinger, K. Tieber, and K. Rehrl, “Virtual Risk “Commonroad drivability checker: Simplifying the development and
Assessment for the Deployment of Autonomous Shuttles,” Transportation validation of motion planning algorithms.” IEEE, Oct 2020, pp. 1013–
Research Record, p. 10, Jul. 2021. 1020. [Online]. Available: [Link]
[49] P. Schneider, M. Butz, C. Heinzemann, J. Oehlerking, and M. Woehrle, [67] J. Eggert, “Risk estimation for driving support and behavior planning in
“Towards threat metric evaluation in complex urban scenarios.” IEEE, intelligent vehicles,” at - Automatisierungstechnik, vol. 66, pp. 119–131,
Sep 2021, pp. 1192–1198. [Online]. Available: [Link] 2018. [Online]. Available: [Link]
document/9564790/ 1515/auto-2017-0132/html
[50] J. Bock, R. Krajewski, T. Moers, S. Runde, L. Vater, and L. Eckstein, [68] J. Eggert and T. Puphal, “Continuous Risk Measures for ADAS and AD,”
“The inD Dataset: A Drone Dataset of Naturalistic Road User Trajectories FAST-zero Symposium, 2017.
at German Intersections,” arXiv:1911.07602 [cs, eess], Nov. 2019, arXiv: [69] J. Eggert, “Predictive risk estimation for intelligent adas functions.”
1911.07602. [Online]. Available: [Link] IEEE, Oct 2014, pp. 711–718. [Online]. Available: [Link]
[51] S. Wagner, K. Groh, T. Kuhbeck, M. Dorfel, and A. Knoll, “Using time- org/document/6957773/
to-react based on naturalistic traffic object behavior for scenario-based [70] E. K. Antonsson, “A General Measure of Collision Hazard in Traffic,”
risk assessment of automated driving.” IEEE, Jun 2018, pp. 1521–1528. May 2022. [Online]. Available: [Link]
[Online]. Available: [Link] [71] S. O. Rice, “Mathematical analysis of random noise,” Bell System Tech. J,
[52] S. Lefèvre, D. Vasquez, and C. Laugier, “A survey on motion prediction vol. 24, pp. 46–156, 1945.
and risk assessment for intelligent vehicles,” ROBOMECH Journal, [72] P. Beckmann, “Statistical distribution of the amplitude and phase of a
vol. 1, no. 1, Dec 2014. [Online]. Available: [Link] multiply scattered filed,” Journal of Research of the National Bureau of
[Link]/articles/10.1186/s40648-014-0001-z Standards, vol. 66D, pp. 231–240, 1962.

VOLUME 4, 2016 21

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see [Link]
 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2023.3242127

Weissensteiner et al.: Operational Design Domain-driven Coverage for the Safety Argumentation of Automated Vehicles

[73] R Core Team, R: A Language and Environment for Statistical Computing, DANIEL WATZENIG (Senior Member, IEEE) re-
Vienna, Austria, 2022. [Online]. Available: [Link] ceived the [Link]. degree in electrical engineering
[74] D. Wollschlaeger, shotGroups: Analyze Shot Group Data, 2022, r and the Ph.D. degree in technical science from
package version 0.8.2. [Online]. Available: [Link] the Graz University of Technology, Graz, Austria,
packages/shotGroups/[Link] in 2002 and 2006, respectively. In 2009, he was
[75] J. Bernhard, P. Hart, A. Sahu, C. Schöller, and M. G. Cancimance, “Risk- the recipient of the Venia Docendi (Habilitation)
Based Safety Envelopes for Autonomous Vehicles Under Perception for Electrical Measurement and Signal Process-
Uncertainty,” arXiv:2107.09918 [cs, eess], Jul. 2021, arXiv: 2107.09918.
ing.
[Online]. Available: [Link]
Since 2006, he has been the Divisional Director
[76] J. Bernhard and A. Knoll, “Risk-constrained interactive safety under
behavior uncertainty for autonomous driving,” pp. 63–70, Jul 2021. and Scientific Head of the Automotive Electronics
[Online]. Available: [Link] and Software Department, Virtual Vehicle Research GmbH, Graz. In 2017,
he was appointed as a Full Professor of autonomous driving with the Institute
of Automation and Control, Graz University of Technology. He is the
Founder and a Team Leader of the Autonomous Racing Graz Team, one
of currently six teams of the global race series Roborace.
He is the Author or Co-Author of more than 200 peer-reviewed papers,
PATRICK WEISSENSTEINER received the book chapters, patents, and articles. His research interests include sense and
[Link]. and [Link]. degree in mechanical engineering control of automated vehicles, signal processing, multi-sensor data fusion,
and business economics from the Graz University uncertainty estimation and quantification, and robust optimization.
of Technology, Graz, Austria, in 2015 and 2017,
respectively.
He is currently working as a Senior Researcher
in the Automotive Electronics and Software De-
partment at Virtual Vehicle Research GmbH in
Graz, Austria. His current research interests con-
cern the safety validation of automated vehicles,
with a particular focus towards coverage methods for the respective opera-
tional design domain of such vehicles.

GEORG STETTINGER received the [Link]. and

[Link]. degrees in electrical engineering from the
Graz University of Technology (TUG), Graz, Aus-
tria, in 2009 and 2011, respectively and the Ph.D.
degree in information technology from the Univer-
sity of Klagenfurt (AAU), Klagenfurt, Austria, in
2015. From 2015 to 2018, he was a Senior Re-
searcher within the Co-Simulation and Software
group with Virtual Vehicle Research GmbH, Graz,
Austria. From 2018 to 2022, he led the Control
Systems team at Virtual Vehicle Research GmbH, Graz, Austria. He is
currently a Senior Project Manager with the R&D funding department at
Infineon Technologies AG, Munich, Germany. His current research interests
include ODD-based testing and validation, particularly certification and
homologation of automated vehicles. He is an active member of the technical
committee of the driving simulator association with a special focus on virtual
verification and validation of ADAS/AD systems.

SIDDARTHA KHASTGIR , (Member, IEEE) re-

ceived the [Link]. and [Link]. dual degree in me-
chanical engineering from the Indian Institute of
Technology (IIT) Kharagpur, India, in 2011, and
the Ph.D. degree from The University of Warwick,
U.K., in 2019.
He is currently the Head of Verification and Val-
idation, Intelligent Vehicles with WMG, The Uni-
versity of Warwick. His research interests include
trust in automation, system safety, and verification
and validation of autonomous vehicles. He is also a Chartered Engineer and
a member of the Institution of Mechanical Engineers (IMechE), U.K. He has
been named in the Forbes 30 under 30 Europe 2018—industry list and was
awarded the prestigious UKRI Future Leaders Fellowship, in 2019.

22 VOLUME 4, 2016

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see [Link]