extfieldsofstudy:
Computer Science
Engineering

text:
A Review of Testing Object-Based Environment Perception for Safe Automated Driving

Safety assurance of automated driving systems must consider uncertain environment perception. This paper reviews literature addressing how perception testing is realized as part of safety assurance. We focus on testing for verification and validation purposes at the interface between perception and planning, and structure our analysis along the three axes 1) test criteria and metrics, 2) test scenarios, and 3) reference data. Furthermore, the analyzed literature includes related safety standards, safety-independent perception algorithm benchmarking, and sensor modeling. We find that the realization of safety-aware perception testing remains an open issue since challenges concerning the three testing axes and their interdependencies currently do not appear to be sufficiently solved.

I. INTRODUCTION
To date, the safety assurance of automated vehicles of SAE Levels 4 and 5 [1] (AVs) has been an open issue, which is partially because the environment perception of AVs is subject to uncertainties. AVs perceive their environment through a sensor setup that typically consists of at least camera, radar, and lidar sensors. Perception algorithms, which typically use machine learning, then process and fuse the raw sensor data to a world model that is handed over to the subsequent functional modules. Typical world models contain a list of objects, where each object corresponds to a road user or other movable entity surrounding the AV [2].
Uncertainties in this world model can be caused by environmental influences, sensing hardware, perception software, and other factors. These uncertainties can be divided into state, existence, and classification uncertainties [3]. Respective examples are that a road user's velocity is estimated too slow, the road user is not detected at all, or a pedestrian is classified as a cyclist.
For AV subsystems that are mostly defined by software, the real-world testing effort can be substantially reduced through simulated tests. However, as the sensor hardware interacts with the real environment in complex ways that are hard to replicate in simulations, real-world tests remain pivotal.
In the current literature, there does not yet exist a common methodology for the execution of real-world perception tests. Under which criteria is the environment perceived "well *The research leading to these results is funded by the Federal Ministry for Economic Affairs and Energy within the project "VVM -Verification and Validation Methods for Automated Vehicles Level 4 and 5". The authors would like to thank the consortium for the successful cooperation.  [4], applied to perception testing: A real-world scenario is simultaneously observed by a reference perception system and by a perception subsystem under test. Their results are compared using metrics, which allows to evaluate test criteria.

Scenario
enough"? In which scenarios should the environment perception be tested? How can one obtain the true environment state during these scenarios? Summarizing these questions, the primary research question of this review is: How can perception tests be realized to assess that an uncertain perception subsystem allows safe AV behavior?
To structure the diverse literature contributions to this general question, we apply the axes of the testing taxonomy by Stellet et al. [4] to perception testing (Fig. 1). The taxonomy states that testing is an evaluation of "A statement on the system-under-test (test criteria) that is expressed quantitatively (metric) under a set of specified conditions (test scenario) with the use of knowledge of an ideal result (reference)."

A. Motivation for a Review
The current literature from the areas of perception development, automotive safety assurance, and safety in artificial intelligence contains valuable approaches to all mentioned testing axes. For a successful safety proof, approaches to individual axes have to seamlessly fit into an overall methodology. Prior to writing, we found it difficult to identify how the available approaches complement each other, which is why we are aiming at providing a structured overview of the state of the art. The goal of this review is to enable the development of holistic concepts for testing the safety-relevant uncertainties of environment perception.  [5], focused on perception. Right: terms that are used in this paper, mapped to the layers of [5].

B. Structure and Contributions
Related Reviews, Surveys, and Overviews (Sec. II) reviews previous secondary literature. Methods  • Approaches to the research question, structured by the three individual testing axes -Test Criteria and Metrics (Sec. VI) -Test Scenarios (Sec. VII) -Reference Data (Sec. VIII) Finally, the Discussion of Review Results (Sec. IX) highlights the largest issues for answering the research question, points out intersection topics between the testing axes, and puts the topics of this review in a larger context.

C. Term Definitions
The following term definitions are used throughout the rest of this paper. Fig. 2 illustrates the relations between different terms by using the the layered decomposition of an automated driving (AD) system by [5].
1) Perception Subsystem: The AV's subsystem for environment perception is defined by at least sensor mounting positions (part of Layer 0), the sensors themselves that generate raw data (Layer 1), and perception software (Layer 2) (Fig. 2).
2) System Under Test: Perception subsystem of a subject vehicle/ego vehicle. Abbreviation: SUT.
3) Environment Perception: The process of environment perception consists of the mentioned perception subsystem, which is being affected by environmental influences that are outside the control of it (rest of Layer 0; Fig. 2). These environmental influences determine if and how the individual parts of the ground truth information are accessible [5].
4) Perception Algorithm: Software that computes an objectbased world model from the raw sensor data (adapted from [8]). Generally includes both traditional code and components that are trained on data. Corresponds to Layer 2 (Fig. 2). 5) World Model: Representation of the ego vehicle's environment and own state that is computed by a perception algorithm (adapted from [8]). Equal to a subjective scene at the interface between Layers 2 and 3 (Fig. 2). The most essential world model component in this paper is the current state of dynamic object tracks. 6) Safety: "absence of accidents, where an accident is an event involving an unplanned and unacceptable loss" [9, p. 11]. Unless stated otherwise, safety in this paper means safety of the intended functionality (SOTIF) of the AV (see [7]) rather than functional safety according to ISO 26262 [10]. 7) Reliability: "probability that something satisfies its specified behavioral requirements over time and under given conditions -this is, it does not fail" [9, p. 10]. 8) Uncertainty: "doubt about the validity of the result of a measurement", or a quantitative measure of this doubt [11]. Uncertainty in object-based world models can be state, existence, or classification uncertainty [3], as mentioned before. 9) Scenario: "A scenario describes the temporal development between several scenes in a sequence of scenes. [...]" [12]. For open-loop perception testing, we assume that the temporal development of all ground truth scenes is given, including environmental influences and all road user trajectories. What is to be determined in a test scenario is how the SUT subjectively observes the objective scenes in its world model (more in Sec. VII-A).
10) Metrics: Functions that compute properties of interest about the SUT. We use the microscopic/macroscopic terminology for metrics of [13], where microscopic refers to a single scene for a single ego vehicle within a single scenario, and macroscopic refers to the average over an entire fleet of identical vehicles that encounters various scenarios and scenes.
II. RELATED REVIEWS, SURVEYS, AND OVERVIEWS Previous reviews partially cover the targeted topic of this paper. For example, the surveys by Stellet et al. [14], [4], Riedmaier et al. [15], Nalic et al. [16], and the PEGASUS method [17] deal with testing and safety assurance of AVs Sensor raw data, object-independent environmental features, occupancy grid map, parametric drivable space in general, but without specifically analyzing the interface between perception and planning. A systematic literature review of the Swedish SMILE project targets verification and validation for machine learning based systems in AD [18], however, without focusing specifically on environment perception.
Literature reviews about AV sensing and perception like [19], [20], and [21] provide the technological state of the art, but do not explicitly focus on testing or safety assurance.
Furthermore, there are publications that provide a valuable overview over current safety challenges for AV environment perception, but which are not explicitly literature reviews [22], [23]. For example, [22] identifies safety concerns and mitigation approaches for safety-critical perception tasks that rely on deep learning.
In summary, we aim to fill the existing gap in literature reviews by analyzing literature related to the main research question in depth throughout the rest of this paper.

III. METHODS
The methods of this literature review are loosely inspired by the guidelines for systematic literature reviews [24] and snowballing search [25], and respective example applications of both in [26] and [18]. The suggested guidelines are purposely not followed exactly, because they were designed for reviewing quantitative and empirical research rather than qualitative or position papers, which are, however, common in the current automotive safety assurance domain. Table I summarizes the focus of this literature review. Literature that matches more focused aspects is more likely to be included and discussed in detail, whereas literature about related aspects outside the main focus might still be referenced to provide context where the actually targeted literature was sparse.

B. Literature Search Process
Three search processes were performed to accumulate the references of this paper. First, undocumented searches provided an initial knowledge base. Second, a keyword-based search complemented the starting set for a final iterative snowballing search. While the undocumented search provided most references about standards and established testing approaches, the inclusion criteria of the documented search targeted specifically the primary research question and its test axes.
1) Undocumented Search: All references that are not part of Table II were obtained through undocumented search processes before and during the writing process.
2) Keyword Search: The following strings were searched in Google Scholar, allowing results in the time frame between 01/01/2010 and 20/08/2020: For both search strings, the top 250 results were analyzed. 20 publications were identified as candidates for the start set of the subsequent snowballing search, out of which 11 were actually included (see Table II).

3) Snowballing Search:
The results from the other search processes formed a start set, based on which we performed a forward and backward snowballing search [25] in fall 2020 using Google Scholar to fill possible gaps. Forward snowballing means searching through the citations that a publication has received, while backward snowballing means searching through the references it itself has included. Both directions were searched within the same iteration, respectively (Table II).
Included snowballing search results that are not directly related to the primary research question or to at least one of the three testing axes were excluded from the further search (marked with * in Table II) to keep the search process efficient and the overall effort feasible. Eventually, we removed the third iteration, which consisted of two sources, and some sources from iterations 1 and 2 due to irrelevance for the final version.

IV. PERCEPTION TESTING IN SAFETY STANDARDS
Safety standards, guidelines, and frameworks can prescribe certain aspects around perception testing in safety assurance. Thus, we first review the explicit role of perception testing in standardizing literature before moving on to more concrete realizations of perception tests in the subsequent sections.

A. ISO 26262
ISO 26262 [10] addresses the assurance of functional safety of electric and electronic systems in the automotive domain. It has been the most essential safety standard for advanced driver assistance systems (ADAS), but has not been developed to assure the safety of AD systems that rely on environment perception or use machine learning [37]. Consequently, the literature investigates how ISO 26262 could be adapted or extended to also cover machine-learning-based components such as modern environment perception systems, see e.g. [18].
One major issue that prevents the concepts of ISO 26262 from being applied to AV environment perception is that the perception subsystem is said to be not fully specifiable (except for defining a training set), while ISO 26262 implicitly assumes that all functionality must be specified [37], [62]. This means that the environment perception is also not fully verifiable and perception testing is in general incomplete [37].
Nevertheless, ISO 26262's workflow for Safety Elements out of Context (SEooC) [10, Part 10] might be a hint on how the development of a perception subsystem (safety element) could at least be generally organized while the rest of the driving function (context) is not known.

B. ISO/PAS 21448
A vehicle that is perfectly functionally safe according to ISO 26262 can still cause accidents if its behavioral logic is wrong [33]. Addressing this issue, ISO/PAS 21448 [7] introduces the concept of the so-called Safety of the Intended Functionality (SOTIF). The standard describes how SOTIF should be assured for ADAS up to SAE Level 2 and acknowledges that it is likely not sufficient for higher levels of automation.
According to [7,Clause 7], so-called triggering events "that can trigger potentially hazardous behavior shall be identified", and their impact on the SOTIF shall be assessed. Triggering events related to sensor disturbances can be caused for example by poor weather conditions or poor-quality reflections. They can cause errors in the world model if corresponding functional insufficiencies in the perception subsystem exist [22]. Once triggering events for the perception subsystem have been identified (see e.g. [40], [7, Annex F]) as part of the so-called sensor verification strategy [7, Sub-Clause 10.2], they can also serve as test scenarios for it. Furthermore, [7, Annex D] lists non-comprehensive example aspects to consider in verification testing of the perception subsystem, but leaves open the concrete realization of those tests.

C. NHTSA Vision and Framework
In "Automated Driving Systems 2.0: A Vision for Safety" [77], the National Highway Traffic Safety Administration (NHTSA) and the U.S. Department of Transportation (DOT) follow a "nonregulatory approach to automated vehicle technology safety". AV manufacturers are allowed and required to follow and document their own individual safety assurance processes, which intends to support innovation rather than hindering it. Twelve so-called safety design elements are mentioned, where the one most relevant to this paper is Object and Event Detection and Response (OEDR) -the capability to "detect any circumstance that is relevant to the immediate driving task" and respond to it.
A more recent and more detailed publication of NHTSA defines a framework for testing AD systems [78]. This framework also addresses OEDR and further defines a taxonomy that can help describing an operational design domain (ODD) for an AV. Such a clear ODD description is necessary to express the requirements of how to also cope with unusual situations such as emergency vehicles.
According to [78, p. 65], testing the perception outputs can significantly facilitate the assessment of OEDR capabilities as compared to testing only the resulting trajectories. It can answer questions such as at which range obstacles are detected, if obstacles are correctly classified, and if their location and size are correctly estimated. However, this NHTSA document also leaves open many details of technical realization of such tests.

D. UL4600
Another approach to achieve a safety standard is UL4600 by Underwriters Laboratories and Edge Case Research [79] (voting version). Its full name is Standard for Safety for the Evaluation of Autonomous Products. UL4600 is a voluntary industry standard for autonomous products and thus also applies to AD functions. The creation of this standard is described in [80].
Similar to the previously addressed standards, UL4600 does not prescribe how to conduct concrete perception tests. Instead, it specifies claims within a safety case for which manufacturers should deliver arguments and evidence. These claims also go into detail about sensing, perception, and machine learning [79,. Manufacturers are given a certain freedom in how to provide evidence for their claims. For example, the claim that the environment perception can provide acceptable functional performance [79,Sec. 8.4.1] must be supported at least by the metrics false negative and false positive rate, and can be supported by "other relevant metrics". In terms of existence uncertainty, false negatives (FNs) are objects missed by the SUT, whereas false positives (FPs) are wrongly detected ghost objects. For completeness, true positives (TPs) are existing objects which are also perceived by the SUT.

E. UNECE Regulation for Level 3 ALKS
The UNECE regulation [81] provides a regulatory framework for automated lane keeping systems (ALKS) of SAE Level 3 on highways. It describes the role of a technical service to whom the manufacturer has to demonstrate its compliance with the framework. Stated requirements related to OEDR are a minimum field of view (FOV) of the perception subsystem and the ability to detect conditions that impair the FOV range [81,Sec. 7]. A minimum set of scenarios for testing the FOV range is specified verbally in [81,Annex 5]. For instance, the perception of small targets like pedestrians or powered two-wheelers should be tested near the edges of the minimum FOV. The technical service is supposed to select and vary the concrete parameters of these test scenarios. Furthermore, the scenario parameters for system-level collision avoidance tests include also the perception-related conditions of the roadway and of lighting and weather [81, Annex 4, Appendix 3]. Even though this regulation provides more specific test scenarios than other standardization approaches, they are not explicitly targeted to perceptual uncertainty, and also lack detailed specifications of for example environmental conditions.

F. Safety First for Automated Driving
This white paper by a consortium of startups, OEMs, and suppliers intends to contribute to an industry-wide safety standard for AD by expanding the considerations of ISO/PAS 21448 to SAE Levels 3 and 4 [82].
Related to this review, [82,] highlights that testing is essential, but not sufficient to assure the safety of AVs. It suggests a test methodology that includes decomposing the AD system into elements such as the perception subsystem, and testing these elements separately. For the statistical validation of the perception subsystem in real-world tests, the usage of a reference perception system and the so-called scenario-based approach (see e.g. [15]) are emphasized. Recorded raw data from the field should be re-processed offline upon perception algorithm updates. The scenario-based approach can argue about a sufficient coverage of relevant traffic scenarios by grouping recorded data that include certain influencing factors into equivalence classes of scenarios (more about interpreting recorded data as logical scenarios in Sec. VII). Four constraints regarding perception testing are outlined [82, Sec. 3.6.3]: • The re-processing environment of the field recordings must be validated in terms of hardware and software. • The test scenario catalog must be both statistically significant and covering the ODD sufficiently (see also Sec. VII-B, VII-C). • The reference data quality must be appropriate for the validation objective (see also Sec. VIII-E). • Concrete test scenarios or data must be separate from those used in development (see also Sec. VII-E).
Additionally, [82, Appendix B] explicitly addresses safety challenges regarding machine learning, which is omnipresent in an AV's perception, but has not yet been sufficiently addressed in ISO 26262 and ISO/PAS 21448.

G. ISO/TR 4804
The technical report "Road vehicles -Safety and cybersecurity for automated driving systems -Design, verification and validation" [83] aims at supplementing existing standards and documents on a more technical level. Since its content is closely related to the previously published "Safety First for Automated Driving" white paper [82], we omit a repetition of the key aspects (Sec. IV-F).

H. Conclusion of Safety Standards
Contemporary automotive safety standards require arguments about successful OEDR for safety assurance, but mostly leave the realization of perception tests open to the AV manufacturer. This motivates the subsequent review of how perception testing is currently realized, and how it could be performed in future to assess if the perception allows safe vehicle behavior.

V. ESTABLISHED ACTIVITIES OF PERCEPTION TESTING
This section expresses perception algorithm benchmarking outside the safety domain (Sec. V-A) and data-driven sensor modeling on object-level (Sec. V-B) in terms of the testing taxonomy [4]. We include both established activities into this separate section because they only partially target this paper's research question. Thereby, we aim at providing a complete context for AV perception testing while distinguishing between directly safety-relevant and less safety-relevant testing activities.

A. Perception Algorithm Benchmarking
Developers of computer vision and moving object tracking algorithms across multiple industries are already using established benchmark datasets and test metrics for their algorithm's results. The idea behind these testing activities is typically to provide a quantitative ranking among different algorithms rather than evaluating whether safety-relevant pass-fail criteria are met. The hardware and software that provide the raw sensor data are usually not analyzed.
In public research, the providers of public benchmark datasets typically determine all of the three testing axes test scenarios, reference data, and metrics (Fig. 3). Test criteria usually mean that novel algorithms should rank higher than previous algorithms. 1) Scenarios: Test scenarios are captured when the dataset provider records the raw sensor data for the benchmark. The aim for such raw data is typically not to capture safety-relevant scenarios, but rather to provide a diverse set of road user types that the algorithms under test will have to detect and distinguish. Therefore, for the purposes of perception algorithm benchmarking so far, it is sufficient that the recorded scenarios are only characterized in the data format of the recordings, and not in a dedicated scenario description language (Sec. VII-B). Examples for such datasets are KITTI [84], nuScenes [85], the Waymo Open Dataset [86], or Argoverse [87].
2) Reference Data: Besides raw sensor data, algorithm benchmarking datasets typically also provide reference data on object level in the form of bounding boxes that are labeled offline with the help of humans, e.g. [84]- [86]. For testing the world model output of the perception subsystem, reference data in a metric coordinate system over the ground plane are relevant, which can be labeled based on dense lidar point clouds. In contrast, pixel-level class labels in camera images do not provide a reference for the resulting world model.
Other developer testing activities provide the reference information of target vehicles by means of high-precision global navigation satellite systems (GNSS), e.g. [88].
3) Metrics: After a perception algorithm has produced an object-based world model from the provided raw sensor data, its tracks can be compared to the provided reference tracks using metrics. Metrics that assess untracked object lists are also widely used, but are not discussed here because tracked object lists seem to be more common at the interface between the perception and planning subsystems. Since a common goal of object tracking metrics is to enable a ranking of algorithms, they often need to summarize various aspects of the perception performance into a small number of scalar values that are used for comparison. This concept of aggregating various detailed low-level metrics into fewer summarizing high-level metrics has already been proposed in 2005 for the performance evaluation of automotive data fusion [88].
Computing lower-level metrics such as the rates of FPs and FNs requires to associate (or match, often used synonymously) the estimated tracks of the SUT to their corresponding reference tracks, which is realized in the following way. The pairwise distances between all estimated and all reference  objects or tracks are computed by means of an object distance function. Examples for such distance functions are the Euclidean distance between object centroids in the ground plane (applied in e.g. [85]- [87]) or the Intersection over Union (IoU) of bounding box areas or volumes (e.g. [84]). Mathematically speaking, however, IoU is not a proper distance function because it is bounded between 0 and 1 and only its complementary value (1 − IoU ) expresses distance.
Using these pairwise distances, a multi-object association algorithm can compute an optimal association such that the sum of distances of all associated object pairs is minimized. Often, a threshold distance is applied to only allow reasonable object associations. For example, two objects with a Euclidean distance larger than, e.g., 2 m [85], or with an IoU smaller than, e.g., 50 % [84], could be prevented from becoming associated object pairs. Examples for association algorithms include the Hungarian/Munkres algorithm [89] and the auction algorithm [90].
Once this association is obtained, different higher-level metrics can be computed to describe the overall perception performance. Since the last step of an overall perception algorithm is usually to track the frame-individual objects over time while potentially fusing them into a single list of tracks, the following metrics are often called multiple object tracking (MOT) metrics.
The most common MOT metrics are the CLEAR MOT metrics MOTA and MOTP, which stand for MOT accuracy and MOT precision, respectively [91]. MOTA measures existence uncertainty by penalizing FPs, FNs, and ID switches, where ID switches are times when the ID of the estimated object changes while the ID of its associated reference object stays the same. MOTP measures state uncertainty by penalizing TPs that are not estimated precisely.
Another state uncertainty metric that additionally measures whether an estimated object state is consistent with the tracker's estimate of its own Gaussian state uncertainty is the Normalized Estimation Error Squared (NEES, see e.g. [92]).
The various forms and derivations of the Optimal Sub-Pattern Assignment (OSPA) metric [93]- [99] penalize existence and state uncertainties at the same time by weighing them to obtain a single score. Unlike most other mentioned metrics, the OSPA metric is a metric in the mathematical sense, meaning it satisfies the three properties 1.) identity of indiscernibles, 2.) symmetry, and 3.) the triangle inequality. This property makes it interesting for research on object tracking on a detailed mathematical level, e.g. [100], [101].
Besides existence and state uncertainties, the nuScenes detection score (NDS) [85] also considers classification uncertainties, as its goal is to represent the entire object tracking performance by only one scalar. Otherwise, metrics for classification uncertainty would be precision, recall, or the area under the so-called receiver-operating characteristic (ROC) curve [57].
One evaluation concept that explicitly considers temporal aspects of the object tracking performance is the Mostly tracked/Partially tracked/Mostly lost approach, which counts the number of reference trajectories that were tracked during more than 80 % or less than 20 % of their lifetimes [102].
The so-called Higher-Order Tracking Accuracy (HOTA) metric has been proposed to balance various sub-metrics in a single higher-order metric [103]. While the sub-metrics can describe individual performance aspects in detail, the higherorder metric can balance those sub-metrics without overemphasizing one aspect over another [103]. We refer the reader to the same publication [103] for a more general introduction to MOT algorithm benchmarking metrics and an in-depth analysis of popular metrics, which further include the mean average precision for tracking (Track-mAP) and the IDF1score.
Recently, the "Planning Kullback-Leibler divergence" (PKL) [104] has emerged from the field of perception algorithm benchmarking. It is the only metric from this field that we are aware of that explicitly considers the actual influence of perceptual uncertainties on the downstream motion planner. Various influences on the metric have been analyzed on the submissions of the nuScenes object detection challenge [105]. Interestingly, the submission rankings would be significantly different if PKL was used as the main benchmark instead of mAP [105]. Moreover, the PKL was found to be more consistent with human intuition than the NDS about which perceptual uncertainties are actually dangerous [104] and has been included into the official submission evaluation. Due to its direct relevance for this paper's research question, technical details on the PKL are discussed later along with other safetyaware microscopic metrics (Sec. VI-B3).
4) Difficulties with Association Uncertainty: There seems to be a fuzzy border between state and existence uncertainties, because a fixed threshold on the distance function that distinguishes TPs from FPs/FNs is not likely to produce intuitive associations under all circumstances. Therefore, the studies [106] and [107] tune their offline object association in a way that it reproduces human annotations of object association. These approaches seem to lead to subjectively better associations, but have the cost of less human understanding of how the trained associator and subsequently also the test metric works. 5) Relevance for Vehicle Safety: Except for PKL [104], the mentioned state-of-the art metrics for object perception and tracking do not consider safety, but rather provide informa-tion about the average similarity to a reference dataset [22,Safety Concern 9]. This issue is described in more detail in [65]. According to [29], it is possible to completely specify pass-fail criteria for such safety-independent metrics. Also ISO/PAS 21448 suggests to use safety-independent perception metrics [7, Annex D, 11)]. However, those criteria would only represent non-functional requirements, while functional requirements needed for safety assurance remain an open issue [29].
Similarly, the authors of [68] point out that metrics like a FP rate cannot provide safety-relevant information because some FPs might be highly safety-relevant while others are not. Therefore, they suggest to formulate realistic fault models, which already exist in ISO 26262 for e.g. hardware faults, also for perception algorithms. Such fault models should depend on the safety requirements of the overall vehicle. Given examples include the differentiation of e.g. a FP for a pedestrian from a FP for a bicycle, or to discretize continuous state uncertainty to obtain boolean faults. Nevertheless, formulating such fault models in accordance with the safety requirements seems nontrivial, which motivates for a further analysis of metrics and test criteria in Sec. VI.

B. Object-Level Data-Driven Sensor Modeling
The idea of data-driven sensor modeling approaches is usually to treat the complex behavior of a sensor as a black box and replace it by a model which can generate artificial sensor data in simulations. To do so, the output of a sensor under test is compared to reference data in order to train or parametrize a model that describes the sensor's perception performance. Hence, sensor modeling can also be interpreted as a testing activity according to the taxonomy of this paper (Fig. 4).
1) System Under Test/System to be Modeled: In the context of sensor modeling, we use the terms system under test and system to be modeled synonymously. Similar to the research interest of this paper, sensor modeling approaches like [108]- [111] model object lists, which have been generated through both sensor hardware and perception software. This is common if the sensor's built-in perception algorithm is inaccessible to the modeling engineer due to sensor supplier IP. Otherwise, the virtually tested AD function could contain parts of the perception algorithm, reducing the system to be modeled to mostly the sensor hardware [112]. However, modeling the already processed perception data has the advantages of less data volume and more comparable data formats.
2) Scenarios: Like in Perception Algorithm Benchmarking (Sec. V-A), test scenarios for the SUT are captured during data recording. Already seemingly simple test scenarios with dry and sunny weather conditions and with only few other road users often provide enough difficulty for data-driven modeling of the sensor hardware behavior. For example, modeling radar reflections on guardrails on a motorway is a topic of current research [112]. Unlike for perception software development, the research field of sensor modeling seems to be lacking common benchmarking datasets that would allow a straightforward comparison of different sensor models on the same data.  Fig. 4: Object-level data-driven sensor modeling expressed in taxonomy (Fig. 1).
3) Reference Data: Modeling the behavior of sensor hardware in a data-driven approach requires an independent reference to the sensor hardware. This makes RTK-GNSS-IMUs (see Sec. VIII-B1) a suitable and typically used source of reference data (see e.g. [108], [113] for their application). 4) Metrics: Metrics in this paper are quantitative statements on the SUT. Since parametrized or trained sensor models can describe how similar the SUT's perception is to the reference perception, they are interpreted as test metrics here. Note that sensor models are usually not single-score metrics, but rather for example probability distributions of the sensor's errors [108]. Note also that the term metric in sensor modeling literature is used differently -there, metrics make a statement on the modeling approach rather than on the SUT [114].
Actual statements on the SUT can be for example the mean values and standard deviations of Gaussian distributions that describe the SUT objects' state errors in e.g. position and velocity [110]. A nonparametric distribution for such errors that is based on a Gaussian mixture model is used e.g. in [108]. There are various further ways of expressing the SUT's errors by sensor models, which are however mostly outside the focus of this paper. 5) Relevance for Safety-Aware Perception Testing: Typically, sensor models aim at describing the specific phenomena of a sensor modality as detailed as feasible, no matter how safety-relevant these phenomena are [109], [114]. However, some sensor modeling activities explicitly address this paper's research question. For example, [65] models perception errors while considering the effect they have on robust decision making. In the context of validation of sensor models, [115] argues that a key property of modeled sensor data should be that they induce the same behavior in the downstream driving function like the real sensor data would do. Such topics are further elaborated in Sec. VI-B2 about the perception-control linkage.

VI. TEST CRITERIA AND METRICS
This section and the following two sections each cover one testing axis of the used taxonomy from [4] (Fig. 1) and are dedicated to the primary research question of this paper.
According to the taxonomy, test criteria and metrics are "a statement on the system-under-test (test criteria) that is expressed quantitatively (metric)" [4]. For example, a criterion that is qualitative at first could be quantified by means of specifying intervals on a related metric that determine passing or failing a test.
After examining how to specify perception requirements and test criteria (Sec. VI-A), we deal with safety-aware microscopic metrics and criteria (Sec. VI-B). Furthermore, metrics on the self-reporting and confidence estimation capabilities of the SUT are discussed (Sec. VI-C), as well as macroscopic metrics towards approval (Sec. VI-D).

A. Specification of Requirements and Criteria
A perception subsystem shall enable the AV to reach its destination safely, comfortably, and in reasonable time. Therefore, it shall provide information in sufficient quality about all road users that are relevant for fulfilling the driving task. However, such requirements are not specific enough to be tested [116]. Thus, how can one specify "relevant for fulfilling the driving task", or "sufficient quality" for usage as binary test criteria?
In the following, it may be useful to differentiate between concept specification and performance specification of environment perception [30]. In the mentioned source, concept specification refers to defining the properties to be perceived such as a pedestrian's pose, extent, and dynamic state, given an ODD. In contrast, the performance specification defines how well these properties should be perceived, for example in terms of detection range, confidence, and timing [30].
1) The Difficulty of Specifying Perception: AVs are expected to operate in unstructured, public, real-world environments, which are called open context in [117]. According to [72], a complete concept specification of the environment perception may not be possible because a model about the environment generally cannot cover all necessary relations and properties in such an open context. This issue has also been called ontological uncertainty [118]. For example, to specify the perception of pedestrians, one would have to specify what a pedestrian is, which is, however, only partially possible using rules such as necessary or sufficient conditions [29]. Providing examples of pedestrians in a training set is how machine learning engineers specify the concept of a pedestrian. On the one hand, this can enable driving automation of SAE Levels 3 and above, but on the other hand, it prevents a traditional specification according to ISO 26262 [29].
Besides specifying the perception concept and performance for discrete environmental aspects like the classification of a road user, another key challenge is to identify when uncertainty in continuous and dynamic environmental aspects, like a car's velocity, leads to safety-relevant failures [116].
2) Concrete Approaches of Specifying Perception: Contributions from the field of machine learning investigate how to specify the perception subsystem by using pedestrian detection as a benchmark example [29], [73]. On a higher level, [29] emphasizes the importance of an adequate language for specification and the potential of deliberate partial specifications. The paper proposes and evaluates several methods for incorporating partial specifications into the development process. Further literature with concrete specification approaches include creating an ontology of the exemplary "pedestrian" domain (concept specification) [73], and taking human perception performance as a reference (performance specification) [119]. A formal language for specifying requirements on the performance of object detection in the absence of reference data is proposed in [120], [121].
Instead of using an environmental concept like a pedestrian as the center of investigation, the methodology of [67] outlines how particular test criteria for AV subsystems can be defined in a top-down way, starting from overall safety goals. The methodology is, however, not yet applied specifically to the perception subsystem.
As mentioned earlier, the complexity of the open context can cause gaps in the specified requirements. To fill these gaps, many organizations collect large numbers of real-world mileage to discover so far unknown scenarios [6]. These recorded scenarios can also serve as test scenarios if they are identified as test-worthy (Sec. VII-C).
Besides the approaches mentioned so far, there are also considerations on perception requirements that aim at complying with the traditional ISO 26262 functional safety standard. The sources [35], [52], [53] propose dynamically associating an Automotive Safety Integrity Level (ASIL) with a given driving situation such that for example, the perception is required to comply with the stricter ASIL D in high-risk situations and with the less strict ASIL A in low-risk situations. An example of how functional safety requirements for the perception subsystem can be derived based on a fault tree analysis (FTA) is given in [122] in the context of automated valet parking.
The following subsections deal with metrics for measuring quantitatively whether specified test criteria are met.

B. Microscopic Test Criteria and Metrics
The currently most established perception performance metrics, which are typically used in machine learning, do not represent whether the perception output is sufficient for safe vehicle operation [22, Safety Concern 9] (Sec. V-A5). This section therefore reviews literature about metrics and criteria that explicitly distinguish safety-relevant from safety-irrelevant perception errors.
1) Heuristic for the Safety-Relevance: The authors of [49] provide a simple approach towards the safety-criticality of perception errors in the existence uncertainty domain. Certain fractions of the binary error types FP and FN are assumed to be safety-critical, where the fractions depend on the error's position of occurrence within the ego vehicle's FOV. For example, perception errors directly in front of the ego vehicle can heuristically be estimated to be more likely to be safetycritical than perception errors occurring farther away or with a lateral offset [49, Figure 3]. The benefit of this approach is that once its numerical values are set, it does not need to consider any downstream driving function for computing safety-critical failure rates.
2) Modeling the Perception-Control Linkage: However, whether a perception error turns out to be safety-critical or not generally does depend on the downstream driving function. Thus, we describe different ways of modeling the interface between perception and planning/control, which is also called the perception-control linkage [28].
The above-mentioned fractions of safety-critical perception errors could be determined by means of closed-loop fault injection simulations [49]. However, if the downstream driving function receives a minor update, the perception metrics would have to be recomputed, which would render such a direct approach unpractical for iterative development.
Alternatively, one could abstract the perception and planning subsystems such that test results of the perception subsystem can be re-used for varying planners. For this purpose, modular functional system architectures [5], [116], [42], [38] could be implemented with contracts, assumptions, and guarantees at the interfaces between the perception and planning subsystems [23], [45], [60], [35].
Such a modular approach would enable different safety assurance methodologies for the different subsystems. For example, the following section discusses data-driven perception testing and formal safety assurance of the planner. The idea behind such formal methods is to always assure a certain safe planning behavior, given that the environment is perceived well enough. Popular formal models for safe planning include Responsibility-Sensitive Safety (RSS) [33] and reachability analysis [43], [123].
3) Downstream Comparison of SUT and Reference Data: Traditionally, perception metrics directly compare the object list of the SUT to the one of the reference system. With an additionally given planner or assumptions about it, safetyaware perception metrics can be computed indirectly. The planner, or a formal abstraction thereof, computes behavior outputs for world model inputs from the SUT and from the reference system. Safety-aware metrics then penalize if the behavior induced by the SUT is different from the one induced by the reference system, especially in terms of danger and worst-case severity (Fig. 5).
The previously mentioned PKL metric [104], [105] (Sec. V-A3) works is this fashion. It uses an end-to-end machine-learned planner, which is trained to imitate the human ego vehicle driving behavior of the recording of the dataset on which it will be applied. During inference, this planner is fed with the reference object list and with the object list of the SUT. For both object lists in a given scene, it computes the likelihoods of the recorded ego vehicle trajectory during a short time interval after the given scene. A "Planning Kullback-Leibler divergence" (PKL) between these likelihoods then represents the degree to which the SUT's uncertainties influence the planner.
While PKL uses a concrete planner, the RSS framework [33] defines safety-relevant perception failures using a formal abstraction of the planner and its formal definition of dangerous situations. According to the publication, safety-critical ghosts are situations in which the environment perceived by the SUT is formally considered dangerous for the ego vehicle even though according to the reference data, it is not. Likewise,  Fig. 5: Safety-aware perception testing of [33] and [28] expressed in taxonomy for testing by [4] (Fig. 1), with decomposition layers of the driving function by [5] (Fig. 2) mapped into the taxonomy. Further explanation is given in Sec. VI-B3.

Scenario
safety-critical misses are situations in which the environment perceived by the SUT is formally not considered dangerous even though it actually is. For example, a safety-critical ghost can be a false-positive pedestrian detection in front of the ego vehicle that could cause a false and dangerous braking maneuver. A safety-critical miss could be a pedestrian at the same location as a false-negative detection, which could lead to the ego vehicle hitting the pedestrian. Safety-critical ghosts and misses can not only be caused by existence uncertainties, but also by state and classification uncertainties [33] because their definition is agnostic of the type of uncertainty. This avoids a potentially ambiguous differentiation between state and existence uncertainties (see Sec. V-A4). However, this concept of safety-critical perception failures [33] so far seems to have been treated mostly theoretically. For example, we are not aware of its extension to consider multiple time steps or its practical demonstration on real data.
Salay et al. [28] expand this concept of binary safetycritical perception failures to a potentially continuous description of the risk that a perception failure can cause. The work formalizes a concept to analyze the so-called incurred severity that an uncertain environment perception can cause. It also requires knowledge about the planning subsystem, for example, the planner's policy for computing actions based on a world model. Furthermore, for actions that can cause harm, there must be a way to assess the worst-case severity of this harm. With these assumptions, the microscopic and safety-aware perception metric incurred severity is defined as the difference between the SUT's and the reference system's induced worst-case severities of control actions [28] (Fig. 5).
The same work [28] applies this concept in a case study dealing with classification uncertainty of road users. For example, if a pedestrian is correctly perceived as a pedestrian, then the worst-case incurred severity is zero. If a pedestrian is however classified as a cyclist, and if the control action for cyclists is less cautious than for pedestrians, then the incurred severity of this misclassification is likely positive. Future work not yet covered in [28] is to generalize the computation of incurred severity to world models that are also subject to existence and state uncertainty. Moreover, representative real-world exposures (probabilities of occurrence) of scenes would be needed for valid risk computations.
A practical challenge in the approach of [28] might be the availability of the planner's behavior policy. This challenge is addressed in the subsequent positional paper by Salay et al. [27] on perceptual uncertainty-aware RSS (PURSS). It proposes to let the RSS framework handle world model uncertainties in a more advanced and more practical way than originally published. A behavior policy of the concrete planner is not needed; instead, it is assumed that the planner complies with the formal RSS rules for guaranteed safe control actions. The RSS model, which is explained in detail in [33], can provide a set of guaranteed safe control actions for any given world model input. Using this set, safety-aware test criteria for the perception subsystem can be defined. The set of safe control actions is computed for both the world model from the SUT and for the reference world model.  (Fig. 6).
Furthermore, the authors propose how RSS could be extended to handle imprecise world models, which are sets of SUT world models that contain the true world model with a certain confidence. For example, the position of a pedestrian could be estimated by an interval of [29 m, 31 m] with a confidence of 95 %. In practice, contemporary perception algorithms do output confidences about their world models, for example Gaussians for state uncertainties, which could be sampled at the mentioned confidence intervals. A corresponding test criterion for an imprecise world model is that its RSS-defined set of safe control actions must be safe for each precise world model contained in it [27, (2)]. The authors show that the mentioned perception confidence can be used as a lower bound for the probability of guaranteed safe control actions [27, (3)].

C. Metrics for Uncertainty/Confidence Calibration
As described in the previous section and also in [6], the explicit consideration of world model uncertainty (or  Fig. 1. inversely, confidence) can facilitate safety assurance, for example through runtime monitoring (more in Sec. IX-C3). In this context, it is crucial that the self-reported uncertainty of the SUT correctly reflects its true uncertainty [22,Safety Concern 5]. If this holds, then the SUT is calibrated [61], which means that it is neither over-confident nor too doubtful of itself.

1) Types of Uncertainty:
The literature on uncertainty in AD (e.g. [64]) describes that epistemic uncertainty, or model uncertainty, constitutes how uncertain the SUT's model of deep neural networks (DNNs) is in correctly describing the environment. In contrast, aleatoric uncertainty is caused by physical sensor properties like finite fields of views, resolutions, and sensor noise. Both types are relevant for correct calibration. Additionally, [118] proposes the concept of ontological uncertainty to describe the complete unawareness of certain aspects of the environment, even in the reference data.
2) Representations of Uncertainty: In object perception, state estimation is a regression task (continuous true value), while existence and classification estimation are classification tasks (discrete true values) [64]. State uncertainties are typically quantified by continuous probability distributions or confidence intervals, while existence and classification uncertainties are usually expressed by scalar probabilities between 0 and 1.
3) Calibration Metrics: This section focuses on metrics that only describe how well-calibrated the world model uncertainties are. In contrast, uncertainty-aware classification or regression metrics [64] are a topic of perception algorithm benchmarking (Sec. V-A).
Multiple literature sources set up so-called calibration curves [61], [124], [125] for visualization and numerical analysis. These curves plot the accuracy (or empirical frequency) of a prediction over the prediction's confidence, often using bins. For example, in classification tasks, one specific bin could contain all events where the detector reports a confidence/existence probability in between 80 % and 90 %. If there is a pedestrian present for only 70 % of all events in that bin, then the calibration curve would be tilted off from the ideal diagonal line. In regression tasks, a similar calibration curve can be set up by defining the accuracy as the empirical frequency of when the estimated confidence interval contained the true value.
Single-score calibration metrics can be computed from such calibration curves [64]. In classification tasks, the Expected Calibration Error (ECE) and the Maximum Calibration Error (MCE) [124] represent the expected and the maximum difference between confidence and accuracy in such calibration curves, respectively. Similar metrics for regression tasks are the calibration error [125] or the related Area Under the Calibration Error Curve (AUCE) [126].
Concrete applications of uncertainty calibration and evaluation in 3D object detection from lidar point clouds are given for example in [75] and the follow-up publication [76]. The latter publication explicitly uses the mentioned calibration curves and the ECE metric.
A further uncertainty evaluation approach that explicitly addresses perception systems in safety-relevant domains is provided in [70] and subsequently [63]. The publications, which focus on out-of-distribution detection for classification tasks, distinguish four different cases in classification results. The cases are combinations of the properties certain/uncertain (decided by a threshold), and correct/incorrect. Metrics are defined based on the fractions of these individual cases among all classification results. The fraction of certain, but incorrect results is defined as the Remaining Error Rate (RER), and the fraction of certain and correct results is defined as the Remaining Accuracy Rate (RAR).
All mentioned metrics so far assume that a precise world model is available as a reference. However, also reference data are generally uncertain and imprecise (Sec. VIII-D), which makes test metrics more complex.

D. Macroscopic Metrics Towards Approval
So far in this paper, the used metrics and test criteria have been statements on the SUT in either individual scenes (microscopic metrics) or in relatively small amounts of test data with a research purpose. While microscopic metrics and criteria are necessary for a detailed analysis of the SUT, they alone are not sufficient for the overall safety assurance of AVs because their results need to be extrapolated to estimate macroscopic [127] metrics (definition in Sec. I-C10).
1) Terminology: Safety vs. Reliability: Since the term reliability is often used in the context of macroscopic safety metrics [49], it is first briefly differentiated from the term safety. The perception subsystem of an AV must enable the overall vehicle to drive safely. Safety, however, is a property of the overall system and not of the perception subsystem. The perception subsystem can at best be reliable, which is generally different from being safe [9, Sec. 2.1] (Sec. I-C). Note that a property that includes both safety and reliability is dependability [128].
2) Mean Time Between Failures: The RSS publication [33] argues that for the acceptance of AVs, the overall probability of occurrence of safety-critical failures must be in the order of magnitude of 10 −9 per hour. This would imply a test criterion of no such failures during about 10 9 hours of driving, where safety-critical failures in their publication are the previously introduced safety-critical ghosts and misses. The related paper [66], which focuses on perception within RSS, states another large required mean time between failures of 10 7 h for the perception subsystem. Only perception failures that can possibly lead to unsafe vehicle behavior should be counted in that number.
Major contributions to the macroscopic reliability assessment of the perception subsystem have come from Berk et al., e.g. [49], [36]. The macroscopic metrics they use are the rate of all perception failures and the rate of safetycritical perception failures, both measured in failures per hour. Safety-critical perception failures are computed by filtering all perception failures with a field-of-view-dependent safetycriticality factor between 0 and 1, as explained in Sec. VI-B1. An overall approval criterion according to [36] is that the sum of the failure rates of the perception, planning, and actuation subsystems is smaller than a given threshold rate, which is called the target level of safety. A corresponding testing methodology to compute these macroscopic perception metrics is discussed in Sec. IX-C4.

E. Conclusion of Test Criteria and Metrics
Different metrics describe different properties of the SUT like its safety-relevant performance in a single scene, the calibration of its self-reported uncertainties, or its statistical safety impact. Specifically the representation of safetyrelevance in microscopic metrics seems challenging. These mentioned different classes of metrics are typically coupled. For example, the SUT's statistical safety impact depends on its safety impact in individual scenes, which in turn may depend on whether the SUT has correctly reported its uncertainty in that scene. Thus, one challenge is to harmonize individually dedicated metrics into an overall framework. Moreover, the specification of test criteria is nontrivial due to the complexity of the open-world context.

VII. TEST SCENARIOS
According to the taxonomy of [4], a test scenario is "a set of specified conditions" under which a test is executed. The following subsections deal with adapting the term scenario to the perception context (Sec. VII-A), describing test scenarios and the ODD (Sec. VII-B), generating a test scenario catalog (Sec. VII-C), executing scenarios as test cases (Sec. VII-D), and splitting scenarios into training and test sets (Sec. VII-E). Much of the literature on scenario-based testing does not yet focus on perceptual uncertainty, which is why this section also cites related literature with a broader focus.

A. Adapting the Term Scenario to the Perception Context
The definition of the term scenario has been discussed extensively in the context of scenario-based testing [12]. According to the mentioned source, a scenario contains actors (mostly road users) whose actions influence the temporal development of scenes within the scenario. For testing purposes, one of such actors within the scenario is the subject vehicle. So far, the focus of scenario-based testing of AVs has been on the plan and act subsystems instead of on the perception subsystem of the subject vehicle [15]. However, if only the perception subsystem is tested in an open-loop fashion, then it typically only has the role of a passive observer of the scenario instead of an influencing actor. At this point, we exclude closed-loop testing because we are interested in the offline test of potentially multiple perception algorithms on the same recorded raw sensor data from the real world.
To the best of our knowledge, we are not aware of literature that explicitly considers the influence that an open-loop perception subsystem may have on the temporal development of a scenario. Such an influence is in fact possible through e.g. electromagnetic waves emitted by active sensors, which could disturb the sensors of other AVs and thus theoretically influence their behavior. However, in this review, we assume for simplicity that perception-specific scenarios are scenarios whose temporal development of objective scenes is pre-determined, but where the subjective observation of these scenes in world models is left open. This corresponds to static scenarios with an additionally predetermined ego vehicle behavior according to [129].
Furthermore, it seems that the different abstraction levels of scenarios (functional, logical, concrete) of [130] can also be used for perception scenarios. A functional scenario is described verbally (e.g. it rains, among other aspects), whereas a logical scenario has parameters (e.g. uniform precipitation in millimeters per hour), and a concrete scenario gives each parameter a numerical value (e.g. 2 mm/h). Generally, different logical scenario parameters are of interest for perception tests than for planning tests [131].

B. Description of Scenarios and ODD
Arguing whether test scenarios cover the entire ODD requires ways to describe scenarios and the ODD.
Current ODD description approaches are the taxonomy of BSI PAS 1883 [132], the NHTSA framework [78,Ch. 3], and the ongoing standardization of a machine-readable ODD format in ASAM OpenODD [133]. Furthermore, ontologies are used to describe ODDs [59] and scenes within scenarios [44]. Ontologies are intended to be standardized in ASAM OpenXOntology [134] and have also been proposed to define a schema for world model data [27]. For ontology-generated scenarios, the notion of abstract scenarios was introduced to describe functional scenarios that are formal and machinereadable, but do not yet have a logical parameter space [131]. The ontology of [44], which is used to generate traffic scenes for scenario-based testing, organizes all scenario entities by means of a 5-layer-model. This model was first defined using four layers in [135], and has been adapted to six layers in [136] and most recently [137]. The model's individual layers according to the most recent and also summarizing publication [137] are 1) Road network and traffic guidance objects 2) Roadside structures 3) Temporary modifications of Layers 1 and 2 4) Dynamic objects 5) Environmental conditions 6) Digital information.
These scenario description layers should not be confused with the functional decomposition layers of [5] (Fig. 2). Even though the present review only focuses on perceiving dynamic objects in Layer 4, the properties of all other layers can influence the perception of dynamic objects and are therefore also relevant.
So far, the layer models for scenario description only provide a way for an objective description of the environment within a traffic scenario. Aspects related to a subjective perception of such scenarios through machine perception are not yet explicitly covered, for example surface materials that affect radar reflections [137]. Such shortcomings of scenario descriptions for perceptual aspects have also already been identified in the context of sensor modeling [112,Sec. V].
The previously mentioned scenario description methods are suited to describe functional, abstract, and perhaps logical scenarios. In contrast, describing concrete scenarios for the actual test execution requires a certain data format. The ASAM OSI [138] classes GroundTruth and SensorView appear to be capable of describing objective scenes and the circumstances for their subjective perception, respectively. However, they are designed for virtual simulations instead of real-world test scenarios. Similarly, the established format OpenSCENARIO [139] is also designed for virtual closed-loop testing of the planner instead of open-loop testing of the perception subsystem.

C. Generating a Test Scenario Catalog
The ability to describe and structure test scenarios in an ideally formal way is a prerequisite to subsequently generate a representative test scenario catalog. Such a test scenario catalog might be part of the specification of the perception subsystem (Sec. VI-A).
The number of test scenarios has to stay feasible, which renders naive sampling of a high-dimensional scenario space unsuitable and instead motivates for the following approaches. We distinguish between knowledge-driven and data-driven scenario generation, as already done in [4], [16], [131]. As pointed out by [16], approaches from both categories usually need to complement each other for a holistic testing strategy. Likewise, also ISO/PAS 21448 mentions dedicated expert scenario generation (knowledge-driven), as well as largescale random data recording (data-driven) for the perception verification strategy [7,Annex D].
In either way, the goal is usually to obtain certain triggering events 1 , which are defined in ISO/PAS 21448 [7] as "specific conditions of a driving scenario that serve as an initiator for a subsequent system reaction possibly leading to a hazardous event". Such a system reaction can occur either in the sensor 1 or similarly, triggering conditions hardware, in the perception software, or the downstream driving function. Similar to triggering events, the terms external influencing factor [41] and criticality phenomenon [131] are also used to describe safety-relevant scenario influences related to perception.
1) Knowledge-Driven Scenario Generation: Since human knowledge of traffic scenarios is often qualitative, knowledgedriven scenario generation usually first yields functional or abstract scenarios. Literature on driving scenarios describes the further steps to logical scenarios [140] and the discretization to concrete scenarios [141].
ISO/PAS 21448 enumerates a list of influencing factors, which shall be used to construct scenarios to identify and evaluate triggering conditions for the perception subsystem [7, Annex F]. This list is included into the standard as an informative example and includes factors like climate, which can be for example fine, cloudy, or rainy.
A specific identification of triggering events according to ISO/PAS 21448 is provided in [40] by means of a hazard and operability study (HAZOP). The approach is applied to the used sensor modalities camera and lidar as part of an extended automated emergency braking system. Sensor modality-specific influencing factors can be analyzed in even more detail by experts using dedicated tests or simulations. Example studies analyze the influences of weather, road dirt, or rainfall on lidar ( [56], [55], [51], respectively), or corner cases specific to visual perception [69].
Such influencing factors or triggering events could also be queried from a domain ontology for driving scenarios [44] or from one that also includes perceptual aspects [131]. The former source argues that without the use of an ontology, human experts are not likely to generate all possible scenarios based on their knowledge. In contrast, if the knowledge is first encoded in an ontology, then the generation of scenarios or scenes from that ontology can cover all possibilities based on the encoded knowledge.
A further approach towards completeness of a scenario catalog is to crowd-source a list of concrete OEDR aspects that are relevant for testing [46]. The mentioned publication provides a detailed list that is supposed to serve as a starting point for further extension by the community.
While so far, the mentioned influencing factors were selected mostly according to their influence on the perception subsystem, one can also construct perception test scenarios with the downstream driving function in mind. This particularly includes scenarios that would be challenging to the driving function in case a perception failure occurs. Such a concept is applied under the name of application-oriented test case generation in [41] and [58] with ACC systems in mind. For an ACC function, it appears possible to manually investigate the perception-based influencing factors that negatively affect its behavior. In contrast, if the targeted application is a general-purpose urban Level 4 system, it will be harder to enumerate all related influencing factors.
2) Data-Driven Scenario Generation: If test scenarios are obtained by analyzing large amounts of randomly recorded vehicle fleet data, then the abstraction level of available data is closer to concrete scenarios than to functional scenarios. However, the recorded data is initially only available in the data schema of the recording format, which usually does not share the same parameters like a logical scenario space. Thus, a challenge in data-driven approaches is to generalize the snippets from the recorded data in a way that they correspond to points in a logical scenario space. This is necessary in order to vary individual parameters such that potentially relevant, but not yet observed scenarios can also be generated. We are not aware of scientific publications that present datadriven scenario generation in the safety context specifically for perception testing, and thus will refer to related literature about general testing, which has already been reviewed in more detail in [16].
One example approach for a data-driven test case generation is described in [142]. In the mentioned publication, a database system of relevant scenarios is described. Its input data from various sources such as field operational testing, naturalistic driving studies, accident data, or others, must first be converted to a common input data format of traffic recordings. Afterwards, individual snippets of the input data are affiliated to pre-determined logical scenario types, such that each snippet corresponds to one point in a logical scenario space. The logical scenario parameters can also be obtained through unsupervised learning on the input data [143]. If the input data were obtained from large amounts of randomly collected driving data, then one can further compute statistical measures such as the scenario's real-world exposure and its parameters' real-world probability distributions [142]. By additionally computing criticality metrics on the given snippets, one could estimate the potential severity of a section of the logical scenario space. The exposure and severity as meta-information on the scenario space then help to turn the obtained logical scenarios into relevant test cases. For example, one could put more testing efforts on scenarios with a high risk, where the risk of a scenario increases with its exposure and severity, and decreases with its controllability [144,Appendix A].
3) Combined Scenario Generation: As pointed out by [16], knowledge-driven and data-driven approaches for scenario generation often complement each other in practice. For example, the previously mentioned database approach from [142] assumes that a concept for logical scenario spaces is given, potentially from expert knowledge. In turn, knowledge-driven scenario generation can also be backed up by measurements and data [16].
One example of a combined test case generation for the perception subsystem is given in [5]. Besides the functional decomposition layers (Fig. 2), the publication also proposes a method for defining particular test cases for each of those decomposition layers. The method takes driving scenarios from a database such as [142], assumes an accident if there was none, and then uses a fault tree analysis (FTA) to define pass/fail criteria for the individual subsystems in a given scenario. This combined approach allows the derivation of perceptionspecific test scenarios from general driving scenarios.

4) Test Scenarios
Specific to the SUT: In the end, it could also be that the test scenarios that result in the highest risk for a given SUT are not found by only considering external influencing factors from the space of all possible scenarios. Instead, the unexpected insufficiencies that the DNNs of the SUT have learned might lead to more critical behavior than what a usually suspected external influence could trigger [22,Safety Concern 4]. For example, [145] identifies perception subsystem inputs that lead to highly uncertain world models for a given SUT by propagating uncertainties through the individual perception components. 5) Difficulties in Covering the ODD with Scenarios: Any attempt to formally describe the ODD and to cover it sufficiently with test scenarios is generally difficult because of the open context problem of the real world [117]. Related to this, [22, Safety Concern 1] expresses that the data or scenarios used for the development and training of the perception subsystem are usually not a good approximation of the system's ODD in the real open world. Furthermore, the real world changes over time, which makes it necessary to iteratively update the ODD description and its coverage with test scenarios [22,Safety Concern 2].
Moreover, in practice only a finite number of concrete test scenarios can be sampled from the infinite number of concrete scenarios that could be theoretically generated from a logical scenario. This sampling makes it generally unclear whether the results of two similar concrete test scenarios also hold in between these discrete points. In fact, DNNs used for object detection can show a potentially extreme non-linear behavior with respect to only slight input variations for example in weather conditions. This brittleness, which has been pointed out in [22,Safety Concern 6], makes it difficult to argue about a sufficient ODD coverage with sampled test scenarios.
Nevertheless, combinatorial testing [146], which is suggested by ISO/PAS 21448 and already applied to computer vision for AVs [147], can potentially keep the number of discretized concrete tests feasible while providing certain coverage guarantees. Furthermore, combinatorial testing that explicitly exploits the functional decomposition approach for AVs [5] is performed in [148] for simulated AV testing.

D. Executing Scenarios as Test Cases
We assume the execution of a test scenario to be an offline execution of a perception algorithm on previously recorded raw sensor data. In this way, only the sensor mounting positions and sensor hardware are fixed by the recording, whereas multiple perception algorithms can be applied.
For scenarios that correspond to snippets from existing real-world vehicle recordings, the corresponding raw data can directly be used for the test without having to be re-recorded. Otherwise, the raw sensor data for the test scenarios must be captured. Public roads offer less influence on the scenario details, but allow encountering many different scenarios over time with little effort per scenario. In contrast, proving grounds allow to set up specific scenarios, which however requires high efforts per scenario and may lack realism, e.g., when reproducing precipitation.

E. Training and Test Sets of Scenarios
If the DNNs of the SUT are trained on data and scenarios that are also used for testing, then overfitting likely takes place during training and an unbiased testing for verification and validation purposes is not possible. Thus, it is important to properly separate recorded perception data into individual datasets for training and for testing [22,Safety Concern 7], [82]. Furthermore, if the developers evaluate the SUT on a test set multiple times, an unintentional optimization with respect to the test set result might take place over multiple development iterations [22].

F. Conclusion of Test Scenarios
Scenario-based approaches to minimize the overall testing effort of AD functions also seem applicable to the perception subsystem [82,Sec. 3.3.4]. An agreed-upon description language and data format of perception scenarios would be beneficial for further research. Various approaches have been proposed to obtain test scenarios, where their combination for a sufficient ODD coverage is an active research topic. Furthermore, it is challenging to obtain real-world recordings of all identified test scenarios.

VIII. REFERENCE DATA
The third and final testing axis that this paper covers is the "knowledge of an ideal result (reference)" [4]. For objectbased environment perception, such an ideal result is an object list containing all road users that the SUT is supposed to perceive. We distinguish between the terms ground truth and reference in a way that a ground truth perfectly and objectively describes the world. A reference is a subjective approximation of the ground truth in terms of data that are generated by a perception system that is superior to the SUT.
Requirements for reference systems for environment perception have been analyzed in depth in the context of ADAS [34]. Apart from the qualitative requirements mobility/portability and reporting of its own uncertainty, the paper states the quantitative requirements reliability, sufficient field-of-view, accuracy, and proper timing of the measurements. The following sections review different ways of generating reference data, where each way has its own advantages and disadvantages in the mentioned categories. We structure the literature according to the position of the reference sensors, which can be mounted only on the ego vehicle (Sec. VIII-A), also on other road users (Sec. VIII-B), or externally and not part of any road user (Sec. VIII-C). Uncertainty in reference data and an appropriate choice of the reference data source are discussed in Sec. VIII-D and Sec. VIII-E, respectively.

A. Reference Data From Ego Vehicle Sensors
The biggest advantage of reference data from sensors of the ego vehicle is that no external measurement equipment is needed. Without an external perspective, however, road users that are occluded for the sensors under test can also hardly be present in the reference data.

1) Reference from Sensors Under Test:
Perhaps the most common approach of generating reference data is to use the raw sensor data from the SUT such that humans can label the reference objects and their properties. Labels in a sensor-specific coordinate system, such as in a camera's pixel space, need to be transformed to the world model's coordinate system (often in meters and defined over the ground plane) to potentially serve as a reference for it. In practice, the human labeling process is often at least semi-automated by offline perception algorithms, which can go back and forth in time. Such reference data is typically used by datasets for perception algorithm benchmarking such as [84]- [86].
Using only offline perception algorithms without human intervention (as also done in e.g. [149]) dramatically reduces the overall effort and could still result in superior data to the SUT, but generally impairs the data quality. A standardization attempt for human-labeled reference data is ASAM OpenLABEL [150].
In either case, no additional sensor hardware is required, which is valid and a big advantage if the perception algorithm is the only component of interest. At the same time, it means that the SUT's sensor mounting positions and sensor hardware do not have an independent reference, which might impede their rigorous assessment.
2) Separate Reference Sensors on Ego Vehicle: Additional sensors on the ego vehicle that are not part of the SUT can potentially also provide a reference for validating the SUT's sensor mounting positions and hardware performance, see e.g. [149]. This approach seems to be more popular in ADAS developments than for AVs because AVs often already use the best available sensors for their regular operation, and hence there are no other sensors of higher quality left as reference.
3) k-out-of-n-vote of High-Level Fusion Inputs: The present review paper focuses on testing the world model that is generated from fused inputs of all environment sensors. In contrast, Berk et al. [32] propose testing the object-level outputs of individual sensor systems before those are fused (more in Sec. IX-C4). Such testing is needed for the authors' proposed safety assessment strategy for the perception subsystem, where the corresponding reference data could be generated without additional sensors and without manual human effort.
If for example, a majority of fusion input systems detects a certain road user while one input system does not, then it can be argued that the majority of agreeing input systems provides the reference data for identifying a perception failure in the single disagreeing system. This idea is generalized to a so-called k-out-of-n vote of sensor systems. It could potentially scale to billions of kilometers of data collection using large vehicle fleets with feasible effort, but its practical implementation on real vehicle data seems to have not yet been described in the available literature.

B. Reference Data from Other Road Users
Other road users can be either equipped with specific test equipment for the estimation of their position and state, or they could communicate their states to the ego vehicle as part of a potential V2X-based regular operation in future.
1) RTK-GNSS-IMUs: Global navigation satellite systems (GNSS) with real-time kinematic correction (RTK) and an attached inertial measurement unit (IMU) have been popular for generating reference data of individual road users. Paired systems of those can be installed both in the ego vehicle and at other road users to determine relative positions, velocities, and orientation angles. RTK-GNSS can measure the absolute position up to a few centimeters, but face difficulties when the signal is affected by nearby structures such as buildings, trees, or a tunnel. IMUs work independently of the surroundings, but only provide relative and incremental positioning information. Data fusion algorithms can combine the strengths of both components to estimate a consolidated state of the respective road user.
Despite the size of the measurement hardware, also vulnerable road users (VRUs) can wear such hardware as a backpack [151], which, however, alters their appearance to other sensors. In general, only specifically equipped road users can provide reference data using RTK-GNSS-IMUs, which excludes the majority of public road users.
The accuracy of RTK corrections of GNSS signals in automotive applications has been analyzed in more detail in [152]. Another work has specifically analyzed the accuracy of GNSS systems that are used for relative positioning of local groups of vehicles [153]. For more information on GNSS in general, we refer to the textbook [154].
2) Collaborative World Model Through V2X: If target vehicles have the capability to accurately localize themselves without dedicated testing hardware, then they could communicate their state to a cloud-based or edge-based collective environment model [155]. The mentioned publication hypothesizes how this could enable a mutual verification of the environment perception of individual road users. The key difference to the previously described RTK-GNSS-IMU approaches is that the publication [155] aims at the regular operation of future production vehicles rather than dedicated testing activities.
The targeted crowd-sourced reference data generation follows a similar idea like the previously described k-out-of-nvote of high-level fusion inputs (Sec. VIII-A3). The difference here is that the inputs to the vote would originate from vehicles instead of from sensors. However, also this approach does not seem to have been demonstrated with real-world vehicles yet.

C. Reference Data from Non-Road Users
The analyzed literature on reference data generation with sensors external to the road users includes stationary infrastructure sensors, unmanned aerial vehicles (UAVs, or drones), and helicopters. Those data are often intended as either standalone naturalistic traffic data or as online enhancements of the AV perception. They can still serve as a reference for the SUT if their quality is superior to the SUT according to context-specific reference data requirements. A data quality superior to the SUT on the ground can be achieved by a more advantageous bird's eye perspective, by intentional overfitting of the perception algorithms to the sensor's location, and by human error-checking. Humans can detect errors in reference object lists easier when comparing them to a raw video from a bird's eye perspective than by comparing them to a potentially disturbed GNSS signal or to potentially incomplete SUT raw data.
1) Reference Data from Stationary Infrastructure Sensors: Stationary infrastructure sensors are sensors that are mounted on buildings, streetlamps, gantries, bridges, or other nearby structures. They are used in urban locations for example in the Ko-PER dataset [48], in the test area autonomous driving Baden-Württemberg [71], and at the AIM test site [156]. Infrastructure sensors at highways or motorways are used in [157] and at the Providentia sensor system [158], where the latter source assesses its performance using a helicopterbased system, which is covered in Sec. VIII-C3. The projects HDVMess and ACCorD aim at installing mobile and fixed infrastructure sensors at various sites [159] and evaluate their data generation with a UAV-based reference system [160].
2) Reference Data from UAVs: Camera-based reference data generation from a bird's eye perspective has been investigated at least since 2014 [47]. The necessary processing steps to track ground vehicles from UAVs were later described in more detail in [161] and [162]. UAVs are used as reference sensors for sensor modeling in [110] and for assessing infrastructure sensors in [160]. A qualitative assessment of UAVbased reference systems in contrast to other reference systems is given in [163]. Reference data from a UAV's perspective tend to suffer from less occlusions than from perspectives closer to the ground, however, new sorts of occlusions from treetops or bridges occur.
3) Reference Data from Helicopters: Additionally, helicopters have been used to record the ground traffic for verification and validation purposes of AD systems. The DLRAD dataset [164] was partially recorded using a helicopter that follows a target vehicle with active environment perception. A previous publication from the same project [165] describes how such a helicopter-based system can be used to validate the behavior of ADAS. Furthermore, a helicopter-based reference data generation is also described and used in [158] to assess a static infrastructure sensor system.

D. Uncertainty in Reference Data
Reference systems do not provide an absolute ground truth, but only have to come closer to it than the SUT does. This has been pointed out and quantified with respect to labeling uncertainty in [166]. Thus, the quality of the reference data (or labels) should be taken into account such that test results are not misleading [22,Safety Concern 8].
For automatically generated reference data from external sensors, there might not be labeling uncertainty, but uncertainty in temporal synchronization of reference and SUT data, as well as uncertainty in their spatial alignment.
Moreover, any automatically generated reference data without human error-checking can suffer from unexpected inaccuracies in the reference measurement system. Those inaccuracies might be rare enough to not matter in small-scale R&D testing, but over large amounts of test scenarios or kilometers, as needed for safety assurance, the probability of encountering relevant reference system errors increases.

E. Choice of Reference Data Source
Different ways of generating reference data are used for different aspects of perception testing, where each way has its unique advantages. For example, human labels might be best for testing what a perception algorithm can infer from given raw data in the absence of additional data sources. Separate high-quality sensors on the ego vehicle could be the most useful choice for benchmarking individually inferior sensor systems. RTK-GNSS-IMUs might be best for testing the perception of small numbers of vehicles in the absence of high buildings. Data from an aerial perspective might be most suitable for testing the perception performance under static and dynamic occlusions in complex naturalistic traffic. Infrastructure test fields that spread over a large area can be best suited for investigations that require a large field of view. Finally, reference data from the k-out-of-n-vote might have the best potential for keeping the testing effort feasible for large amounts of kilometers or scenarios.

F. Conclusion of Reference Data
The most appropriate choice of reference data depends on the detailed objectives of the respective testing activity within the safety case ([82, Sec. 3.6.3]). In any case, the reference data's limitations should be considered.

IX. DISCUSSION OF REVIEW RESULTS
Sec. IX-A summarizes the particular conclusions of each testing axis in terms of the largest issues regarding the primary research question. Intersection topics between the axes are covered in Sec. IX-B. Finally, testing as reviewed in this paper is likely necessary, but not sufficient for safety assurance of AVs. Therefore, Sec. IX-C places the reviewed topics into a broader context by drawing connections to related activities around perceptual aspects in safety assurance.

A. Open Issues per Testing Axis
In terms of metrics and test criteria (Sec. VI), current challenges are the incompleteness of criteria due to the openworld context and also the consideration of a potential safety impact in the metrics.
Open issues about perception-specific test scenarios (Sec. VII) are a common description language and format, a combination of scenario generation approaches for sufficient ODD coverage, and the difficulty of recording the identified test scenarios in the real world.
The most appropriate source of reference data (Sec. VIII) depends on the investigated aspect of environment perception.
A combination of multiple such investigations into a harmonized methodology targeting the safety proof appears to be not yet demonstrated in the literature.
The mentioned open issues from the literature might be the reason why as of now, safety standards (Sec. IV) leave many details around perception testing open.

B. Open Issues Between the Testing Axes
Solutions to the individual testing axes must be able to fit into a consolidated methodology that also considers intersection topics between the axes. For example, at the intersection of test criteria and scenarios, the criteria on the SUT's FOV might be relaxed if the ego vehicle drives slowly due to rain in the respective scenario. In between the axes scenarios and reference data, the fact that rain is present in the scenario might affect some ways of reference data generation more than others. The intersection between reference data and metrics/criteria includes the metrics' sensitivity to details in the reference data. For example, a metric that penalizes false negatives behind occlusions produces different results depending on whether the reference data contain these occluded objects or not.
While an all-encompassing methodology that covers all of such details seems to be not yet demonstrated, approaches such as [30] or [49] already propose general ways of how the dependability of the perception subsystem could be managed and tested.

C. Further Safety Assurance Activities Regarding Perception
Testing as described in this review (see Table I) is only one activity out of many within an all-encompassing safety argumentation. The following subsections intend to reveal this larger context by drawing connections to related activities in perception dependability. We structure the approaches according to the means for attaining dependability by [128], which have recently been adopted to uncertainties in AD [118]. They consist of prevention, removal, tolerance, and forecasting of faults or uncertainty, respectively. 1) Uncertainty Prevention: Formal verification of the perception subsystem could ideally prevent the occurrence of certain world model uncertainties. So far, we are not aware of formal verification methods that cover the entire perception subsystem. Also, the review paper [15] only mentions formal verification methods for the downstream driving function. However, formal analysis and verification of DNNs for perception tasks is an active topic of research [167], [168]. Moreover, the way in which uncertain world models affect formally verified downstream driving functions is actively being researched, for example in the context of RSS [27].
Another means of uncertainty prevention is the restriction to certain ODDs [118].
2) Uncertainty Removal: Perceptual uncertainties can be removed during development and during its use after the initial release. Uncertainty removal during development includes safety analysis methods such as fault tree analysis (FTA) [118], or insightful developer testing with the goal of reducing found uncertainties afterwards. For example, white-box-testing of the perception subsystem can reveal uncertainties related to the physically complex measurement principles (see e.g. [112]). Human-understandable insights into the DNNs for object detection are, however, difficult [22,Safety Concern 3].
Testing on simulated raw data allows the perception algorithm to be tested without the need for physical sensor hardware (e.g. [169] for grid mapping). For testing object perception algorithms, a simulation environment such as [170] could produce both the raw sensor data as an input to the algorithm under test and a ground truth object list as a reference. A hybrid approach is to test the perception algorithm on recorded real data with simulated injected faults [171].
After the initial release, incremental safety assurance through system updates is a necessary means to remove uncertainties because the ODD of the AV changes continuously [22,Safety Concern 2]. Ideally, the safety argumentation should allow minor system updates without having to repeat major testing efforts [6]. Agile safety cases [172] could enable the safe deployment of such incremental updates through a DevOps-like workflow [74].
3) Uncertainty Tolerance: Runtime monitoring of the uncertainty of the produced world model can help mitigating the safety consequences of perception errors [173]. An RSScompliant planner would become more cautious for uncertain world models due to a reduced set of guaranteed safe control actions [27] (Sec. VI-B3). Similarly, in a service-oriented architecture, sensor data that is monitored as too uncertain or faulty could lead to a bypass of the usual information processing pipeline in favor of a safe behavior for degraded modes [174, Fig. 3]. An overview of self-representation and monitoring of subsystems is given in [175]. Similarly, in the ISO 26262 context, it was proposed that the perception subsystem dynamically outputs one world model for each ASIL level such that the planner can dynamically take just that world model that it needs to comply to the ASIL requirement of the current driving situation [35], [52], [53]. In any of such applications, it is crucial to assure that the reported uncertainties are correctly calibrated (Sec. VI-C).
Furthermore, the mentioned collaborative world model through V2X (Sec. VIII-B2) can also serve as a means for uncertainty tolerance if participating cars can reduce their world model uncertainties through, e.g., a cloud-based fusion [176].
4) Uncertainty Forecasting: The failure rates in a fused world model must be extremely low, which means that the direct test effort for proving or forecasting it is extremely high.
One approach to overcome this so-called approval trap [177] is to test the more failure-prone world models of individual sensor systems before they are fused [33], [49]. The latter source investigates how this could be realized without separately generated reference data, allowing the tests to be potentially executed in shadow-mode by production vehicles. In this approach (Fig. 7), the reference data take the form of a k-out-of-n-vote of object-level fusion inputs (Sec. VIII-A3). Such a mutual cross-referencing mechanism is also suggested in Safety Fig. 7: Perception testing for macroscopic safety assurance without separately generated reference data [32], [49], expressed in taxonomy (Fig. 1).
The mentioned testing approach exploits the redundancy of fusion inputs in the following way [31]. If a mean time between failures of for example 10 4 h can be demonstrated for two statistically independent inputs to a data fusion system, then the fusion output has a mean time between failures of about 10 8 h. Such a redundancy exploitation has also been stated in the RSS papers [33], [66] and in literature on perception subsystem architectures [178], [39] and perception requirements [35]. The actual number of necessary test kilometers is then computed in a statistically sound way based on how statistically dependent the inputs to the object-level fusion system actually are [50], [54].
Whether or not perceptual uncertainties lead to safetycritical failures on the vehicle level might require a sensitivity analysis of the driving function. Reachability analysis (e.g. [43], [123]) could be used to analyze in detail whether an uncertain world model can lead to unsafe future states, as outlined in the context of the incurred severity metric [28]. An impact analysis of perceptual errors on the downstream driving function has also already been described in [65].
Furthermore, uncertainty forecasting for the release argumentation could make use of the tests mentioned under Uncertainty Removal (Sec. IX-C2) if they are validated and executed on separate test datasets.

X. CONCLUSION
We analyzed literature from multiple neighboring fields, attempting to provide a structured overview of safety-relevant testing activities for the environment perception of AVs. These neighboring fields included, but are not limited to, AV safety assurance, perception algorithm development, and safeguarding of artificial intelligence. A combined literature search process that consisted of undocumented search, keywordbased search and snowballing search required large efforts, but appeared to be suitable for covering the relevant literature.
The analyzed test methods from the available literature seem to be not yet capable of demonstrating the dependability of the perception subsystem for series-production Level 4 vehicles. To overcome this issue, the provided overview can serve as a common basis for a further harmonization of primary literature contributions into all-encompassing novel testing methodologies.

ACKNOWLEDGEMENT
The authors would like to thank their colleagues at the Institute for Automotive Engineering and their project partners of the project VVM, a project of the PEGASUS family, for their valuable feedback on the initial version of this paper. Furthermore, the authors would like to thank Robert Krajewski for fruitful discussions.
Despite all efforts, we might have unintentionally overlooked individual literature contributions to the research question. In that case, we apologize to the corresponding authors and are welcoming such feedback from the community.