Autonomous Vehicles and Urban Infrastructure:
AI Decision-Making Under Uncertainty
Prof. Carlos Mendez, Dr. Yuki Tanaka, Dr. Fatima Al-Hassan
Transportation AI Research Center, Carnegie Mellon University
February 2025
Abstract
Autonomous vehicles (AVs) represent one of the most complex real-world deployments of
artificial intelligence, requiring real-time decision-making under uncertainty in dynamic, safety-
critical environments. This paper analyzes the current state of AV AI architectures, with
particular emphasis on perception, prediction, and planning under distributional shift and
adversarial conditions. We review 98 empirical studies and accident reports, develop a
taxonomy of failure modes, and examine the regulatory and ethical frameworks that govern AV
deployment. We argue that existing evaluation methodologies are insufficient for certifying safe
deployment at scale and propose a new multi-layered safety validation protocol.
Keywords: autonomous vehicles, reinforcement learning, perception, safety validation,
distributional shift, AI regulation
1. Introduction
The promise of autonomous vehicles — safer roads, improved mobility for underserved
populations, reduced emissions through optimized routing — has driven more than $200 billion
in cumulative investment since 2010. Yet despite decades of research and extensive public
road testing, fully autonomous vehicles (SAE Level 4 and 5) remain commercially limited to
specific geographic domains. The gap between current performance and the robustness
required for unrestricted public deployment reflects deep technical challenges in the AI systems
that underpin AV operation.
AV AI systems must solve a stack of interdependent problems: perceiving the environment from
multimodal sensor data (cameras, LiDAR, radar), predicting the future trajectories of other road
users, planning safe and comfortable maneuvers in response, and executing those plans
through low-level vehicle control. Each layer introduces its own sources of uncertainty, and
errors can compound across the stack in ways that create dangerous situations.
2. Perception Systems
2.1 Sensor Fusion
Modern AV perception architectures fuse data from cameras, LiDAR, and radar using deep
neural networks. Camera-based systems benefit from rich semantic information and low cost
but are sensitive to lighting conditions. LiDAR provides precise depth measurements but is
vulnerable to adverse weather and has high per-unit cost. Radar is robust to weather but has
limited resolution. Transformer-based multi-modal fusion architectures such as BEVFusion have
demonstrated superior performance to unimodal systems on standard benchmarks.
2.2 Distributional Shift
A fundamental challenge for AV perception is distributional shift: the degradation of model
performance when deployment conditions differ from training data. Studies have shown that
perception models trained on data from one geographic region exhibit 15-35% accuracy
degradation when evaluated on data from climatically or architecturally distinct regions. Long-tail
events — construction zones, unusual vehicles, rare pedestrian behaviors — are particularly
problematic due to their underrepresentation in training datasets.
3. Prediction and Planning
Accurate prediction of other road users' future trajectories is essential for safe AV planning.
State-of-the-art prediction models use social force models, graph neural networks, and
transformer architectures to jointly model the interactions of multiple agents. However, human
behavior is fundamentally uncertain and context-dependent: the same physical configuration
can elicit different behaviors depending on cultural norms, driver intent, and road conditions.
Planning under this uncertainty requires approaches that trade off efficiency against safety.
Conservative planners prioritize safety but create traffic disruptions through overly cautious
behavior. Aggressive planners optimize for efficiency but increase accident risk. Probabilistic
planning frameworks attempt to navigate this tradeoff through risk-aware optimization, but
calibrating acceptable risk levels in life-safety systems remains philosophically and legally
unresolved.
4. Failure Mode Analysis
Analysis of publicly reported AV accidents and simulation studies reveals four primary failure
mode categories. Perception failures account for 41% of documented incidents, with object
detection failures in adverse weather and edge-case scenarios being most common. Prediction
failures — where the AV correctly perceives but mismodels the behavior of other agents —
account for 28%. Planning failures, where the AV generates unsafe trajectories despite accurate
perception and prediction, account for 19%. Software and systems failures account for the
remaining 12%.
5. Safety Validation
The automotive industry's traditional approach to safety validation — billions of miles of real-
world testing — is insufficient for AV certification. Rare but critical events occur at frequencies
that make statistical validation through real-world driving intractable: at accident rates of 1 per
10 million miles, demonstrating a 90% confidence interval around that rate requires hundreds of
millions of test miles. Simulation-based testing can accelerate evaluation but raises questions
about simulation-to-reality transfer.
We propose a multi-layered safety validation protocol combining: formal verification for system-
level safety properties, simulation testing for systematic scenario coverage including adversarial
scenarios, closed-course empirical testing, shadow deployment (running AV decisions
alongside human drivers without intervening), and staged public deployment with geographic
and speed constraints progressively relaxed as evidence accumulates.
6. Regulatory and Ethical Dimensions
AV regulation varies dramatically across jurisdictions. The United States has pursued a primarily
permissive federal framework deferring to state regulation, while the European Union has
implemented more prescriptive type-approval requirements. China has established AV testing
zones with systematic performance reporting requirements. The lack of international
harmonization creates challenges for companies seeking global deployment.
Ethical questions around AV decision-making — colloquially framed as 'trolley problems' —
have attracted significant public and academic attention. While such dilemmas are rare in
practice, they highlight deeper questions about how moral responsibility should be allocated
among AV developers, regulators, and operators when algorithmic decisions cause harm.
7. Conclusion
Autonomous vehicles represent a test case for the deployment of complex AI systems in safety-
critical, open-world environments. The technical challenges of perception under distributional
shift, prediction of human behavior, and planning under uncertainty are deeply connected to
broader questions in AI safety and reliability. Progress will require not only algorithmic advances
but the development of rigorous validation methodologies and governance frameworks
commensurate with the stakes involved.
References
Caesar, H., et al. (2020). nuScenes: A multimodal dataset for autonomous driving. CVPR 2020.
Liu, Z., et al. (2023). BEVFusion: Multi-task multi-sensor fusion with unified bird's-eye view
representation. ICRA 2023.
NTSB. (2024). Autonomous vehicle accident investigation report AV-2023-042.