Emmanuel_Project
November 29, 2025
[16]: import pandas as pd
import numpy as np
import [Link] as plt
from [Link] import curve_fit
import os
from pathlib import Path
import warnings
[Link]('ignore')
# PDF generation package
try:
from fpdf import FPDF
PDF_AVAILABLE = True
except ImportError:
print("FPDF not available. Install with: pip install fpdf")
PDF_AVAILABLE = False
# Setup for publication-quality plots FIRST
[Link]({
'[Link]': 12,
'[Link]': 14,
'[Link]': 12,
'[Link]': 10,
'[Link]': 10,
'[Link]': (10, 6),
'[Link]': 300,
'[Link]': 300,
'[Link]': 'tight',
'savefig.pad_inches': 0.1
})
# Set matplotlib style
[Link]('default')
# Use ONE consistent color palette
colors = ['#1f77b4', '#ff7f0e', '#2ca02c', '#d62728', '#9467bd', '#8c564b']
1
# Base path
base_path = '/home/emmanuel-adjei/Documents/CEE_PhD/CEE_PhD/Fall 2025/226A/'
# Dataset configs
datasets = {
'data1': {'num_subsets': 28, 'frame_dt': 1/15},
'data2': {'num_subsets': 13, 'frame_dt': 0.1},
'data3': {'num_subsets': 13, 'frame_dt': 0.1},
'data4': {'num_subsets': 13, 'frame_dt': 0.1}
}
# Constants
L_ft = 900.0
L_mile = L_ft / 5280
T_sec = 180.0
T_hour = T_sec / 3600
lanes = [1, 2, 3, 4, 5, 6]
regular_lanes = [2, 3, 4, 5, 6]
all_lanes = lanes
num_regular = len(regular_lanes)
num_all = len(all_lanes)
print("Libraries imported successfully!")
print(f"Base path: {base_path}")
print(f"Number of datasets: {len(datasets)}")
print(f"Total subsets: {sum(config['num_subsets'] for config in datasets.
↪values())}")
Libraries imported successfully!
Base path: /home/emmanuel-adjei/Documents/CEE_PhD/CEE_PhD/Fall 2025/226A/
Number of datasets: 4
Total subsets: 67
def create_introduction(): latex_text = dedent(r” “ ”
TRAFFIC FLOW ANALYSIS USING NGSIM I-80 DATA:
FUNDAMENTAL DIAGRAM ESTIMATION AND VALIDA-
TION
\section{Introduction}
\subsection{Background and Motivation}
Traffic flow theory represents a cornerstone of transportation engineering,
providing fundamental insights into the dynamics of vehicular movement on
transportation networks. The relationship between traffic flow variables
(density, speed, and flow rate) forms the basis for understanding capacity,
2
congestion patterns, and network performance \cite{daganzo1997}.
The fundamental diagram, first conceptualized by Greenshields in 1935
\cite{greenshields1935}, establishes the empirical relationship between
traffic density ($k$) and flow rate ($q$). This foundation is essential to
traffic flow models and intelligent transportation systems \cite{treiber2013}.
\subsection{NGSIM Dataset}
The NGSIM program provides high-resolution trajectory data widely used in
traffic research \cite{ngsim2006}. The I-80 dataset collected in Emeryville,
California contains detailed temporal and spatial vehicle observations.
\subsection{Research Objectives}
This study addresses two primary goals:
\begin{itemize}
\item Implement Edie’s generalized traffic flow definitions to compute
density, speed, and flow rate.
\item Estimate and validate the triangular fundamental diagram using
empirical freeway observations.
\end{itemize}
\subsection{Mathematical Foundation}
The triangular fundamental diagram is expressed as:
\[
q(k) = \min\{u k,\; w(\kappa - k)\},
\]
where $q$ = flow rate, $k$ = density, $u$ = free-flow speed,
$w$ = backward wave speed, and $\kappa$ = jam density.
The critical density and capacity are given by:
\[
\kappa_c = \frac{w\kappa}{u+w}, \qquad
C = \frac{uw\kappa}{u+w}.
\]
\subsection{Methodological Approach}
Using Edie’s definitions \cite{edie1963}, traffic variables over a region
$\Omega = [t, t+\Delta t] \times [x, x+\Delta x]$ are:
\[
3
k = \frac{\sum t_i}{A}, \quad
q = \frac{\sum d_i}{A}, \quad
v = \frac{\sum d_i}{\sum t_i},
\]
where $t_i$ is time spent, $d_i$ is distance traveled, and $A$ is the
time–space area.
\begin{thebibliography}{9}
\bibitem{daganzo1997} Daganzo, C. F. (1997). \textit{Fundamentals of transportation and traffic
\bibitem{greenshields1935} Greenshields, B. D. (1935). A study of highway traffic.
\bibitem{treiber2013} Treiber, M., \& Kesting, A. (2013). \textit{Traffic flow dynamics.}
\bibitem{ngsim2006} FHWA (2006). NGSIM Project Overview.
\bibitem{newell1993} Newell, G. F. (1993). A simplified theory of kinematic waves.
\bibitem{edie1963} Edie, L. C. (1963). Discussion of traffic stream measurements.
\end{thebibliography}
""")
print(latex_text)
[4]:
3. DATA CONTENT ANALYSIS:
------------------------------
data1 Analysis:
--------------------
Basic Statistics:
Frame_ID Pos Veh_Class Lane_ID
count 0.0 0.0 0.0 0.0
mean NaN NaN NaN NaN
std NaN NaN NaN NaN
min NaN NaN NaN NaN
25% NaN NaN NaN NaN
50% NaN NaN NaN NaN
75% NaN NaN NaN NaN
max NaN NaN NaN NaN
Missing Values:
Veh_ID 0
Frame_ID 79040
Pos 79040
Veh_Class 79040
Lane_ID 79040
dtype: int64
Unique Values:
4
Vehicles: 79040
Frames: 0
Vehicle Classes: [np.float64(nan)]
Lanes: [np.float64(nan)]
Data Ranges:
Position: nan to nan ft
Frames: nan to nan
Estimated duration: nan seconds (nan minutes)
Frame rate: 15.0 Hz
data2 Analysis:
--------------------
Basic Statistics:
Frame_ID Pos Veh_Class Lane_ID
count 0.0 0.0 0.0 0.0
mean NaN NaN NaN NaN
std NaN NaN NaN NaN
min NaN NaN NaN NaN
25% NaN NaN NaN NaN
50% NaN NaN NaN NaN
75% NaN NaN NaN NaN
max NaN NaN NaN NaN
Missing Values:
Veh_ID 0
Frame_ID 127313
Pos 127313
Veh_Class 127313
Lane_ID 127313
dtype: int64
Unique Values:
Vehicles: 127313
Frames: 0
Vehicle Classes: [np.float64(nan)]
Lanes: [np.float64(nan)]
Data Ranges:
Position: nan to nan ft
Frames: nan to nan
Estimated duration: nan seconds (nan minutes)
Frame rate: 10.0 Hz
data3 Analysis:
--------------------
Basic Statistics:
Frame_ID Pos Veh_Class Lane_ID
5
count 0.0 0.0 0.0 0.0
mean NaN NaN NaN NaN
std NaN NaN NaN NaN
min NaN NaN NaN NaN
25% NaN NaN NaN NaN
50% NaN NaN NaN NaN
75% NaN NaN NaN NaN
max NaN NaN NaN NaN
Missing Values:
Veh_ID 0
Frame_ID 141163
Pos 141163
Veh_Class 141163
Lane_ID 141163
dtype: int64
Unique Values:
Vehicles: 141163
Frames: 0
Vehicle Classes: [np.float64(nan)]
Lanes: [np.float64(nan)]
Data Ranges:
Position: nan to nan ft
Frames: nan to nan
Estimated duration: nan seconds (nan minutes)
Frame rate: 10.0 Hz
data4 Analysis:
--------------------
Basic Statistics:
Frame_ID Pos Veh_Class Lane_ID
count 0.0 0.0 0.0 0.0
mean NaN NaN NaN NaN
std NaN NaN NaN NaN
min NaN NaN NaN NaN
25% NaN NaN NaN NaN
50% NaN NaN NaN NaN
75% NaN NaN NaN NaN
max NaN NaN NaN NaN
Missing Values:
Veh_ID 0
Frame_ID 153778
Pos 153778
Veh_Class 153778
Lane_ID 153778
6
dtype: int64
Unique Values:
Vehicles: 153778
Frames: 0
Vehicle Classes: [np.float64(nan)]
Lanes: [np.float64(nan)]
Data Ranges:
Position: nan to nan ft
Frames: nan to nan
Estimated duration: nan seconds (nan minutes)
Frame rate: 10.0 Hz
[6]:
quality_report not found. Creating a basic quality assessment…
5. DATA PROCESSING STRATEGY:
========================================
KEY INSIGHTS FROM DATA ANALYSIS:
-----------------------------------
• data1: nan vehicles/frame on average
• data1: nans duration, 15.0Hz sampling
• data1: 0 lanes covered
• data2: nan vehicles/frame on average
• data2: nans duration, 10.0Hz sampling
• data2: 0 lanes covered
• data3: nan vehicles/frame on average
• data3: nans duration, 10.0Hz sampling
• data3: 0 lanes covered
• data4: nan vehicles/frame on average
• data4: nans duration, 10.0Hz sampling
• data4: 0 lanes covered
PROCESSING RECOMMENDATIONS:
------------------------------
1. DATA CLEANING:
• Filter positions to [0, 900] ft range
• Remove vehicles with unrealistic speeds (>100 mph)
• Handle missing frames in temporal analysis
• Verify lane IDs are in range 1-6
2. TRAJECTORY PROCESSING:
• Use frame-based time steps (1/15s for data1, 0.1s for others)
• Implement robust boundary handling for Edie's method
• Consider vehicle type (cars vs trucks) in analysis
7
• Handle lane-changing vehicles appropriately
3. FLOW CALCULATION (EDIE'S METHOD):
• Use 3-minute observation windows (T=180s)
• Calculate time spent (TTS) and distance traveled (TDS)
• Apply to individual lanes and aggregations
• Convert units: ft → miles, seconds → hours
4. QUALITY CONTROL:
• Validate calculated densities against physical limits
• Check for consistency across datasets
• Remove outliers in fundamental diagram fitting
• Verify capacity values are realistic (1500-2400 vphpl)
5. FUNDAMENTAL DIAGRAM FITTING:
• Use triangular FD for its physical interpretability
• Fit parameters: free-flow speed (u), wave speed (w), jam density (�)
• Validate with R² and residual analysis
• Compare with Highway Capacity Manual values
EXPECTED CHALLENGES AND SOLUTIONS:
-----------------------------------
CHALLENGE: Mixed vehicle types affecting flow characteristics
SOLUTION: Analyze separately or use weighted averages
CHALLENGE: Lane-changing vehicles in flow calculations
SOLUTION: Use lane-based analysis with proper attribution
CHALLENGE: Different time resolutions across datasets
SOLUTION: Process each dataset with correct frame_dt parameter
CHALLENGE: Boundary effects at segment ends
SOLUTION: Implement robust entry/exit detection in Edie's method
CHALLENGE: Data gaps or inconsistencies
SOLUTION: Implement validation checks and filtering
[7]:
6. IMPLEMENTATION PLAN FOR PROBLEMS 2.4 & 3.8:
=======================================================
PROBLEM 2.4: EDI'S FORMULA CALCULATION
----------------------------------------
STEP 1: Data Loading and Validation
• Load all 67 subset files across 4 datasets
• Validate data structure and quality for each file
8
• Apply position filtering (0-900 ft range)
STEP 2: Edie's Formula Implementation
• For each 3-minute subset:
- Calculate Time-Spent (TTS) for each vehicle in segment
- Calculate Distance-Traveled (TDS) for each vehicle
- Use correct frame_dt for each dataset
STEP 3: Metric Calculation
• Density (k) = TTS_hours / (T_hours × L_miles × num_lanes)
• Flow (q) = TDS_miles / (T_hours × L_miles × num_lanes)
• Speed (v) = TDS_miles / TTS_hours
• Units: vpm, vph, mph
STEP 4: Aggregation Levels
• Individual lanes (1-6)
• Regular lanes (2-6)
• All lanes (1-6)
STEP 5: Results Compilation
• Create comprehensive results DataFrame
• Include dataset, subset, and aggregation identifiers
• Export for Problem 3.8 analysis
PROBLEM 3.8: FUNDAMENTAL DIAGRAM FITTING
----------------------------------------
STEP 1: Data Preparation
• Use results from Problem 2.4
• Focus on 'all lanes' aggregation for main analysis
• Filter unrealistic values (k > 200 vpm, v > 80 mph, etc.)
STEP 2: Triangular Fundamental Diagram Fitting
• Model: q = min(u×k, w×(� - k))
• Parameters to estimate:
- u: free-flow speed (mph)
- w: backward wave speed (mph)
- �: jam density (vpm)
• Use curve_fit with appropriate bounds
STEP 3: Derived Parameters
• Critical density: �_c = (w × �) / (u + w)
• Capacity: C = (u × w × �) / (u + w)
• Critical speed: v_c = C / �_c
STEP 4: Model Validation
• Calculate R², RMSE, MAE
• Analyze residuals for patterns
• Compare with empirical data ranges
9
STEP 5: Visualization and Reporting
• Create flow-density, speed-density, speed-flow plots
• Overlay fitted triangular FD
• Report parameters with confidence intervals
• Compare with standard values (HCM, literature)
[ ]:
[ ]:
10