NYC Yellow Taxi Trip Network Analysis using Neo4j

Project Documentation
Team Members
Team Member 1: [Name] - Roll No: [Number]
Team Member 2: [Name] - Roll No: [Number]
Team Member 3: [Name] - Roll No: [Number]
Team Member 4: [Name] - Roll No: [Number]

Guide: [Professor Name]

Department: [Department Name]
Institution: [College/University Name]

Date: [Submission Date]

 Abstract
Urban transportation systems generate massive amounts of data that can reveal valuable insights about
city mobility patterns. This project analyzes New York City's yellow taxi trip data using graph
database technology to understand travel patterns, identify key locations, and discover hidden
relationships in the transportation network.
Traditional databases struggle to represent the interconnected nature of transportation networks. We
chose Neo4j, a graph database, because it naturally models locations as nodes and trips as
relationships, making it ideal for analyzing how people move through the city.
Our analysis addresses several important questions: Which locations are most important in the taxi
network? What are the busiest travel routes? How do travel patterns change throughout the day?
Where do passengers struggle to find available taxis? The answers help taxi companies optimize their
services, city planners improve transportation infrastructure, and drivers maximize their earnings.
 Problem Statement
New York City's taxi system serves millions of passengers daily, but several challenges exist:
Service Optimization Challenge: Taxi companies need to understand where and when demand is
highest to position vehicles efficiently. Without this knowledge, taxis waste time and fuel driving
empty while passengers wait for rides.
Revenue Maximization: Drivers want to know which routes and times offer the best earnings.
Currently, they rely on experience rather than data-driven insights.
Urban Planning Needs: City planners require insights into transportation flow to improve traffic
management, identify underserved areas, and plan infrastructure investments.
Network Complexity: With thousands of pickup and dropoff locations connected by millions of trips,
understanding the overall network structure using traditional analysis methods is nearly impossible.
This project solves these challenges by representing the taxi system as a graph network and applying
advanced algorithms to discover patterns, identify important locations, and optimize operations.
 Dataset Description and Source
Dataset Overview
We analyzed the NYC Yellow Taxi Trip dataset from January 2016, which contains real trip records
collected by the New York City Taxi and Limousine Commission.
 Dataset Name: yellow_tripdata_2016-[Link]
 Source: NYC Taxi and Limousine Commission Open Data Portal
 Time Period: January 2016
 Number of Records: Approximately 10.9 million trips
 The dataset includes comprehensive information about each taxi trip
 Location Information:
1. Pickup coordinates (longitude and latitude)
2. Dropoff coordinates (longitude and latitude)
 Temporal Information:
1. Pickup date and time
2. Dropoff date and time
 Trip Characteristics:
1. Distance traveled in miles
2. Number of passengers
3. Trip duration
 Financial Information:
1. Base fare amount
2. Tip amount
3. Taxes and surcharges
4. Total amount paid
5. Payment method (credit card, cash, etc.)
 Operational Data:
1. Vendor identification
2. Rate code (standard, JFK airport, Newark, etc.)
3. Store and forward flag (indicates if trip was held in memory before sending)
 Data Quality : Before analysis, we cleaned the data by removing invalid records:
1. Trips with missing coordinates
2. Trips with zero distances or fares
3. Records with coordinates of (0, 0) indicating GPS errors
4. Trips with unrealistic values
This cleaning process ensures our analysis reflects actual taxi operations rather than data collection
errors.

Graph Data Model

Why Graph Database?
Traditional relational databases store data in tables, but taxi trips are fundamentally about connections
between places. Graph databases excel at representing and querying these relationships. In our model,
we can instantly find questions like "What are all trips starting from Times Square?" or "Which
location connects the most different areas?" that would require complex joins in SQL.
Node Structure
Location Nodes represent physical places where taxis pick up or drop off passengers.
Each Location node contains:
 Geographic coordinates (longitude and latitude) - uniquely identifies the location
 Auto-generated unique identifier
 Location type classification (Taxi Zone)
 Calculated centrality metrics (added during analysis)
 Community assignment (discovered through algorithm)
We created Location nodes by merging trips with identical coordinates, meaning each unique pickup
or dropoff point appears only once in the graph regardless of how many trips occurred there.
Relationship Structure
TRIP Relationships connect two Location nodes, representing a taxi journey from pickup to dropoff.
Each TRIP relationship stores:
Trip Identification:
 Unique trip identifier
 Vendor who operated the trip
Temporal Attributes:
 Exact pickup timestamp
 Exact dropoff timestamp
 Calculated duration in minutes
 Hour of pickup (0-23)
 Day of week (1=Monday through 7=Sunday)
Spatial Attributes:
 Total distance travelled
Financial Attributes:
 Base fare charged
 Extra charges
 MTA tax
 Tip amount given
 Toll charges
 Improvement surcharge
 Total payment received
 Payment method used
Derived Metrics:
 Average speed during trip
  Tip as percentage of fare
 Rate code applied
 Graph Structure
 The complete graph contains:

Thousands of Location nodes (exact count depends on coordinate precision)

Millions of TRIP relationships connecting these locations
Direction matters: a trip from Location A to Location B is different from B to A
This structure creates a weighted, directed network where locations with many connections represent
important hubs, and heavily-traveled routes appear as multiple relationships between the same two
locations.

Data Creation and Analysis Process

Phase 1: Database Preparation
Before loading data, we established rules to ensure data quality and improve performance:
Uniqueness Constraints: We prevented duplicate locations by requiring each combination of longitude
and latitude to appear only once. This automatically creates an index that speeds up location lookups.
Performance Indexes: We created indexes on frequently-queried attributes like pickup time and fare
amount. Indexes work like book indexes, allowing the database to quickly find relevant records
without scanning everything.
Phase 2: Data Import Strategy
Loading 10 million trips requires careful planning to avoid overwhelming the system:
Batch Processing: Instead of loading all trips at once, we processed them in batches of 10,000. Each
batch completes as a transaction, meaning if something fails, we don't lose all progress.
Data Validation: As each row loads, we check for valid coordinates and positive distances/fares.
Invalid records are skipped automatically.
Merge Strategy: For each trip, we check if pickup and dropoff locations already exist. If yes, we reuse
them; if no, we create new Location nodes. This prevents duplicate locations.
Derived Calculations: While creating trips, we calculate additional useful metrics like trip duration,
average speed, and tip percentage. These calculations happen once during import rather than
repeatedly during analysis.
Phase 3: Descriptive Analytics
We began with basic questions to understand the dataset:
Overall Statistics: We calculated averages and medians for all trip metrics (distance, fare, duration,
speed, passengers). This provides baseline understanding of typical taxi trips.

Network Size: We counted total locations and how many serve as pickups versus dropoffs. Some
locations are predominantly origins (residential areas) while others are mainly destinations (business
districts).
Temporal Patterns: By grouping trips by hour and day of week, we discovered when taxi demand
peaks. Morning and evening rush hours show distinct patterns, as do weekdays versus weekends.
Payment Analysis: We examined payment method distribution and found that credit card users tip
more consistently than cash users (cash tips aren't always recorded).
Distance Distribution: Most trips are short (under 3 miles), but a long tail of airport and outer-borough
trips exists. Different distance ranges may benefit from different pricing strategies.
Phase 4: Spatial Network Analysis
Next, we explored geographic patterns:
Hotspot Identification: We identified locations with the most pickups and dropoffs. Major hubs like
Penn Station, Times Square, and airports dominate.
Popular Routes: By grouping trips by origin-destination pairs, we found the most traveled corridors.
These represent regular commuter routes or connections between major points of interest.
Connection Diversity: We counted how many different destinations can be reached from each
location. Well-connected hubs link many parts of the city, while peripheral locations connect to fewer
places.
Supply-Demand Imbalance: By comparing pickups to dropoffs at each location, we identified areas
where taxis frequently drop off passengers but rarely pick up new ones. These locations may need
better taxi availability.
Phase 5: Advanced Graph Algorithms
We applied sophisticated network analysis algorithms to discover hidden patterns:
PageRank Analysis: This algorithm, originally used by Google to rank web pages, identifies the most
"important" locations. A location has high PageRank if it receives trips from other important
locations. Results highlight central business districts and major transportation hubs.
Degree Centrality: This simply counts connections (trips) for each location. High degree locations are
major hubs with heavy traffic volume.
Betweenness Centrality: This identifies locations that sit "between" many other locations on shortest
paths. These are critical transfer points or bottlenecks in the network. If these locations experience
problems, many routes are affected.
Community Detection: Using the Louvain algorithm, we grouped locations into communities based
on travel patterns. Locations within a community have many trips between them but fewer trips to
other communities. This reveals natural zones like residential neighborhoods, business districts, and
entertainment areas.
Shortest Path Analysis: We calculated optimal routes between locations based on total distance. This
helps understand the most efficient paths through the network.

Phase 6: Behavioral Analysis

We investigated relationships between different variables:
Fare-Distance Correlation: We examined how fare relates to distance across different trip lengths.
This reveals if pricing is consistent or if certain distances offer better value.
Tipping Behavior: By analyzing tips across time periods, we discovered when passengers are most
generous. Evening trips and weekend rides show higher tip percentages.
Traffic Patterns: Speed analysis by hour reveals congestion patterns. Rush hours show significantly
slower speeds, affecting trip duration and driver earnings.
Group Size Impact: We compared trips by passenger count to understand if group size affects trip
characteristics. Solo travelers and groups show different patterns in distance and fare.

Phase 7: Optimization Insights

Finally, we generated actionable recommendations:
Revenue Optimization: We identified routes with the best revenue per minute, helping drivers
maximize earnings. Short, high-fare trips often outperform long, low-fare trips.
Deadheading Reduction: We found routes where return trips are unlikely, meaning drivers often drive
empty after dropoff. Identifying these helps optimize dispatch strategies.
Dynamic Pricing Opportunities: By combining time and location data, we pinpointed when and where
surge pricing would be most effective based on demand spikes.

Graph Data Science Algorithms Applied

PageRank Algorithm
Purpose: Identify the most influential locations in the network.
How it works: PageRank treats trips as votes. A location gains importance not just from receiving
many trips, but from receiving trips from other important locations. The algorithm iteratively updates
each location's score based on incoming connections.
Business Value: High PageRank locations are prime spots for:
 Taxi stands and waiting areas
 Premium pricing zones
 Targeted advertising to drivers
 Service quality monitoring
Results: Major business districts, transportation hubs like Penn Station and Grand Central, and
popular entertainment venues scored highest, confirming they are central to NYC's taxi network.
Degree Centrality
Purpose: Measure direct connectivity of each location.

How it works: Simply counts the number of trips starting or ending at each location. High degree
means many direct connections.
Business Value: High degree locations need:
 Adequate taxi supply
 Efficient passenger pickup/dropoff infrastructure
 Special attention during peak hours
Results: Airports, major train stations, and central Manhattan locations showed the highest degree,
indicating they are major traffic generators.

Betweenness Centrality
Purpose: Find critical transfer points and bottlenecks.
How it works: Calculates how many shortest paths between other locations pass through each
location. High betweenness means the location is a bridge connecting different parts of the network.
Business Value: High betweenness locations are:
  Critical for network flow
 Vulnerable points where disruption affects many routes
 Strategic positions for service optimization
Results: Major crosstown streets and hub stations showed high betweenness, indicating their
importance in connecting different neighborhoods.
Louvain Community Detection
Purpose: Discover natural zones and travel clusters.
How it works: Groups locations into communities where trips within the community are common but
trips between communities are rare. The algorithm maximizes connections inside communities while
minimizing connections between them.
Business Value: Communities reveal:
 Natural service zones for targeted operations
 Geographic markets with distinct characteristics
 Opportunities for zone-based pricing
 Areas for focused marketing campaigns
Results: Communities generally aligned with neighborhoods (Upper East Side, Financial District,
etc.), confirming that taxi trips follow neighborhood boundaries and revealing distinct travel patterns
for each area.
Shortest Path Analysis
Purpose: Optimize route planning and understand network efficiency.
How it works: Uses Dijkstra's algorithm to find the path between two locations that minimizes total
distance, considering actual trip data rather than straight-line distance.
Business Value: Shortest path analysis enables:
 Route optimization for drivers
 Travel time estimation
 Identification of indirect routes (possible traffic avoidance)
 Network efficiency assessment
Results: Many shortest paths differed from direct routes, suggesting drivers navigate around traffic,
one-way streets, or other obstacles not visible in coordinate data alone.
Interpretation of Results
Key Findings
Network Structure: The NYC taxi network exhibits classic hub-and-spoke characteristics. A small
number of locations (less than 5% of all locations) account for the majority of trips. Manhattan,
particularly Midtown and Lower Manhattan, dominates the network with the highest connectivity and
centrality scores.
Temporal Patterns: Clear demand cycles emerge throughout the day and week. Weekday mornings
(7-9 AM) and evenings (5-7 PM) show peak demand, coinciding with commuter patterns. Weekend
patterns differ significantly, with late-night demand (after 10 PM) much higher than weekdays. This
suggests different service strategies should apply to weekdays versus weekends.
Spatial Imbalances: Significant supply-demand imbalances exist. Popular dropoff locations in outer
boroughs show much lower pickup rates, creating "taxi deserts" where passengers struggle to find
rides. Conversely, Midtown Manhattan shows balanced pickup and dropoff activity, indicating good
service availability.
Financial Insights: Average fares correlate strongly with distance, but variation exists. Airport trips
show higher per-mile rates due to flat fares. Tip percentages vary significantly by payment method,
with credit card users tipping more consistently than cash users. Evening and weekend trips show
slightly higher tip percentages, possibly due to leisure travel versus business travel.
Travel Communities: The community detection algorithm revealed distinct travel zones that align
with well-known neighborhoods but also uncovered less obvious patterns. For example, some
residential areas in Brooklyn form communities with specific Manhattan destinations, suggesting
regular commuter flows. Business districts show high internal connectivity, indicating frequent short
trips within the area.
Optimization Opportunities: High-revenue routes concentrated in Manhattan, with trips between
major business districts and hotels showing the best revenue per minute. Significant deadheading
problems exist for trips from Manhattan to outer boroughs, suggesting dispatch algorithms should
prioritize return trips or consecutive trips in the same area.
Business Implications
For Taxi Companies: Understanding network centrality helps optimize fleet positioning. Instead of
distributing taxis evenly, concentrating vehicles near high-PageRank locations during peak hours
maximizes utilization. Community-based dispatching can reduce deadheading by keeping taxis within
their assigned zones.
For Drivers: Revenue optimization analysis provides concrete guidance. Drivers earn more per
minute focusing on short trips in high-demand areas rather than long trips to outer boroughs. Time-of-
day analysis shows when and where to position for maximum earnings.
For City Planners: Betweenness centrality identifies critical infrastructure points. High-betweenness
locations need adequate road capacity and taxi infrastructure. Supply-demand imbalances highlight
areas needing improved taxi service or alternative transportation options.

For Passengers: Understanding demand patterns helps passengers time their trips. Avoiding peak
hours or knowing alternative pickup locations with better taxi availability improves service
experience.
Validation and Reliability
Our findings align with known NYC geography and transportation patterns, providing confidence in
the analysis. For example, the highest-traffic locations match known major destinations, and temporal
patterns match expected commuter behavior. The community structure discovered by algorithms
corresponds to actual neighborhoods, suggesting the analysis captured real-world patterns rather than
statistical artifacts.
Some limitations exist: the data represents only one month, so seasonal patterns aren't visible. Yellow
taxis represent only one segment of NYC transportation, excluding green cabs, ride-sharing services,
and public transit. Weather and special events affect patterns but aren't captured in this dataset.
Conclusion and Future Scope
This project successfully demonstrated how graph database technology and network analysis
algorithms can extract valuable insights from urban transportation data. By representing taxi trips as a
network of connected locations, we uncovered patterns invisible in traditional analysis approaches.
The analysis achieved several key objectives. We identified the most important locations in the taxi
network, discovered natural travel communities that align with neighborhoods, quantified supply-
demand imbalances that affect service quality, and provided concrete recommendations for revenue
optimization and service improvement.
Graph databases proved ideal for this problem because relationships between locations are first-class
entities, making complex network analysis queries simple and efficient. Traditional relational
databases would require complex joins and recursive queries to achieve similar insights.
The methodologies developed here are not limited to taxi data. Any system involving movement
between locations—delivery services, bike sharing, public transit, even disease transmission—can
benefit from similar graph-based analysis.
Future Scope
Temporal Expansion: Analyzing multiple months or years would reveal seasonal patterns, weather
effects, and long-term trends. Combining historical data with real-time feeds would enable predictive
modeling for demand forecasting.
Multi-Modal Integration: Incorporating other transportation modes (subway, bus, bike-share, ride-
sharing) would create a comprehensive urban mobility network. This would reveal how different
modes complement each other and identify gaps in overall transportation coverage.
Machine Learning Integration: Graph neural networks could learn from network structure to predict
trip demand, estimate travel times, or recommend optimal driver positioning. Embedding algorithms
could represent locations as vectors for similarity analysis and clustering.
Real-Time Applications: Connecting the graph database to live taxi data would enable real-time
decision support. Drivers could receive dynamic recommendations for positioning, passengers could
see predicted wait times, and dispatch systems could optimize assignments based on current network
state.
Enhanced Spatial Analysis: Incorporating additional data layers (road networks, traffic signals, points
of interest, demographics) would provide richer context for understanding travel patterns. Spatial
clustering algorithms could identify micro-zones within neighborhoods.
Economic Modeling: Detailed revenue analysis could inform dynamic pricing strategies, driver
incentive structures, and service level agreements. Simulation tools could test policy changes before
implementation.
Sustainability Analysis: Calculating environmental impact (carbon emissions, fuel consumption) and
identifying opportunities for route optimization could support sustainability goals. Analysis could
guide transition to electric vehicle fleets.
Social Equity Assessment: Examining service quality across different neighborhoods and
demographic groups would identify transportation equity issues. This could inform policies ensuring
fair service access for all communities.
Event Impact Analysis: Studying how special events (concerts, sports games, conventions) affect the
network would enable better service planning for predictable demand spikes.
Comparative Analysis: Applying the same methodology to other cities would enable cross-city
comparisons, identification of best practices, and understanding of how different urban structures
affect transportation patterns.
The foundation established in this project provides a robust platform for these future enhancements,
demonstrating the long-term value of graph-based transportation analysis.

References
1. Neo4j Documentation
Neo4j Graph Database Platform. "Neo4j Graph Data Science Library."
[Link]
2. NYC Taxi Data
New York City Taxi and Limousine Commission. "TLC Trip Record Data."\
[Link]
3. Graph Algorithms
Needham, M., & Hodler, A. E. "Graph Algorithms: Practical Examples in Apache Spark and
Neo4j."
O'Reilly Media, 2019.
4. PageRank Algorithm
Page, L., Brin, S., Motwani, R., & Winograd, T. "The PageRank Citation Ranking: Bringing
Order to the Web." Stanford InfoLab Technical Report, 1999.
5. Community Detection
Blondel, V. D., Guillaume, J. L., Lambiotte, R., & Lefebvre, E. "Fast unfolding of
communities in large networks."
Journal of Statistical Mechanics: Theory and Experiment, 2008.
6. Network Analysis
Newman, M. E. J. "Networks: An Introduction."
Oxford University Press, 2010.
7. Urban Mobility
Zheng, Y., Capra, L., Wolfson, O., & Yang, H. "Urban Computing: Concepts,
Methodologies, and Applications."
ACM Transactions on Intelligent Systems and Technology, 2014.
8. Cypher Query Language
Neo4j Inc. "Cypher Query Language Reference."
[Link]
9. Graph Data Science Applications
Robinson, I., Webber, J., & Eifrem, E. "Graph Databases: New Opportunities for Connected
Data." O'Reilly Media, 2015.
10. Transportation Network Analysis
Rodrigue, J. P., Comtois, C., & Slack, B. "The Geography of Transport Systems."
Routledge, 2020.