0% found this document useful (0 votes)
34 views5 pages

Divvy Bike Data Wrangling Guide

The document outlines a data analysis process using R to wrangle and analyze Divvy bike trip data from Q1 2019 and Q1 2020. It includes steps for data collection, cleaning, and preparation, such as renaming columns for consistency, removing unnecessary data, and calculating ride lengths. The final analysis involves descriptive statistics and visualizations of ride data by user type and day of the week, culminating in exporting the summarized data for further analysis.

Uploaded by

pboss16.pp
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
34 views5 pages

Divvy Bike Data Wrangling Guide

The document outlines a data analysis process using R to wrangle and analyze Divvy bike trip data from Q1 2019 and Q1 2020. It includes steps for data collection, cleaning, and preparation, such as renaming columns for consistency, removing unnecessary data, and calculating ride lengths. The final analysis involves descriptive statistics and visualizations of ride data by user type and day of the week, culminating in exporting the summarized data for further analysis.

Uploaded by

pboss16.pp
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

library(tidyverse) #helps wrangle data

# Use the conflicted package to manage conflicts


library(conflicted)

# Set dplyr::filter and dplyr::lag as the default choices


conflict_prefer("filter", "dplyr")
conflict_prefer("lag", "dplyr")

#=====================
# STEP 1: COLLECT DATA
#=====================
# # Upload Divvy datasets (csv files) here
q1_2019 <- read_csv("Divvy_Trips_2019_Q1.csv")
q1_2020 <- read_csv("Divvy_Trips_2020_Q1.csv")

#====================================================
# STEP 2: WRANGLE DATA AND COMBINE INTO A SINGLE FILE
#====================================================
# Compare column names each of the files
# While the names don't have to be in the same order, they DO need to match perfectly before
we can use a command to join them into one file
colnames(q1_2019)
colnames(q1_2020)

# Rename columns to make them consistent with q1_2020 (as this will be the supposed
going-forward table design for Divvy)

(q1_2019 <- rename(q1_2019


,ride_id = trip_id
,rideable_type = bikeid
,started_at = start_time
,ended_at = end_time
,start_station_name = from_station_name
,start_station_id = from_station_id
,end_station_name = to_station_name
,end_station_id = to_station_id
,member_casual = usertype
))

# Inspect the dataframes and look for incongruencies


str(q1_2019)
str(q1_2020)
# Convert ride_id and rideable_type to character so that they can stack correctly
q1_2019 <- mutate(q1_2019, ride_id = [Link](ride_id)
,rideable_type = [Link](rideable_type))

# Stack individual quarter's data frames into one big data frame
all_trips <- bind_rows(q1_2019, q1_2020)#, q3_2019)#, q4_2019, q1_2020)

# Remove lat, long, birthyear, and gender fields as this data was dropped beginning in 2020
all_trips <- all_trips %>%
select(-c(start_lat, start_lng, end_lat, end_lng, birthyear, gender, "tripduration"))

#======================================================
# STEP 3: CLEAN UP AND ADD DATA TO PREPARE FOR ANALYSIS
#======================================================
# Inspect the new table that has been created
colnames(all_trips) #List of column names
nrow(all_trips) #How many rows are in data frame?
dim(all_trips) #Dimensions of the data frame?
head(all_trips) #See the first 6 rows of data frame. Also tail(all_trips)
str(all_trips) #See list of columns and data types (numeric, character, etc)
summary(all_trips) #Statistical summary of data. Mainly for numerics

# There are a few problems we will need to fix:


# (1) In the "member_casual" column, there are two names for members ("member" and
"Subscriber") and two names for casual riders ("Customer" and "casual"). We will need to
consolidate that from four to two labels.
# (2) The data can only be aggregated at the ride-level, which is too granular. We will want to
add some additional columns of data -- such as day, month, year -- that provide additional
opportunities to aggregate the data.
# (3) We will want to add a calculated field for length of ride since the 2020Q1 data did not have
the "tripduration" column. We will add "ride_length" to the entire dataframe for consistency.
# (4) There are some rides where tripduration shows up as negative, including several hundred
rides where Divvy took bikes out of circulation for Quality Control reasons. We will want to
delete these rides.

# In the "member_casual" column, replace "Subscriber" with "member" and "Customer" with
"casual"
# Before 2020, Divvy used different labels for these two types of riders ... we will want to make
our dataframe consistent with their current nomenclature
# N.B.: "Level" is a special property of a column that is retained even if a subset does not
contain any values from a specific level
# Begin by seeing how many observations fall under each usertype
table(all_trips$member_casual)

# Reassign to the desired values (we will go with the current 2020 labels)
all_trips <- all_trips %>%
mutate(member_casual = recode(member_casual
,"Subscriber" = "member"
,"Customer" = "casual"))

# Check to make sure the proper number of observations were reassigned


table(all_trips$member_casual)

# Add columns that list the date, month, day, and year of each ride
# This will allow us to aggregate ride data for each month, day, or year ... before completing
these operations we could only aggregate at the ride level
# [Link] more on date formats in R found at that link
all_trips$date <- [Link](all_trips$started_at) #The default format is yyyy-mm-dd
all_trips$month <- format([Link](all_trips$date), "%m")
all_trips$day <- format([Link](all_trips$date), "%d")
all_trips$year <- format([Link](all_trips$date), "%Y")
all_trips$day_of_week <- format([Link](all_trips$date), "%A")

# Add a "ride_length" calculation to all_trips (in seconds)


# [Link]
all_trips$ride_length <- difftime(all_trips$ended_at,all_trips$started_at)

# Inspect the structure of the columns


str(all_trips)

# Convert "ride_length" from Factor to numeric so we can run calculations on the data
[Link](all_trips$ride_length)
all_trips$ride_length <- [Link]([Link](all_trips$ride_length))
[Link](all_trips$ride_length)

# Remove "bad" data


# The dataframe includes a few hundred entries when bikes were taken out of docks and
checked for quality by Divvy or ride_length was negative
# We will create a new version of the dataframe (v2) since data is being removed
# [Link]
all_trips_v2 <- all_trips[!(all_trips$start_station_name == "HQ QR" | all_trips$ride_length<0),]

#=====================================
# STEP 4: CONDUCT DESCRIPTIVE ANALYSIS
#=====================================
# Descriptive analysis on ride_length (all figures in seconds)
mean(all_trips_v2$ride_length) #straight average (total ride length / rides)
median(all_trips_v2$ride_length) #midpoint number in the ascending array of ride lengths
max(all_trips_v2$ride_length) #longest ride
min(all_trips_v2$ride_length) #shortest ride

# You can condense the four lines above to one line using summary() on the specific attribute
summary(all_trips_v2$ride_length)

# Compare members and casual users


aggregate(all_trips_v2$ride_length ~ all_trips_v2$member_casual, FUN = mean)
aggregate(all_trips_v2$ride_length ~ all_trips_v2$member_casual, FUN = median)
aggregate(all_trips_v2$ride_length ~ all_trips_v2$member_casual, FUN = max)
aggregate(all_trips_v2$ride_length ~ all_trips_v2$member_casual, FUN = min)

# See the average ride time by each day for members vs casual users
aggregate(all_trips_v2$ride_length ~ all_trips_v2$member_casual + all_trips_v2$day_of_week,
FUN = mean)

# Notice that the days of the week are out of order. Let's fix that.
all_trips_v2$day_of_week <- ordered(all_trips_v2$day_of_week, levels=c("Sunday", "Monday",
"Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"))

# Now, let's run the average ride time by each day for members vs casual users
aggregate(all_trips_v2$ride_length ~ all_trips_v2$member_casual + all_trips_v2$day_of_week,
FUN = mean)

# analyze ridership data by type and weekday


all_trips_v2 %>%
mutate(weekday = wday(started_at, label = TRUE)) %>% #creates weekday field using
wday()
group_by(member_casual, weekday) %>% #groups by usertype and weekday
summarise(number_of_rides = n() #calculates
the number of rides and average duration
,average_duration = mean(ride_length)) %>% # calculates the average
duration
arrange(member_casual, weekday) # sorts

# Let's visualize the number of rides by rider type


all_trips_v2 %>%
mutate(weekday = wday(started_at, label = TRUE)) %>%
group_by(member_casual, weekday) %>%
summarise(number_of_rides = n()
,average_duration = mean(ride_length)) %>%
arrange(member_casual, weekday) %>%
ggplot(aes(x = weekday, y = number_of_rides, fill = member_casual)) +
geom_col(position = "dodge")

# Let's create a visualization for average duration


all_trips_v2 %>%
mutate(weekday = wday(started_at, label = TRUE)) %>%
group_by(member_casual, weekday) %>%
summarise(number_of_rides = n()
,average_duration = mean(ride_length)) %>%
arrange(member_casual, weekday) %>%
ggplot(aes(x = weekday, y = average_duration, fill = member_casual)) +
geom_col(position = "dodge")

#=================================================
# STEP 5: EXPORT SUMMARY FILE FOR FURTHER ANALYSIS
#=================================================
# Create a csv file that we will visualize in Excel, Tableau, or my presentation software
# N.B.: This file location is for a Mac. If you are working on a PC, change the file location
accordingly (most likely "C:\Users\YOUR_USERNAME\Desktop\...") to export the data. You can
read more here: [Link]
counts <- aggregate(all_trips_v2$ride_length ~ all_trips_v2$member_casual +
all_trips_v2$day_of_week, FUN = mean)
[Link](counts, file = 'avg_ride_length.csv')

Common questions

Powered by AI

The data analysis techniques ensure validity and reliability by employing rigorous data cleaning and standardization protocols, such as removing erroneous data, ensuring uniform column types, and standardizing categorical labels. The calculated fields and aggregations are carefully structured to reflect actual usage patterns, allowing for meaningful comparisons and insights . By consistently applying statistical summaries and checking for correct assignment of labels, the analyses prioritize accuracy and replicability, which are essential for reliable outcome interpretation . Furthermore, using ordered factors for weekdays corrects data ordering issues, contributing to a more accurate temporal analysis .

The methodologies used for aggregating ride data include grouping data by user type and weekday, then summarizing metrics such as ride count and average ride length . This approach allows for detailed analysis of user behavior patterns by time, facilitating insights into peak usage times and patterns distinguishing between members and casual riders . Additionally, transforming date fields into day, month, and year components enables more granular temporal analysis, allowing for seasonal or periodic trend identification .

Key insights from comparing ride lengths by member type include understanding usage patterns between members and casual riders. Members typically exhibit different ride behavior, such as shorter average ride times, which may indicate regular commuting usage, whereas casual riders might have longer, more leisure-focused rides . These insights can inform strategic decisions, such as targeted marketing efforts or operational adjustments. For instance, optimizing bike availability during peak times for members or enhancing tourist engagement strategies for casual riders can enhance service efficiency and customer satisfaction .

The preparation of Divvy datasets involves multiple critical steps: 1. **Collect Data**: Import datasets from different quarters (e.g., 2019 Q1, 2020 Q1). This initial step ensures the data is available for further processing . 2. **Wrangle Data and Combine into a Single File**: Standardize column names across datasets to ensure consistent format, allowing them to be combined. This avoids errors that could emerge from mismatched data schema . 3. **Clean Up and Add Data**: Convert necessary columns to appropriate data types, ensure consistency in categorical data (e.g., renaming 'Subscriber' to 'member' and 'Customer' to 'casual'), and add calculated fields like 'ride_length'. This ensures the data is tidy and ready for analysis . 4. **Remove Inconsistent Data**: Filter out erroneous data entries, such as negative ride lengths or operational checks. This step ensures the analysis is based on accurate datasets . By following these steps, the data is clean, consistent, and ready for detailed analysis, reducing biases and inaccuracies in subsequent interpretation .

Renaming 'Subscriber' to 'member' and 'Customer' to 'casual' impacts the analysis by ensuring consistency in categorical data, which is crucial for accurate aggregations and comparisons. Without this standardization, aggregation functions and grouping in data analyses might yield incorrect results or could fail to reflect the true usage patterns, as mismatched labels would be considered as distinct categories even if they represent the same concept . This step also aligns the data with current nomenclature, facilitating easier integration with more recent datasets .

Removing entries with negative ride lengths is necessary because they represent errors or specific cases where bikes were taken out for operational checks rather than genuine rides . Including these erroneous data could skew descriptive statistics and analysis, such as average ride length and aggregate user behavior. By eliminating such anomalies, the dataset more accurately reflects actual user interactions, leading to more valid and reliable insights .

Challenges in manipulating date fields include format inconsistencies and the computational complexity of extracting components like day, month, and year . These are addressed by first ensuring dates are converted into a standard format (yyyy-mm-dd) and then using R functions such as `format()` to systematically extract the desired components . These transformations allow for accurate temporal analysis and aggregation without loss of data integrity .

Setting column data types correctly before binding datasets ensures data consistency and integrity across different data files . If columns are not standardized, it can result in errors or loss of information during the merge process. For instance, converting 'ride_id' and 'rideable_type' to character ensures these identifiers retain their uniqueness across datasets, which is crucial for subsequent analyses . Proper data typing also facilitates accurate transformations and computations within datasets .

Visualizing ride data by member type and weekday enhances understanding by clearly illustrating patterns or trends not easily visible in raw data tables . By representing data graphically, such as through bar charts, it is easier to identify peak usage periods, compare ride frequencies across different days for members versus casual users, and convey insights at a glance . This form of visualization facilitates more intuitive analysis and communication of findings to stakeholders, aiding decision-making processes .

The 'dplyr' package plays a central role in the data wrangling process by providing functions to manipulate data frames efficiently, such as filtering rows, selecting columns, and summarizing data . 'Conflicted' is used to manage function conflicts, ensuring that the correct version of a function, like 'filter' from 'dplyr', is utilized when there are multiple packages loaded that define the same function . This aids in maintaining a smooth workflow and avoiding unintended errors or computations that could arise from using incorrect functions .

You might also like