Exploratory Data Analysis (EDA)
Definition
EDA is an iterative cycle used to generate and refine questions and preliminary models as understanding
improves with each iteration.
Purpose
- Identify issues with the dataset
- Assess whether the data can answer the research question
- Provide preliminary insights to guide further analysis
Key idea: Simple graphs rapidly convey information.
Core Goals of EDA
• Detect data issues
• Understand variables (mean, median, mode, range, etc.)
• Evaluate whether existing data is sufficient or more data is required
EDA Checklist
1. Formulate the question
2. Read the data
3. Check packaging (format, types, structure)
4. Inspect top and bottom of the data
5. Check dimensions (rows, columns)
6. Try an easy solution
7. Challenge the results
8. Repeat if necessary
Tidy Data Principles
A dataset is tidy if:
• Each variable is in its own column
• Each observation is in its own row
• Each value is in its own cell
1
Descriptive Statistics
• Mean
• Median
• Empirical quantiles
• Five-number summary
• Range
• Interquartile range (IQR)
• Sample variance
Choosing a Graph or Plot
Ask two key questions:
1. Within-variable variation
Used to understand the distribution of a single variable.
• Boxplots
• Bar plots
• Histograms
• Density plots
2. Between-variable covariation
Used to examine relationships between variables.
• Scatterplots
• Grouped boxplots
Graphs and Plots
Univariate Plots (Variation Within a Variable)
• Histogram: Uses bars to show distribution and descriptive statistics
• Density plot: Uses a smooth curve to show distribution
• Boxplot:
• Q1 (25th percentile): lower edge of box
• Q2 (median): line inside box
• Q3 (75th percentile): upper edge of box
• Whiskers: minimum and maximum values
• Bar plot: Counts observations in categories
2
Key comparisons: - Histogram vs Density: bars vs smooth curve - Histogram vs Bar plot: values vs
categories
Bivariate Plots (Covariation Between Variables)
• Scatterplot: Relationship between two continuous variables
• Grouped boxplots: Numeric variable across categories or subcategories
• Facet histograms: Grid of histograms for subsets of data
Categorical Data
• Count plot
Time Series Data
• Line plot
Relational Data
Definition
Relational data consists of multiple related tables that must be combined using a key or identifier.
Keys
• Primary key: Uniquely identifies an observation within a table
• Foreign key: References a primary key in another table
Types of Joins
• Inner join
Keeps rows where keys match in both tables
• Left join
Keeps all rows from the left table, matching rows from the right table; unmatched rows are filled
with NA
• Right join
Same as left join, but the right table is the base
3
• Full join
Keeps all rows from both tables, filling missing values with NA
• Semi join
Returns rows from the first table that have a match in the second table
• Anti join
Returns rows from the first table that do not have a match in the second table
Missing Data
Consequences of Ignoring Missing Values
• Loss of information
• Biased or distorted results
• Altered conclusions
• Misrepresentation of the population
Types of Missing Data
• MCAR (Missing Completely At Random)
Missingness unrelated to data → deletion is acceptable
• MAR (Missing At Random)
Missingness depends on observed variables → model-based imputation
• MNAR (Missing Not At Random)
Missingness depends on unobserved variables → deterministic imputation
Imputation Methods
Deterministic Imputation
• Fixed rule (e.g. mean substitution)
• Fast and simple
• May bias relationships and underestimate variance
Model-based Imputation
• Predicts missing values using a model
• Preserves relationships and uncertainty
• Requires modelling assumptions and may be misspecified
4
Missing Data Workarounds
• Diagnose: Identify patterns and causes of missingness
• Represent tidily: Use shadow variables to mark imputed values
• Strategy: Drop observations with missing values if only a few exist
Text Analysis
Purpose
Quantify and analyse information from text data.
Key Concepts
• Tidy text: Each text unit is an observation and remains identifiable
• Token: A unit of text (word, sentence, character, etc.)
• Tokenisation: Splitting text into tokens
• Stop words: Common words with little analytical value
Sentiment Analysis
• Assign sentiment scores to tokens
• Aggregate scores to obtain overall sentiment
Word Importance (TF–IDF)
• Term frequency – inverse document frequency
• High values: informative terms
• Low values: ubiquitous/common terms
Cluster Analysis
Purpose
Supports EDA by identifying groups and suggesting segment-specific patterns.
Use clusters as ideas, not final answers. Validate externally when possible.
5
Key Limitations
• Results depend on chosen settings
• Sensitive to outliers
• Sensitive to mixed variable types
• Assumes certain cluster shapes
Hierarchical Clustering
• Builds clusters using a dendrogram
• Cutting the tree at a height determines the number of clusters
Steps: 1. Choose distance metric 2. Choose linkage method 3. Compute distance matrix 4. Agglomerate to
form dendrogram 5. Choose cut height
Choosing the Number of Clusters
k-means
• Use diagnostics such as:
• Elbow plot (within-cluster sum of squares vs k)
• Stability across random starts
Hierarchical
• Look for large jumps in dendrogram height
• Long branches suggest meaningful separation
Pros and Cons
• k-means:
• Fast and simple
• Requires k in advance
• Sensitive to initialisation
• Hierarchical:
• No need to pre-specify k
• Deterministic
• Computationally heavier
• Sensitive to linkage and distance choice
6
Modelling
Linear Model
A linear model has the form:
Y = Xβ + ε
• ε: error term (difference between observed and predicted values)
• ε ~ N(0, σ²)
Error Term
Represents unobserved or omitted factors and quantifies uncertainty.
Assumptions:
• Zero mean
• Constant variance (homoskedasticity)
• No serial correlation
• Normality
Violations include:
• Heteroskedasticity
• Nonlinearity
• Serial correlation
• Non-normal residuals
Model Validation
Check residuals to assess assumptions.
Diagnostic Plots
• Residual vs Fitted plot
• Index plot (Residuals vs observations)
• Should fluctuate randomly around zero with constant variance
• Q–Q plot
7
• Residual histogram
• Straight line and normal shape indicate assumption compliance
Interpretation
• For every one-unit increase in xⱼ, y changes by βⱼ on average
Model Fit and Comparison
• R²: Proportion of variance explained
• Adjusted R²: Penalises unnecessary predictors
• AIC / BIC: Used for model comparison
Dummy Variables
• Encode categorical (factor) variables
• One category is used as the base dummy
R Packages
EDA
• tidyverse (dplyr, tidyr, readr)
Visualisation
• ggplot2
• ggrepel
Dates and Times
• lubridate
• rwalkr
Relational Data
• dplyr
8
Missing Data
• naniar
• simputation
Text Analysis
• tidytext
Cluster Analysis
• stats
Modelling
• stats
• broom