DATA SCIENCE & BIG DATA

Complete Study Guide

Units I, II & III | Simple. Clear. Memorable.
UNIT I UNIT II UNIT III
Data Science & R ML Algorithms Graphs & Viz
 UNIT I: INTRODUCTION TO DATA SCIENCE & R
PROGRAMMING

1. What is Data Science?

Data Science is an interdisciplinary field that uses scientific methods, algorithms and systems to extract
knowledge and insights from both structured and unstructured data.

KEY POINT: Think of it as: Statistics + Programming + Domain Knowledge = Data Science

The Data Science Lifecycle (6 Phases)

Phase What Happens
Phase 1: Discovery Understand goals, resources & frame the
problem
Phase 2: Data Prep Clean, explore & prepare data (ETLT
process)
Phase 3: Model Planning Decide methods & apply EDA to find
relationships
Phase 4: Model Building Build training/test sets, run algorithms
(clustering, etc.)
Phase 5: Operationalize Pilot project, generate reports, code &
documents
Phase 6: Communicate Share results with stakeholders, determine
success/failure

NB: Over 70% of a data scientist's time is spent on data cleaning and exploration (Phase 3). It's
the most time-consuming step.

Tools for Data Science

R Python SQL Hadoop Tableau Weka

Applications of Data Science

• Healthcare – predicting diseases, drug discovery
• E-commerce – product recommendations
• Transport – self-driving cars, route optimization
• Manufacturing – predictive maintenance
• Conversational AI – chatbots, virtual assistants
2. Big Data
Big Data refers to datasets so large or complex that traditional databases cannot handle them.

The 3 V's of Big Data

VOLUME VELOCITY VARIETY
Amount of Data Speed of Data Types of Data
Facebook: 500TB/day Real-time streaming Text, video, audio, etc.

Types of Big Data

Type Description
Structured Organized, fixed format. E.g., employee table
in a database (name, salary, position)
Unstructured No fixed structure. Hard to process. E.g.,
emails, videos, social media posts
Semi-structured Mix of both. Has some tags/labels but not
fully organized. E.g., JSON, XML

NB: Unstructured data is the most common type in the real world — images, tweets, emails. Deep
Learning excels at handling it.

Underfitting vs. Overfitting

Problem What It Means
Underfitting Model is too simple — misses important
patterns. Like assuming all women will
respond to an ad based only on gender.
Overfitting Model is too specific — memorizes noise in
training data. Like saying only married
women aged 25-27 with one dog will
respond.

KEY POINT: Good fit = model generalizes well. The sweet spot between under and overfitting is
what we aim for.

Data Science vs. Machine Learning vs. AI

Term Simple Definition
Data Science A discipline — uses code + data to build
models that make predictions/explanations
Machine Learning A set of techniques/algorithms that let
computers learn patterns automatically
Deep Learning Subset of ML using multiple hidden layers of
neural networks (e.g., image recognition)
Artificial Intelligence Broad category of systems that operate with
human-like autonomy and scope

NB: Deep Learning is NOT AI. A customer churn model is NOT an AI solution. A self-driving car IS
an AI solution.

Big Data Analysis Techniques

• A/B Testing – compare two versions to see which performs better
• Data Fusion & Integration – combining data from multiple sources
• Data Mining – discovering patterns in large datasets
• Machine Learning – algorithms that learn from data
• Natural Language Processing (NLP) – understanding human language
• Statistics – classical data analysis methods
3. Statistical Inference & Modeling
Inferential statistics lets us make conclusions about a population based on a sample.

Key Concepts
Term Meaning
Population The entire group being studied
Sample A representative subset of the population
Sampling Bias When sample doesn't represent the
population (e.g., comfort bias, size bias)
Standard Error Decreases as sample size increases

Central Limit Theorem (CLT)

CLT says: As sample size increases: (1) Sample mean approaches population mean, (2)
Sample distribution becomes more normal (bell-shaped), (3) Sample variation decreases.

NB: CLT is HUGE in statistics. It's why we can use normal distribution assumptions even when the
population isn't normally distributed.

Hypothesis Testing (4 Steps)

1. State the null (H0) and alternative (H1) hypotheses
2. Choose significance level (alpha, usually 0.05)
3. Compute test statistic (z-score, t-score, F-score)
4. Interpret: if test stat falls in critical region, reject H0

Types of Tests
Test When to Use
Z-test Population mean & SD known
T-test (one sample) Population parameters unknown — uses t-
distribution (wider, thicker tails)
T-test (two sample) Independent (unpaired) or dependent
(paired) samples
Chi-square test Categorical data — goodness of fit or test of
independence
ANOVA Comparing means of 3+ groups. F-value > F-
 critical → reject H0

NB: Chi-square df = (rows-1)(cols-1) for test of independence. For goodness-of-fit: df = categories

- 1.

Correlation & Regression

Method Description
Simple Linear Regression y = B0 + B1*x — one predictor. B0 =
intercept, B1 = slope
Multiple Regression y = B0 + B1x1 + B2x2 + ... — multiple
predictors. Avoid multicollinearity!
Nonlinear Regression Non-linear combination of parameters. For
curved relationships
Pearson Correlation Measures strength/direction of linear
relationship between two variables

NB: Multicollinearity = when two predictors are highly correlated — it destabilizes multiple
regression. Check pairwise correlations first.

4. Probability Distributions
P(E) = Number of ways event can happen / Total possible outcomes. Range: 0 (impossible)
to 1 (certain)

Types of Random Variables

• Discrete – can only take specific values (e.g., number of heads in a coin flip)
• Continuous – can take any value within a range (e.g., height, temperature)

Important Distributions
Distribution Key Feature
Binomial Distribution Fixed n trials, each with probability P of
success. Binary outcomes only (success/fail)
Normal (Gaussian) Distribution Symmetrical bell curve around mean.
Defined by mean (μ) and standard deviation
(σ). Total area = 1
KEY POINT: Bayes' Theorem is the backbone of many ML algorithms (Naive Bayes). It updates
probability when new evidence arrives.

Bayes Theorem: P(class|features) = [P(class) × P(features|class)] / P(features)

5. Introduction to R
R is an open-source statistical programming language created in 1995 by Ross Ihaka and Robert
Gentleman (hence 'R'). It holds 10,000+ packages on CRAN.

Why R for Data Science?

• Best-in-class data visualization (ggplot2, plotly, highcharter)
• Excellent data wrangling tools (dplyr, tidyr, purrr)
• Strong statistical modeling capabilities
• Free and open-source, used by Facebook, Google, Airbnb, Uber, IBM

Key R Packages to Know

ggplot2 dplyr tidyr caret
Visualization Data Wrangling Data Tidying ML Models

NB: R is different from Python in one key way: R was built for statistics first, while Python is a
general-purpose language that added statistical libraries. For pure stats, R is often better.

6. Exploratory Data Analysis (EDA)

EDA is the process of visually and statistically exploring data to understand it before modeling. It's
about finding patterns, anomalies, and verifying assumptions.

Types of Data in EDA

Data Type Description & Use
Categorical (Nominal) No ordering. E.g., gender
(Male/Female/Other). Use: bar chart,
frequency table
Categorical (Ordinal) Has clear ordering. E.g., toaster setting
(Low/Medium/High). Use: bar chart
Numeric (Interval) Equal gaps, no true zero. E.g., temperature
in Celsius. Can add/subtract
Numeric (Ratio) Equal gaps + true zero. E.g., weight. Can
multiply/divide. Most informative scale

NB: Pie charts are widely considered bad visualization. Humans are poor at comparing angles.
Use bar charts instead!
Visualization by Data Type
Chart Type Best For
Box plot Univariate — shows min, max, median,
quartiles, outliers
Histogram Univariate — shows distribution, mode,
skewness, variance
Scatter plot Bivariate — shows relationship between two
numeric variables
Line chart Time series or ordered data — shows trends
over time
Bar chart Categorical data — shows
frequency/comparison of categories

Useful EDA Functions in Python/Pandas

[Link]() First 5 rows | [Link] → rows & columns | [Link]() → count, mean, std, min,
max, quartiles | [Link]().sum() → missing values | [Link] → data types

Binning & Encoding

• Binning (Discretization) – convert numeric to categorical. E.g., age: 0-12=child, 13-19=teen, 20-
65=adult
• Encoding – convert categorical to numeric. E.g., gender: Male=0, Female=1
• One-Hot Encoding – create k binary columns for k categories (increases dimensionality)
 UNIT II: BASIC MACHINE LEARNING ALGORITHMS

1. Linear Regression
Core Idea: Model a linear relationship between input (x) and output (y). Simple: y = B0 +
B1*x | Multiple: y = B0 + B1x1 + B2x2 + ...

Learning Methods
Method When/How to Use
Simple Linear Regression Single input variable. Use statistics (mean,
covariance) to find coefficients
Ordinary Least Squares (OLS) Multiple inputs. Minimizes sum of squared
residuals. Fast but needs all data in memory
Gradient Descent Iterative optimization. Best for very large
datasets that don't fit in memory
Regularization Reduces overfitting by penalizing large
coefficients. LASSO (L1), Ridge (L2)

NB: Linear Regression is NOT suitable for binary outcomes (like spam/not spam) because it
produces continuous values, not 0/1.

Data Prep Tips for Linear Regression

• Ensure linear relationship between X and Y (transform if needed)
• Remove outliers from output variable (y)
• Remove multicollinearity — check pairwise correlations
• Use Gaussian (normal) distribution for better predictions
• Rescale inputs using standardization or normalization

2. K-Nearest Neighbors (KNN)

KNN classifies a new data point by looking at its K nearest neighbors and using majority vote.

How KNN Works (Step by Step)

5. Load training and test data
6. Choose value of K (number of neighbors)
7. For each test point, calculate distance to ALL training points (Euclidean is most common)
8. Sort by distance, pick top K nearest
 9. Assign the class with the most votes among K neighbors

NB: KNN is a lazy learning algorithm — it doesn't train a model, it just memorizes all training data.
This makes prediction slow for large N.

KNN Pros & Cons

PROS CONS
Simple and easy to understand Computationally expensive (stores ALL data)
No training phase needed Slow prediction for large datasets
Works for both classification & regression Sensitive to irrelevant features
Good for non-linear data Struggles in high dimensions (curse of
dimensionality)

KEY POINT: Applications: Loan approval, credit ratings, fraud detection, speech/image
recognition, political voting prediction.

3. K-Means Clustering
An unsupervised algorithm that groups data into K clusters by minimizing the within-cluster variance.

Algorithm: 1. Choose K 2. Randomly pick K centroids 3. Assign each point to nearest

centroid 4. Recalculate centroids 5. Repeat until centroids stop changing

NB: K-Means uses Expectation-Maximization (EM): E-step assigns points to clusters, M-step
recalculates centroids. Standardize data first since it's distance-based!

Applications of K-Means
• Market segmentation — group customers by behavior
• Document clustering — group similar articles
• Image segmentation and compression
• Medical subgroup clustering (e.g., heart attack risk)

4. Filtering Spam
Spam = Unsolicited Commercial Email (UCE). 80% of all spam is sent by fewer than 200 spammers.

Types of Spam Filters

Filter Type How It Works
Header Filters Checks email headers for forgery
Content Filters Scans email body text using fuzzy logic
Bayesian Filters Statistical filtering using Naive Bayes —
learns per-user
Whitelist/Blacklist Accepts only from approved addresses or
blocks known spam

Bayesian Spam Filtering

• Words like 'Viagra' or 'refinance' → high spam probability
• Each word updates the overall spam probability using Bayes Theorem
• Trained per-user — learns your specific patterns
• Weakness: Bayesian Poisoning — spammers insert random innocent words

Why Linear Regression & KNN Fail for Spam

Method Why It Fails for Spam
Linear Regression Produces continuous output, not 0/1. Also:
with 10K emails & 100K word features, the
matrix is not invertible.
K-Nearest Neighbors Curse of dimensionality — in 100,000-
dimensional space, even nearest neighbors
are far away. Distances become
meaningless.

5. Naive Bayes
Core Assumption: All features are INDEPENDENT of each other (hence 'Naive'). Rarely
true in real life, but it still works surprisingly well!

Types of Naive Bayes

Type Best For
Gaussian NB Data follows Gaussian distribution. Most
common.
Multinomial NB Features are counts (e.g., word frequency in
text)
Bernoulli NB Features are binary (0 or 1). Good for bag-of-
words text classification
 NB: Zero Frequency Problem: If a word appears in test data but never in training data, NB assigns
0 probability. Fix with Laplace Smoothing.

Naive Bayes Pros & Cons

PROS CONS
Fast and easy to implement Feature independence rarely holds in
practice
Needs less training data Zero frequency problem
Works for both binary & multi-class Not ideal for continuous data without tuning
Great for real-time prediction & text Probabilities may be less calibrated
classification

6. Data Wrangling
Data Wrangling (aka Data Munging) is the process of transforming raw data into a usable format.

Steps in Data Wrangling

Step What to Do
1. Acquiring Data Collect from reliable sources. Verify: Is it
updated regularly? Can you verify with other
sources?
2. Data Cleaning Fix missing values, remove errors,
standardize formats. Most time-consuming
step!
3. Formatting Convert data into consistent, readable
formats (e.g., date formats, string case)
4. Finding Outliers Use if-not statements, statistics, or visual
tools to detect anomalies and remove them

NB: Data Leakage = when test data bleeds into training data. Leads to artificially inflated accuracy
but terrible real-world performance. Data wrangling helps prevent it.

Text Preprocessing for ML

• Lowercasing, removing punctuation, numbers, whitespace, hyperlinks
• Removing stop words (a, the, about, doing...)
• Word Stemming – chops word endings. 'running' → 'run', 'studying' → 'studi'
• Word Lemmatization – uses dictionary. 'running' → 'run', 'studies' → 'study' (smarter)
 KEY POINT: Difference: Stemmer is faster but rough. Lemmatizer is slower but linguistically
accurate.

Feature Extraction from Text

Method What It Does
CountVectorizer Counts word occurrences. Simple but treats
all words equally
TfidfVectorizer Counts words BUT reduces weight for
common words across all documents
Word Embedding (Word2Vec) Maps words to vectors in multi-dimensional
space. Supports math: King - Man + Woman
= Queen

7. Feature Selection
Feature selection = choosing the most relevant input variables for your model.

Why Feature Selection Matters

• Reduces model complexity → simpler, faster, easier to explain
• Can improve accuracy by removing noise
• Reduces computational cost

3 Types of Feature Selection

Method Description
Filter Methods Use statistical tests to score and rank
features (Chi-square, correlation). Fast but
independent of model
Wrapper Methods Train model with different subsets of
features, pick best combo. Computationally
expensive. Example: RFE
Embedded Methods Feature selection built into model training.
Example: LASSO, Ridge Regression

NB: Feature Selection is DIFFERENT from Dimensionality Reduction. Feature selection removes
variables. Dimensionality reduction (PCA, SVD) creates NEW variables.
8. Model Evaluation Metrics
Accuracy Warning: Accuracy alone is misleading when classes are imbalanced. A model
that labels ALL emails as 'not spam' in a 80/20 dataset gets 80% accuracy — but is
useless!

Confusion Matrix Terms

TP (True Positive) FP (False Positive) FN (False Negative) TN (True Negative)
Predicted +, Actually + Predicted +, Actually - Predicted -, Actually + Predicted -, Actually -

Metric What It Measures

Precision Of all predicted positives, how many were
actually positive? TP / (TP + FP) — 'Don't cry
wolf'
Recall Of all actual positives, how many did we
catch? TP / (TP + FN) — 'Don't miss
anything'

KEY POINT: In spam filtering: Recall matters more — missing a spam is bad. In medical
diagnosis: both matter — false negatives (missed disease) can be fatal.
 UNIT III: SOCIAL NETWORK GRAPHS & DATA
VISUALIZATION

1. Social Networks as Graphs

A social network graph represents relationships between entities (usually people) as nodes connected
by edges.

Graph Basics
Term Meaning
Node (Vertex) Represents an entity (person, account,
company)
Edge Represents a relationship between two
nodes
Degree Number of edges connected to a node
Undirected Graph Relationships go both ways (e.g., Facebook
friends)
Directed Graph Relationships have direction (e.g., Twitter
follows)
k-partite Graph Nodes divided into k groups — no edges
within same group

Examples of Social Network Graphs

• Facebook/Twitter — social connections
• Telephone Networks — call patterns
• Email Networks — communication patterns
• Airport Networks — flight routes
• Collaboration Networks — research co-authorship

2. Graph Clustering & Communities

Graph Clustering Methods
• Direct Community Discovery — identify tightly connected subgroups
• Graph Partitioning — divide graph into parts (minimizing cuts)
• Neighborhood Properties — nodes with many common neighbors are grouped
 NB: A 'community' in a graph = a subgroup of nodes more connected to each other than to the
rest of the network. Think friend groups on Facebook.

3. Data Visualization
Data visualization transforms data into visual formats to communicate insights clearly and quickly.

Why Visualization Matters

Anscombe's Quartet: 4 datasets with nearly identical statistics (mean, SD, correlation) but
look completely different when graphed. Proof that you MUST visualize data — never trust
numbers alone!

5 Types of Big Data Visualization Groups

Type Example Use
Temporal Data over time — line charts, timelines
Network Connections between entities — node-link
diagrams
Hierarchical Parent-child relationships — trees,
dendrograms
Geospatial Location-based data — maps, choropleth
Multidimensional Multiple variables — scatter matrices, parallel
coordinates

Common Visualization Tools

Python R Business Tools
matplotlib, seaborn, plotly, [Link] ggplot2, lattice, R Shiny Tableau, Power BI, Excel

4. Data Science & Ethical Issues

Key Ethics Areas: Data Privacy | Algorithmic Bias | Misinformation | Security of models |
Fairness in predictions | Transparency of AI decisions

NB: AI systems trained on biased data can perpetuate and amplify discrimination. Data scientists
have a responsibility to audit models for fairness.
 QUICK REFERENCE: KEY FORMULAS & DEFINITIONS

Critical Formulas
Concept Formula
Probability P(E) = n(E) / n(T)
Bayes Theorem P(class|features) = [P(class) × P(features|
class)] / P(features)
Linear Regression y = B0 + B1*x1 + B2*x2 + ...
Precision TP / (TP + FP)
Recall TP / (TP + FN)
Chi-square df (independence) (rows - 1) × (columns - 1)
ANOVA: reject H0 if F-value > F-critical

Algorithm Comparison
Algorithm Type Use Case Supervised? Notes
Linear Regression Predicting Yes Assumes linear
Regression continuous values relationship
KNN Classification/Regression Pattern Yes Lazy learner,
recognition, fraud slow on large
data
K-Means Clustering Market No Need to specify
segmentation K upfront
Naive Bayes Classification Spam, text Yes Assumes
classification feature
independence
Logistic Classification Binary outcomes Yes Better than
Regression LinReg for
binary

Good luck, Samuel! You've got this.