data
Comprehensive Course Notes: Advanced Foundations of Data
Science and Information Theory
1. Introduction to Data and Information Science
In the modern academic and professional landscape, data is defined as a collection of facts,
observations, or measurements represented as numbers, characters, or symbols. While often
used interchangeably with "information," in a technical context, data is the raw input that
becomes information only after it has been processed and interpreted.
1.1 The Data Hierarchy (DIKW Pyramid) - Deep Analysis
The DIKW pyramid represents the structural and functional relationships between data,
information, knowledge, and wisdom. Each layer adds a level of organization and human
interpretation:
• Data: Raw symbols, signals, or stimuli. It lacks context and inherent meaning.
Example: A sensor output reading "102.4".
• Information: Data that has been cleaned, processed, and contextually localized to
answer questions like "who," "what," "where," and "when." Example: "The
temperature of Processor A is 102.4°C."
• Knowledge: The synthesis of information over time, leading to patterns. It answers the
"how" question. Example: "Historically, Processor A malfunctions when
exceeding 100°C for more than 5 minutes."
• Wisdom: The pinnacle of the hierarchy, involving judgment, ethics, and long-term
foresight. It answers the "why" question. Example: "We must install a secondary
cooling system or move the workload to a different server to prevent hardware failure."
2. Expanded Classifications of Data
Understanding the nature of data is critical for selecting the correct statistical methods and
algorithms. Beyond the basic qualitative/quantitative split, we must examine Dimensionality
and Cardinality.
2.1 Qualitative (Categorical) Data Details
Qualitative data describes qualities or characteristics. It cannot be measured numerically but
can be categorized.
• Nominal Data: Labels without any inherent numerical value or order. Case
Study: In a clinical trial, "Blood Type" (A, B, AB, O) is nominal. You cannot say Type A
is "greater" than Type B.
• Ordinal Data: Data with a set order or scale, but the intervals are inconsistent.
Example: Socioeconomic status (Low, Middle, High). While we know "High" is more
than "Middle," the gap between them isn't mathematically quantifiable in the same way
10cm is from 20cm.
2.2 Quantitative (Numerical) Data and Distribution
Quantitative data represents measurable quantities. The distribution of this data determines
which statistical tests (parametric vs. non-parametric) are applicable.
Deep Dive: Continuous vs. Discrete
A "Number of Children" (Discrete) can never be 2.4. However, "Average Children per
Household" (Continuous) can be 2.4 because it is a calculated ratio. Understanding the
difference prevents logic errors in model building.
2.3 Levels of Measurement (Stevens' Typology Expanded)
Level
Type
Mathematical Operations
Examples
Nominal
Qualitative
Counting (Mode), Frequency
Marital status, Programming languages
Ordinal
Qualitative
Ranking, Median, Percentiles
LinkedIn seniority level, Scoville heat units
Interval
Quantitative
Addition, Subtraction, Mean
Calendar years, IQ Scores
Ratio
Quantitative
Multiplication, Division, Absolute Zero
Revenue, Stock price, Molecular weight
3. Data Structures and Big Data Architecture
Modern data pipelines must handle varied formats at scale.
3.1 Structured, Semi-Structured, and Unstructured Data
• Structured Data: Highly organized, following a schema. Tools: MySQL,
PostgreSQL, Oracle.
• Semi-Structured Data: Self-describing data. Technical Example: A JSON
object: {"id": 1, "tags": ["data", "science"]}.
• Unstructured Data: The most prevalent form. Analysis Challenge: Requires
Natural Language Processing (NLP) or Computer Vision to extract features.
3.2 The 5 V's of Big Data: An Analytical View
• Volume: Zettabytes of data. Requires distributed storage like HDFS.
• Velocity: Stream processing (e.g., Apache Kafka) vs. Batch processing.
• Variety: Handling logs, social media video, and sensor data simultaneously.
• Veracity: Dealing with "Noise." If 10% of your sensors are malfunctioning, how does
that skew the result?
• Value: The ROI. Data without an actionable business outcome is just a storage cost.
4. Statistical Fundamentals and Probability Distributions
4.1 Advanced Central Tendency
While the Mean is common, it is sensitive to Outliers. Formula for Mean: $\bar{x} = \
frac{\sum_{i=1}^{n} x_i}{n}$
In skewed distributions (like wealth distribution), the Median provides a more realistic
"middle" than the Mean.
4.2 Measures of Dispersion and the Normal Distribution
Dispersion quantifies the risk or uncertainty in a dataset.
• Standard Deviation ($\sigma$): The square root of Variance. In a Normal Distribution:
68% of data falls within 1$\sigma$. 95% of data falls within 2$\sigma$.
99.7% of data falls within 3$\sigma$ (The Empirical Rule).
Practice Problem 1:
A dataset contains the following values: [10, 12, 10, 15, 90]. 1. Calculate the Mean.
2. Calculate the Median. 3. Which measure best represents the "typical" value in this
set? (Hint: Notice the outlier 90).
5. The Data Lifecycle and ETL Processes
Data moves through a pipeline known as ETL (Extract, Transform, Load).
• Collection: Scraping, APIs, or direct database queries.
• Wrangling (Transformation): Imputation: Filling missing values with
the Mean/Median. Normalization: Scaling values to a range [0, 1].
• Exploratory Data Analysis (EDA): Using summary statistics and plots to find hidden
patterns before formal modeling.
6. New Module: Machine Learning Paradigms
Data Science is often applied through Machine Learning (ML), which is categorized into
three main types:
6.1 Supervised Learning
The model is trained on labeled data. Tasks: Regression (predicting price), Classification
(Spam vs. Not Spam).
6.2 Unsupervised Learning
The model finds hidden structures in unlabeled data. Tasks: Clustering (segmenting
customers), Dimensionality Reduction (PCA).
6.3 Reinforcement Learning
Learning through trial and error using rewards and penalties. Example: Training a robot to
navigate a room or an AI to play Chess.
7. New Module: Data Warehousing and SQL
To analyze data, one must be able to retrieve it. Structured Query Language (SQL) is the
industry standard.
Sample SQL Query:
SELECT user_name, purchase_amount
FROM sales_data
WHERE purchase_amount > 1000
ORDER BY purchase_amount DESC;
8. Data Ethics, Privacy, and Security
With great data comes great responsibility. The ethical landscape is governed by several
pillars:
• Transparency: Users should know how their data is being used.
• Differential Privacy: A technique to share information about a dataset while
withholding information about individuals.
• Algorithmic Fairness: Ensuring that predictive models do not discriminate based on
protected classes (race, gender, age).
9. Common Data File Formats and Interoperability
• .CSV: Human-readable, but lacks metadata about data types.
• .JSON: The backbone of web APIs. hierarchical and flexible.
• .Parquet: A columnar format. For example, if you only need 2 columns out of 100,
Parquet allows the system to read only those 2, drastically speeding up Big Data queries.
• .Avro: A row-based format frequently used in streaming data (Apache Kafka).
Chapter Summaries & Key Takeaways
Module 1-2: Foundations
Data is the raw ingredient; statistics is the recipe. Understanding the "Level of Measurement"
is non-negotiable for any researcher. Failure to distinguish between Nominal and Ratio data
leads to "garbage in, garbage out" scenarios in modeling.
Module 3-5: Engineering & Analysis
The transition from small-scale statistics to Big Data requires an understanding of the 5 V's
and the ETL pipeline. Data cleaning (Wrangling) typically consumes 80% of a data scientist's
time.
Module 6-9: Advanced Applications
Machine Learning provides the predictive power, but SQL provides the data access. Ethics
must be integrated at every stage of the lifecycle, not added as an afterthought. Future data
scientists must be as proficient in legal compliance (GDPR/CCPA) as they are in Python or
R.
Reference Materials:
Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning.
Silver, N. (2012). The Signal and the Noise.
End of Expanded Course Notes. Prepared for University Data Science Graduate Module.
(Ver 2.0)