DATA SCIENCE
UNIT 2 STUDY NOTES
Data Collection, Sources, Types, Datasets and Exploration with Pandas
2.1 Data Collection Methods
Data collection is the systematic process of gathering information relevant to a problem before any analysis can
begin. The quality of every later step — cleaning, modelling, and conclusions — depends directly on how well the
data was collected.
Common Data Collection Methods
Method Description Example
Surveys / Questionnaires Structured questions given to a target group Student feedback form
Interviews Direct one-on-one or group conversation Faculty interviewing an alumnus
Observation Recording behaviour or events as they happen, Foot traffic count in a library
without direct interaction
Experiments Controlled tests that measure the effect of A/B testing a website layout
changing one variable
Web Scraping / APIs Automated extraction of data from websites or Collecting product prices from an e-
services commerce site
Sensors / IoT Devices Automatic, continuous measurement from Temperature logger, fitness tracker
hardware
Choosing a Method
• Consider cost, time available, required accuracy, and whether the data already exists somewhere (see 2.2).
• Quantitative methods (surveys with rating scales, sensor logs) suit statistical analysis; qualitative methods
(interviews, open-ended responses) suit deeper contextual understanding.
• Always plan how the data will be stored and structured at the point of collection — retrofitting structure onto
messy data later is far more expensive.
2.2 Sources of Data
Beyond how data is collected, it's important to distinguish where it originates — this affects reliability, cost, and how
much cleaning will be needed.
2.2.1 Primary Data Sources
Data collected first-hand by the researcher/organisation for a specific purpose. It doesn't exist anywhere else before
you gather it.
• Examples: a survey you design and run yourself, sensor readings from your own device, an experiment you
conduct.
• Advantages: highly relevant to your exact question, you control data quality and collection method.
• Disadvantages: time-consuming and often expensive to collect at scale.
2.2.2 Secondary Data Sources
Data that already exists, originally collected by someone else for a different purpose, and reused for your analysis.
• Examples: government census data, published research datasets, Kaggle datasets, company sales records from a
previous project.
• Advantages: fast and cheap to obtain, often much larger in scale than you could collect yourself.
• Disadvantages: may not perfectly match your needs, quality/collection methodology may be unknown or
outdated.
Primary vs Secondary — Quick Comparison
Aspect Primary Data Secondary Data
Collected by You / your organisation Someone else, earlier
Cost & time Higher Lower
Relevance Exactly fits your need May need adaptation
Control over quality High Limited
Learning Outcome for 2.2
You should be able to classify a given dataset as primary or secondary, and explain one advantage and one
disadvantage of each.
2.3 Types of Data
Data is also classified by how it's organised internally — this determines which tools and techniques can be used to
process it.
2.3.1 Structured Data
Data that follows a fixed schema — organised into rows and columns, each with a defined data type. This is the
easiest type to query and analyse directly.
# Structured data — a relational table / CSV
id | name | age | branch
1 | Riya | 22 | MCA
2 | Aman | 21 | MCA
• Examples: SQL database tables, Excel spreadsheets, CSV files.
• Tools: SQL, Pandas DataFrames — both assume a clear, consistent row/column structure.
2.3.2 Semi-Structured Data
Data that doesn't sit in a strict table but still has organisational tags/markers that give it some structure — fields can
vary between records.
{
"name": "Riya",
"branch": "MCA",
"subjects": ["Python", "DBMS"]
}
// Another record in the same file could have extra or missing fields
• Examples: JSON, XML, NoSQL documents (MongoDB), emails (headers + free text body).
• Requires some parsing/normalisation before it can be analysed like structured data.
2.3.3 Unstructured Data
Data with no predefined format or organisation at all — the majority of data generated today falls into this category.
• Examples: plain text documents, images, audio, video, social media posts.
• Requires specialised techniques to extract meaning — NLP for text, computer vision for images, etc. — before it
can be analysed quantitatively.
Structured vs Semi-Structured vs Unstructured
Type Schema Example Typical Tools
Structured Fixed SQL table, CSV SQL, Pandas
Semi-Structured Flexible/tagged JSON, XML Python dict parsing, MongoDB
Unstructured None Images, free text NLP, computer vision
2.4 Introduction to Datasets
A dataset is an organised collection of data, typically arranged as rows (records/observations) and columns
(features/variables), used as the input to analysis or a machine learning model.
Key Dataset Terminology
Term Meaning
Observation / Record One row — a single data point, e.g. one student
Feature / Variable One column — one measured attribute, e.g. 'age'
Label / Target The variable you want to predict or explain (in supervised
ML)
Missing Value A record where a feature's value is absent or null
Outlier A value that differs significantly from the rest of the data
Common Public Dataset Sources
• Kaggle Datasets — [Link]/datasets — huge variety, community-contributed.
• UCI Machine Learning Repository — classic, well-documented datasets for learning.
• Government open-data portals — e.g. [Link] — census, economic, and public-service data.
• Built-in datasets in libraries — e.g. seaborn.load_dataset('iris'), [Link] — useful for practice.
2.5 Data Import and Export Using Pandas
Importing Data
import pandas as pd
df = pd.read_csv("[Link]") # from CSV
df = pd.read_excel("[Link]", sheet_name="Sheet1") # from Excel
df = pd.read_json("[Link]") # from JSON
df = pd.read_sql("SELECT * FROM students", conn) # from a database connection
Useful read_csv() Parameters
Parameter Purpose
sep Specify delimiter, e.g. sep=';' for semicolon-separated files
header Row number to use as column names (default 0)
na_values Extra strings to treat as missing, e.g. na_values=['NA', '-']
usecols Load only specific columns, e.g. usecols=['name', 'age']
nrows Load only the first N rows — useful for large files
Exporting Data
df.to_csv("[Link]", index=False) # index=False avoids writing row numbers
df.to_excel("[Link]", sheet_name="Data", index=False)
df.to_json("[Link]", orient="records")
2.6 Basic Data Exploration
Before any cleaning or modelling, it's essential to explore a new dataset first — understanding its shape, types, and
basic statistics prevents mistakes later and reveals obvious data-quality issues early.
2.6.1 Dataset Inspection
import pandas as pd
df = pd.read_csv("[Link]")
[Link]() # first 5 rows — quick sanity check
[Link](3) # last 3 rows
[Link] # (rows, columns) — overall size
[Link] # list of column names
[Link] # data type of each column
[Link]() # combined summary: types, non-null counts, memory usage
[Link]().sum() # count of missing values per column
[Link]().sum() # number of duplicate rows
2.6.2 Summary Statistics
[Link]() # count, mean, std, min, quartiles, max — numeric
columns
[Link](include="object") # summary for text/categorical columns
df["age"].mean()
df["age"].median()
df["age"].std() # standard deviation
df["age"].min(), df["age"].max()
df["branch"].value_counts() # frequency of each category
[Link](numeric_only=True) # correlation between numeric columns
What to Look For at This Stage
• Unexpected data types — e.g. a numeric column read as text because of a stray symbol.
• Missing values concentrated in specific columns — may need imputation or removal later.
• Obvious outliers in min/max values (e.g. a negative age, or marks over 100).
• Imbalanced categories in value_counts() — relevant for later classification tasks.
Unit 2 — Quick Revision Map
Sub-topic Core Idea Key Tool/Concept
2.1 How data gets gathered Surveys, interviews, sensors, scraping
2.2 Where data originates Primary vs Secondary sources
2.3 How data is organised internally Structured / Semi-structured /
Unstructured
2.4 What a dataset actually is Observations, features, labels, public
datasets
2.5 Getting data in and out of Pandas read_csv/read_excel/
read_json, to_csv etc.
2.6 First look at a new dataset head(), info(), describe(),
value_counts()