DATA SCIENCE — UNIT 1

21CSS303T | Complete Notes

Syllabus Overview
• Benefits and uses of Data Science, Facets of data, The Data Science Process
• Introduction to NumPy: creating arrays, attributes, basic operations (join, split, search, sort), indexing,
slicing, iterating, copying, shape manipulation, identity array, eye function
• Pandas: Series, DataFrames, Index objects, Re-index, Drop Entry, Selecting Entries, Data Alignment,
Rank and Sort, Summary Statistics, Index Hierarchy
• Data Acquisition: gathering from different sources, Web APIs, Open Data Sources, Web Scraping
PART 1: INTRODUCTION TO DATA SCIENCE

1.1 Big Data vs Data Science

Big Data: A blanket term for any collection of datasets so large or complex that traditional data management
techniques (e.g. RDBMS) cannot process them.
Data Science: Involves using methods to analyze massive amounts of data and extract the knowledge it
contains.
Think of the relationship as: Big Data = crude oil, Data Science = the oil refinery.

Characteristics of Big Data (3 Vs)

• Volume — How much data is there?
• Variety — How diverse are different types of data?
• Velocity — At what speed is new data generated?

1.2 Benefits and Uses of Data Science

• It's in Demand
• Abundance of Positions
• A Highly Paid Career
• Data Science is Versatile
• Data Science Makes Data Better
• Data Scientists are Highly Prestigious
• No More Boring Tasks (automation)
• Data Science Makes Products Smarter
• Data Science Can Save Lives

1.3 Facets of Data

Data comes in many forms:
• Structured
• Unstructured
• Natural Language
• Machine-generated
• Graph-based
• Audio, Video, and Images
• Streaming

Structured Data
Data that depends on a data model and resides in a fixed field within a record. Rows and columns — easy to
query with SQL.

Unstructured Data
Data that isn't easy to fit into a data model because the content is context-specific or varying. Examples: emails,
PDFs, social media posts.

Natural Language
A special type of unstructured data. Challenging to process — requires NLP techniques and linguistics
knowledge. NLP has had success in: entity recognition, topic recognition, summarization, text completion, and
sentiment analysis. However, models trained in one domain don't generalize well to other domains.
Machine-Generated Data
Information automatically created by a computer, process, application, or machine without human intervention.
Becoming a major and growing data resource (IoT sensors, logs, telemetry, etc.).

Graph-Based / Network Data

• Based on mathematical graph theory — models pair-wise relationships between objects.
• Uses nodes, edges, and properties to represent and store data.
• Natural way to represent social networks.
• Allows calculating metrics like influence of a person, shortest path between two people.
• "Graph" here does NOT mean a bar chart — it refers to graph theory structures.

Audio, Video, and Image Data

Pose specific challenges to a data scientist — require specialized preprocessing, feature extraction, and deep
learning models.

Streaming Data
Can take any of the previous forms but has an extra property: data flows into the system when an event happens
instead of being loaded in a batch. Requires real-time processing pipelines.
PART 2: THE DATA SCIENCE PROCESS

The data science process consists of six main steps:

Step 1: Setting the Research Goal

Data science is mostly applied in an organizational context. A good research goal document should include:
• A clear research goal
• The project mission and context
• How you're going to perform your analysis
• What resources you expect to use
• Proof that it's an achievable project (proof of concept)
• Deliverables and a measure of success
• A timeline

Step 2: Retrieving Data

Data can be stored in many forms, from simple text files to tables in a database. Start with internal data sources:
• Databases
• Data Marts
• Data Warehouses
• Data Lakes

Data Lake
A centralized storage repository that holds a massive amount of structured and unstructured data. According to
Gartner: 'a collection of storage instances of various data assets additional to the originating data sources.'

Data Warehouse (DWH)

Collection of data from varied sources for meaningful business insights. An electronic storage of massive
information — a blend of technologies enabling strategic use of data.

Data Mart (DM)

A subtype of a data warehouse focused on a specific department or business unit.

DWH vs Data Mart — Key Differences

Criteria Data Warehouse Data Mart

Scope All departments in an organization Specific group/department

Size 100 GB to 1 TB+ Less than 100 GB

Design Complexity Complicated Easy to design

Data Handling Time Long Short

Implementation Time 1 month to 1 year A few months

DWH vs Data Lake — Key Differences

Data lakes are a newer concept and experts predicted they might reduce the need for data warehouses. With the
rise of unstructured data, data lakes are becoming popular. However, structured data is still preferred in data
warehouses.
Step 3: Cleansing, Integration and Transformation
Data Cleansing
A subprocess focused on removing errors so data becomes a true and consistent representation of the processes
it originates from.
• Interpretation errors — data misread or misrecorded
• Inconsistencies — same entity represented differently

Types of Data Errors

• Data Entry Errors — human typos, loss of concentration, machine/hardware failure
• Redundant Whitespaces — hard to detect but cause errors. Capital letter mismatches are common. Fix
with .lower() in Python
• Impossible Values / Sanity Checks — e.g., check = 0 <= age <= 120
• Outliers — observations distant from other observations; follow different logic. Find using plots or tables
• Missing Data — handle by imputation, dropping rows, or filling with defaults
• Codebook Deviations — a codebook is metadata describing your dataset (number of variables,
observations, encodings)

Combining Data from Different Sources

• Joining — enriching an observation from one table with information from another table using keys
(common fields like date, country, SSN). When keys uniquely define records they are called Primary Keys
• Appending / Stacking — adding observations of one table to another table
• Views — virtual layers that combine tables to avoid data duplication; behave like tables but store no extra
data
• Enriching Aggregated Measures — adding calculated information (e.g., total sales, percentage of stock
sold)

Transforming Data
Certain models require data in a specific shape. Transformations include:
• Reducing the number of variables — too many variables add noise, make models hard to handle, and
hurt performance
• Turning variables into dummies — dummy variables take only 0 or 1 to indicate presence/absence of a
categorical effect

Step 4: Data Exploration

Information becomes easier to grasp in pictures. Primarily use graphical techniques to understand data and
interactions between variables.

Visualization Techniques
• Simple graphs
• Histograms
• Sankey diagrams
• Network graphs
• Bar charts, Line charts, Distribution plots, Overlaying, Brushing and Linking

Step 5: Build the Models

Building a model is an iterative process. Depends on whether you use classic statistics or machine learning.

Main Steps in Modeling

• 1. Selection of modeling technique and variables
• 2. Execution of the model
• 3. Diagnosis and model comparison
Model Selection Considerations
• Must the model move to production? Is it easy to implement?
• How difficult is maintenance? How long will it stay relevant?
• Does it need to be easy to explain?

Step 6: Presentation and Automation

Present results to stakeholders and automate the pipeline for repeated use.
PART 3: INTRODUCTION TO NUMPY

3.1 What is NumPy?

• Numerical Python
• General-purpose array-processing package
• High-performance multidimensional array object with tools for working with arrays
• Fundamental package for scientific computing in Python
• Open-source software

NumPy Features
• A powerful N-dimensional array object
• Sophisticated broadcasting functions
• Tools for integrating C/C++ and Fortran code
• Useful linear algebra, Fourier transform, and random number capabilities

NumPy Array vs Python List

NumPy arrays are faster and more memory-efficient than Python lists for numerical operations because they are
stored in contiguous memory and support vectorized operations without loops.

3.2 Array Basics

Array: A data type used to store multiple values using a single variable name. Contains an ordered collection of
elements of the same type, referenced by index.
• Zero-based indexing: [10, 9, 99, 71, 90] → index 0 = 10
• NumPy arrays are officially called ndarray but commonly called array
• Used to store lists of numerical data, vectors, and matrices

3.3 Importing NumPy

import numpy as np

3.4 Creating NumPy Arrays

Method 1: Using NumPy Functions
# 1D array using arange
array = [Link](20)
# Output: array([ 0, 1, 2, ..., 19])

# 2D array using reshape

array = [Link](20).reshape(4, 5)
# Output: 4 rows, 5 columns

# Other useful functions

[Link]((2, 4)) # all zeros
[Link]((3, 6)) # all ones
[Link]((2, 2), 3) # filled with 3
[Link]((2, 3)) # uninitialized
[Link](3, 3) # identity matrix
[Link](0, 10, num=4) # evenly spaced: [0, 3.33, 6.67, 10]

NumPy Array Creation Functions Reference

 Function Description

[Link](n) Array from 0 to n-1

[Link](shape) Array of all zeros

[Link](shape) Array of all ones

[Link](shape, val) Array filled with val

[Link](shape) Uninitialized array

[Link](n) Identity matrix n×n

[Link](a,b,n) n evenly spaced values from a to b

np.empty_like(a) New array same shape and type as a

np.ones_like(a) Ones array same shape as a

np.zeros_like(a) Zeros array same shape as a

np.full_like(a, val) Filled array same shape as a

[Link](obj) Convert input to array

[Link](a) Copy of array a

[Link](v) Diagonal array

[Link](a,b,n) Evenly spaced on log scale

[Link](x) Vandermonde matrix

[Link](m) Upper triangle of matrix

[Link](m) Lower triangle of matrix

[Link](...) Coordinate matrices from vectors

[Link](a,b,n) Numbers spaced on log scale

Method 2: From Python Lists

import numpy as np
array = [Link]([4, 5, 6])
# Output: [4 5 6]

Working with ndarray — Quick Reference

Constructor / Function Purpose

[Link](shape, type) Creates array of given shape with random numbers

[Link](list_or_tuple) Creates array from list or tuple

[Link](shape) All zeros

[Link](shape) All ones

[Link](shape, obj, dtype) Array with complex numbers

[Link](range) Array with specified range

3.5 Array Attributes and Basic Operations

Attribute/Method Description
 [Link] Returns number of dimensions

[Link] Byte size of each element

[Link] Data type of elements

[Link] Tuple of array dimensions

[Link](r,c) Returns new view with given shape

arr[start:stop] Slicing — extracts subset

[Link](a,b,n) Returns n evenly spaced elements

[Link]() / [Link]() Maximum / minimum value

[Link]() Sum of all elements

[Link](arr) Element-wise square root

[Link]() Flattens array to 1D

3.6 Array Dimensions (ndim)

a = [Link](10) # 0D (scalar) → ndim = 0
b = [Link]([1,1,1,1]) # 1D → ndim = 1
c = [Link]([[1,1],[2,2]]) # 2D → ndim = 2
d = [Link]([[[1,1],[2,2]],[[3,3],[4,4]]]) # 3D → ndim = 3

# Force higher dimensions using ndmin

arr = [Link]([1,1,1], ndmin=10) # ndim = 10

3.7 Indexing and Slicing

Indexing
arr = [1, 2, 5, 6, 7]
arr[3] # → 6

Slicing
arr[2:5] # → [5, 6, 7] (start:stop, stop not included)
arr[:3] # → first 3 elements
arr[::2] # → every 2nd element
2D Slicing: arr[row_start:row_stop, col_start:col_stop]

3.8 Copying Arrays

Method 1: np.empty_like()
Returns new array with same shape and type as given array.
copy = np.empty_like(ary)
copy = ary # then assign
Syntax: numpy.empty_like(a, dtype=None, order='K', subok=True)

Method 2: [Link]()
Returns an array copy of the given object.
copy_array = [Link](org_array)
Syntax: [Link](a, order='K', subok=False)
Method 3: Assignment Operator
copy_array = org_array
WARNING: This creates a reference, NOT a true copy. Modifying org_array also modifies copy_array.

3.9 Iterating Arrays

1D Arrays
for x in arr:
print(x) # prints each element

2D Arrays
for x in arr:
print(x) # prints each row
# To get individual elements:
for x in arr:
for y in x:
print(y)

3D Arrays
for x in arr:
for y in x:
for z in y:
print(z)

Using nditer() — Efficient Iteration

The nditer() function iterates over every scalar element regardless of dimension — no nested loops needed.
for x in [Link](arr):
print(x)

3.10 Identity Array and eye()

identity()
Returns a square array with ones on the main diagonal.
[Link](n, dtype=None)
# n: dimension (n×n), dtype: float by default
[Link](4) # 4×4 identity matrix

eye()
Returns a 2D array with ones on a specified diagonal (not just main).
[Link](R, C=None, k=0, dtype=float)
# R: rows, C: columns (default = R), k: diagonal (0 = main)
# k > 0: above main diagonal, k < 0: below main diagonal

identity() vs eye() — Key Difference

Feature identity() eye()

Output shape Always square (n×n) Can be non-square (R×C)

Diagonal control Always k=0 (main diagonal) k can be set to any value

Use case When you need a standard identity When you need custom diagonal or
matrix non-square shape

3.11 Shape and Reshaping

Shape
[Link] # returns tuple e.g. (2, 4)

Reshape
Reshaping changes the shape without changing the data. Number of elements must remain the same.
[Link](4, 3) # 1D → 2D (4 rows, 3 cols)
[Link](2, 3, 2) # 1D → 3D
[Link](-1) # Flatten to 1D
If element count doesn't match: ValueError: cannot reshape array of size X into shape (Y,Z)

Flattening
newarr = [Link](-1) # converts any shape to 1D
Other related functions: flatten(), ravel(), rot90(), flip(), fliplr(), flipud()
PART 4: INTRODUCTION TO PANDAS

4.1 What is Pandas?

• Popular open-source data manipulation and analysis library for Python
• Provides easy-to-use data structures: DataFrame and Series
• Designed to make working with structured data fast, easy, and expressive
• Widely used for data cleaning, transformation, and exploration

4.2 Series
What is a Series?
• A one-dimensional array-like object that can hold data of any type (int, float, string, etc.)
• Labelled — each element has a unique identifier called an index
• Like a single column in a spreadsheet or database table
• Can be created from lists, arrays, dictionaries, or existing Series
• Building block for DataFrame

Creating a Series
import pandas as pd

# From a list
data = [1, 2, 3, 4, 5]
s = [Link](data) # index: 0,1,2,3,4

# From a dictionary
data = {'a': 1, 'b': 2, 'c': 3}
s = [Link](data) # index: a,b,c

# With custom index

s = [Link]([1,2,3], index=['a','b','c'])

Series Operations
• Indexing: series[0] or series['a']
• Vectorized Operations: s1 + s2 applies element-wise
• Alignment: Operations between Series auto-align on index labels; missing matches → NaN
• NaN Handling: Missing values represented as NaN (Not a Number); handled gracefully in operations
# Alignment example
s_a = [Link]([1,2,3], index=['a','b','c'])
s_b = [Link]([4,5,6], index=['b','c','d'])
s_a + s_b
# Output: a=NaN, b=6.0, c=8.0, d=NaN

4.3 DataFrame
What is a DataFrame?
• A two-dimensional, tabular data structure with rows and columns
• Similar to a spreadsheet or relational database table
• Three main components: data (stored in rows/columns), rows (labeled by index), columns (labeled with
actual data)
Creating a DataFrame
import pandas as pd

# From a dictionary
data = {'Name': ['John','Alice','Bob'], 'Age': [25,30,35], 'City':
['NY','LA','Chicago']}
df = [Link](data)

# From a list of lists

data = [['John',25,'NY'], ['Alice',30,'LA'], ['Bob',35,'Chicago']]
df = [Link](data, columns=['Name','Age','City'])

DataFrame Indexing
df['Name'] # Access a column → returns Series
[Link][0] # Access row by label
[Link][0] # Access row by integer position
[Link][0, 'Name'] # Access individual element by label

Column Operations
df['Salary'] = [50000, 60000, 70000] # Add new column
df[df['Salary'] > 60000] # Filter rows by condition
df.sort_values(by='Age', ascending=False) # Sort by column

Handling NaN
[Link]() # Drop rows with missing values
[Link](0) # Fill missing values with 0

Grouping and Aggregation

[Link]('City')['Age'].mean() # Group by City, get mean Age

4.4 Indexing in Pandas

Indexing is a fundamental operation for accessing and manipulating data efficiently.

Index Features
• Immutability — once created, an index cannot be modified
• Alignment — index objects align data structures (Series and DataFrames)
• Flexibility — Pandas offers integer-based, datetime, and custom index types

Creating a Custom Index

df = [Link](data, index=['A', 'B', 'C'])

4.5 Re-index
Reindexing creates a new DataFrame/Series with a different index. Missing labels get NaN.
new_index = ['A', 'B', 'D', 'E']
df_reindexed = [Link](new_index)
# Rows A, B carry forward; D and E get NaN

4.6 Drop Entry

Removing specific rows or columns from a dataset. Common in data cleaning to handle missing values, outliers,
or irrelevant data.
[Link]('Age', axis='columns') # Drop a column
[Link](0, axis='rows') # Drop a row by index
4.7 Selecting Entries
By Position
[Link][1] # Second row (integer position)

By Condition
df[df['Age'] > 30] # Rows where Age > 30

4.8 Data Alignment

Data alignment is intrinsic — it happens automatically based on labels, not position.
• align() function aligns two data objects according to their labels
• Works on both Series and DataFrame objects
• Returns new object of same type with labels compared and aligned
• Unmatched labels get NaN by default
df1_aligned, df2_aligned = [Link](df2, fill_value=[Link])

4.9 Rank
Ranking assigns ranks/positions to data elements based on their values. Rank is based on position after sorting.

Rank Parameters
Parameter Description

method='average' Default — average rank for ties

method='max' Maximum rank for ties

method='min' Minimum rank for ties

na_option='bottom' Place NaN at bottom of rankings

na_option='top' Place NaN at top

pct=True Return percentage rank (0 to 1)

df['rank'] = df['Number_legs'].rank()
df['pct_rank'] = df['Number_legs'].rank(pct=True)

4.10 Sort
Sort a DataFrame by values of one or more columns.

Key Points
• sort_values(by='col') — sort by column values
• ascending=False — descending order
• sort_index() — sort by index
• inplace=True — sort in place, no copy
• NaN values can be organized with na_position parameter
df.sort_values(by=['Country']) # Ascending
df.sort_values(by=['Population'], ascending=False) # Descending
df.sort_index() # Sort by index

4.11 Summary Statistics

Pandas provides built-in methods for descriptive statistics:
Method Description
 [Link]() Summary stats: count, mean, std, min, max, quartiles

[Link]() Mean of each column

[Link]() Median of each column

[Link]() Standard deviation

[Link]() Variance

[Link]() Sum of values

[Link]() Count of non-NaN values

[Link]() / [Link]() Minimum / Maximum values

[Link]() Correlation between columns

4.12 Index Hierarchy (MultiIndex)

Pandas supports hierarchical indexing, allowing multiple levels of indexing on an axis.
• Enables working with higher-dimensional data in a 2D table
• Created using [Link] or by setting multiple columns as index
• Access levels with [Link][(level0_val, level1_val)]
# Create MultiIndex
arrays = [['A','A','B','B'], [1,2,1,2]]
index = [Link].from_arrays(arrays, names=('letter','number'))
df = [Link]({'val': [10,20,30,40]}, index=index)
PART 5: DATA ACQUISITION

Gathering information from different sources is a critical step in the data science process.

5.1 Sources of Data

• Internal company databases, data marts, data warehouses, data lakes
• Web APIs
• Open Data Sources
• Web Scraping

5.2 Web APIs

• API = Application Programming Interface
• Web APIs allow programs to communicate with web services and retrieve data programmatically
• Data is typically returned in JSON or XML format
• Common use: social media data (Twitter, Facebook), financial data, weather data
import requests
response = [Link]('[Link]
data = [Link]()

5.3 Open Data Sources

• Government datasets: [Link], [Link]
• Research repositories: UCI Machine Learning Repository, Kaggle
• UN, WHO, World Bank open data portals
• Typically provided in CSV, JSON, or Excel format

5.4 Web Scraping

• Extracting data directly from web pages when no API is available
• Tools: BeautifulSoup, Scrapy, Selenium
• Parse HTML/CSS structure to extract required elements
• Respect [Link] and website terms of service
from bs4 import BeautifulSoup
import requests
page = [Link]('[Link]
soup = BeautifulSoup([Link], '[Link]')
data = soup.find_all('td') # extract table data

End of Unit 1 Notes | 21CSS303T | Data Science