MODULE 1

Data Science:-
Data Science is the study of data that uses statistical methods, programming, and analytical
techniques to discover patterns, trends, and knowledge from structured and unstructured data.
or
Data science is the study of data that helps us derive useful insight for business decision
making. Data Science is all about using tools, techniques, and creativity to uncover insights
hidden within data. It combines math, computer science, and domain expertise to tackle
real-world challenges in a variety of fields.

Data Science processes the raw data and solves business problems and even makes
predictions about the future trend or requirement.

🔹 Key Components of Data Science

1.​ Data Collection – Gathering data from sources such as databases, sensors, websites,
and logs​

2.​ Data Cleaning – Handling missing values, errors, duplicates, and noise​

3.​ Data Exploration & Analysis – Understanding data using statistics and visualizations​

4.​ Data Modeling – Applying statistical models or machine learning algorithms​

5.​ Interpretation & Communication – Presenting insights using reports, charts, and
dashboards

Importance of Data Science (Easy Explanation)

1.​ Better Decision Making​
Data Science helps companies make better decisions by studying past data. It shows
patterns and trends, which help in understanding customer behavior and predicting
future needs.
2.​ Improves Work Efficiency​
Data Science finds problems and delays in work processes. This helps organizations
save time, reduce cost, and improve productivity.​
Example: Finding faster delivery routes in logistics.
3.​ Encourages Innovation​
Data Science helps in creating new products and services. By using machine learning
and data analysis, companies can try new ideas and improve existing systems.​
 4.​ Helps in Prediction​
Data Science is used to predict future events using old data.​
Example: Weather prediction, disease prediction, and sales forecasting.
5.​ Helps Society and Science​
Data Science is useful in solving real-life problems like tracking diseases, studying
climate change, and improving public transport systems.
6.​ Manages Big Data​
Large amounts of data are produced every second from mobiles, websites, and
sensors. Data Science helps in storing, processing, and understanding this huge data.
7.​ Supports Automation and AI​
Data Science is the base for Artificial Intelligence. It helps machines learn from data and
make decisions automatically.​
Example: Chatbots, recommendation systems, self-driving cars.

Advantages of Data Science (Easy Explanation)

1.​ Better Decision Making​
Data Science helps organizations take decisions based on data and facts instead of
guesswork or intuition.
2.​ Prediction of Future Trends​
Using past data, Data Science can predict future trends such as sales demand, risks, or
customer behavior.
3.​ Automation​
Data Science supports Artificial Intelligence systems like chatbots, fraud detection
systems, and recommendation engines, reducing manual work.
4.​ Improved Customer Experience​
It helps provide personalized services such as product recommendations and targeted
marketing.
5.​ Innovation and Efficiency​
Data Science improves existing processes and helps in creating new data-driven
products and services.​

Disadvantages of Data Science (Easy Explanation)

1.​ Data Privacy Issues​
Collecting and using personal data can cause ethical and legal problems if not handled
properly.
2.​ High Cost and Complexity​
Data Science needs skilled professionals, advanced software, and powerful computers,
which can be expensive.
 3.​ Bias in Data​
If the data used is biased or incorrect, the results and decisions will also be unfair or
wrong.
4.​ Dependence on Data Quality​
Data Science results are reliable only when the data is accurate, complete, and up to
date.​

Applications of Data Science (Easy Explanation)

1.​ E-commerce​
Used for product recommendations and personalized shopping (Amazon, Flipkart).
2.​ Healthcare​
Helps in disease prediction, drug discovery, and medical image analysis.
3.​ Finance​
Used for fraud detection, credit scoring, and stock market analysis.
4.​ Education​
Predicts student performance and supports smart learning systems.
5.​ Transportation​
Used for route optimization and demand prediction (Uber, Ola).
6.​ Social Media​
Helps in sentiment analysis, content recommendation, and targeted advertisements.

Big Data
Big Data refers to very large and complex sets of data that cannot be easily stored, processed,
or analyzed using traditional databases or tools.​
This data is generated from many sources such as social media, sensors, online
transactions, mobile phones, and websites.

Big Data is not only about the size of data, but also about the speed, different types, and
quality of data being generated.

Big Data helps organizations find hidden patterns and trends. These insights help in better
decision-making, improving efficiency, and innovation.​
For example:

●​ Amazon and Netflix use Big Data to give personalized recommendations.

●​ Governments use Big Data for public health monitoring and city planning.
Key Characteristics of Big Data (5 V’s)
1. Volume

●​ Refers to the huge amount of data generated every second.

●​ Example: Social media platforms generate terabytes to petabytes of data daily.

2. Velocity

●​ Refers to the speed at which data is created and processed.

●​ Example: Stock markets and IoT systems need real-time data processing.

3. Variety

●​ Refers to different types of data:

○​ Structured – Tables, databases
○​ Semi-structured – XML, JSON
○​ Unstructured – Images, videos, social media posts
●​ Example: Combining customer purchase data with social media reviews.

4. Veracity

●​ Refers to the accuracy, consistency, and reliability of data. Since Big Data comes from
multiple sources, ensuring data quality is critical.
●​ Example: Incorrect or incomplete sensor data can lead to wrong results.

5. Value

●​ Refers to the useful insights obtained from data.

●​ Example: Retail companies use customer data to improve sales and marketing
strategies.

Big Data and Data Science

Aspect Big Data Data Science

Definition Deals with handling very large and Deals with analyzing data to get
complex data insights

Main Focus Storage, processing, and Analysis, prediction, and

management of data decision-making

Data Size Mainly works with huge datasets Works with small to large datasets
(TBs, PBs)
 Goal Efficient data storage and processing Extract meaningful information and
insights

Techniques Distributed computing, parallel Statistics, machine learning, data

Used processing analysis

Tools Used Hadoop, Spark, HDFS, NoSQL Python, R, SQL, Tableau, ML

databases algorithms

Output Processed and organized data Predictions, models, insights,

reports

Users Data Engineers, System Architects Data Scientists, Analysts

Datafication
Definition:
Datafication is the process of transforming real-world activities, behaviors, and interactions into
data that can be stored, analyzed, and used for decision-making.
It means turning aspects of human life into quantified data that can be tracked and studied using
technology.
In simple terms, datafication converts things that were never considered data before — such as
social interactions, habits, emotions, and movements — into digital information.

Key Points:
• Datafication enables organizations to collect, analyze, and gain insights from everyday
actions.
• It forms the foundation of Big Data and Data Science.
• It is not just about digitization (converting analog to digital), but about capturing data from
every activity.

Example:
• Social media platforms like Instagram and Facebook classify user actions — likes, comments,
and shares become measurable data points used for targeted advertising.
• Fitness trackers (like Fitbit) convert steps, heart rate, and sleep patterns into digital data for
health analysis.

Importance: Datafication enables personalized experiences, improved decision-making, and

automation. It fuels AI applications by generating the vast datasets needed for training models
Statistical Inference
Statistical inference is the process of drawing conclusions about a population based on
information obtained from a sample.

In simple words:

Statistical inference helps us use sample data to make decisions or

predictions about the entire population.

Why Statistical Inference is Needed

●​ Studying the whole population is often impossible or expensive
●​ Samples save time, cost, and effort
●​ Helps in decision-making under uncertainty

Main Types of Statistical Inference

1. Estimation

Estimation is used to estimate unknown population parameters using sample data.

●​ Point Estimation: Gives a single value​

Example: Sample mean used to estimate population mean
●​ Interval Estimation: Gives a range of values​
Example: Confidence intervals

2. Hypothesis Testing

Hypothesis testing is used to test assumptions or claims about a population using sample
data.

Steps involved:

1.​ State the null hypothesis (H₀)

2.​ State the alternative hypothesis (H₁)
3.​ Choose a significance level (α)
4.​ Calculate test statistic
5.​ Make a decision (accept or reject H₀)
Example:​
Testing whether the average marks of students is greater than 70.

Applications of Statistical Inference

●​ Business: Market research, quality control
●​ Healthcare: Drug testing, disease studies
●​ Finance: Risk analysis, investment decisions
●​ Government: Census analysis, policy decisions

Conclusion
Statistical inference is a key part of Data Science that helps in making reliable conclusions
and predictions about a population using limited data.

Population and Sample

In statistics, population and sample are fundamental concepts used to describe groups of
data.

Population
A population refers to the entire set of individuals, objects, or data points that you want to
study.​
The size of a population can be large or small, depending on the scope of the research.

When to Use a Population

A population is used when:

●​ Your research requires information from every member

●​ You have access to data from the entire population
●​ The population is small, accessible, and cooperative

Usually, collecting data from a whole population is practical only when the population size is
limited.
Example

●​ A marketing manager at a small local bakery wants to understand customer

preferences.
●​ They collect data on every customer’s bread purchase over one month.
●​ Since the customer base is limited and accessible, they analyze the entire population
to identify trends.

Sample
A sample is a subset of the population selected for analysis.​
Sampling is used when studying the entire population is impractical or impossible.

Sampling allows researchers to make inferences about the population using statistical
techniques.

When to Use a Sample

A sample is used when:

●​ The population is very large

●​ The population is geographically dispersed
●​ It is difficult or costly to contact everyone

Using sample data, statisticians can estimate population values or test hypotheses.

Example

●​ You are researching smartphone usage among teenagers in a city.

●​ The population includes all teenagers aged 13–18, which could be tens of thousands.
●​ You select a random sample of 500 teenagers from different schools.

Formulas for Population and SamplePopulation Mean

The population mean is denoted by μ (mu) and is calculated as:

​
Population Standard Deviation

The population standard deviation is denoted by σ (sigma) and is given by:

Sample Mean

The sample mean is denoted by x̄ (x-bar) and is calculated as:

Sample Standard Deviation

The sample standard deviation is denoted by s and is given by:

Statistical Modelling
Definition:
Statistical Modelling is the process of creating mathematical equations or models that describe
the relationship between variables in a dataset.
It helps in understanding, predicting, and explaining how one variable changes when others
change.
In simple terms, a statistical model is a representation of real-world data using mathematical
relationships.
Key Points:
• It involves assumptions about data distribution and relationships.
• Used to summarize complex data and make predictions.
• A statistical model estimates parameters (like coefficients) that describe how variables are
related.
Example:
• In healthcare, logistic regression models predict disease likelihood based on age, BMI, and
medical history.
• In finance, time-series models predict stock market trends based on historical prices.

Components of Statistical Modelling

1.​ Data – Collected observations or values
2.​ Variables
○​ Independent variables (inputs)
○​ Dependent variable (output)
3.​ Model Equation – Mathematical representation
4.​ Parameters – Values estimated from data
5.​ Error Term – Difference between actual and predicted values

Types of Statistical Models

1.​ Regression Models
○​ Linear Regression
○​ Multiple Linear Regression
2.​ Classification Models
○​ Logistic Regression
3.​ Probability Models
○​ Normal distribution
○​ Binomial distribution

Statistical Modeling for Prediction and Decision-Making

Statistical modeling is an important tool used for prediction and decision-making in fields
such as business, healthcare, finance, and many others. By analyzing data and identifying
patterns, statistical models help predict future outcomes, estimate probabilities, and support
better decisions.

Role of Statistical Modeling in Prediction

1.​ Identifying Relationships​
Statistical models identify relationships between different variables. This helps predict
outcomes based on input data.​
Example: Predicting customer churn using demographic and behavioral data.
 2.​ Estimating Probabilities​
Models estimate the probability of events such as loan default or disease occurrence.
This helps decision-makers understand the chances of different outcomes.
3.​ Forecasting​
Statistical models are used to forecast future values like sales, demand, or stock prices.
This allows organizations to prepare for future situations.

Role of Statistical Modeling in Decision-Making

1.​ Risk Assessment​
Statistical models help assess risks before making decisions.​
Example: Predicting the risk of loan default using credit scores.
2.​ Optimization​
Models help optimize processes such as pricing strategies, resource allocation, and
production planning, leading to better efficiency and profits.
3.​ What-If Analysis​
Statistical models simulate different scenarios to study possible outcomes. This helps
decision-makers test ideas and understand the impact of various choices.

Benefits of Statistical Modeling

1.​ Data-Driven Decisions​
Models provide objective insights based on data, reducing personal bias and
guesswork.
2.​ Improved Accuracy​
Statistical models reduce uncertainty and improve prediction accuracy, leading to better
planning.
3.​ Increased Efficiency​
Automating predictions and decisions saves time and resources, allowing organizations
to focus on strategic goals.

Applications of Statistical Modeling

1.​ Business​
Used to predict customer behavior, optimize pricing, and identify market trends.
2.​ Healthcare​
Used for disease prediction, identifying high-risk patients, and planning treatments.
3.​ Finance​
Used for stock price prediction, credit risk assessment, and investment optimization.

Probability Distribution
A probability distribution is a mathematical function that assigns the probabilities of different
outcomes to the possible values of a random variable. It provides a way of modeling the
likelihood of each outcome in a random experiment.

Properties of a probability distribution include:

The probability of each outcome is greater than or equal to zero. (P(X)≥0)

The sum of the probabilities of all possible outcomes equals 1. (∑P(X)=1 or ∫ 𝑓(𝑥)𝑑𝑥 = 1)

Definition of Probability Distribution:

A **probability distribution** is a way of showing how the probabilities are distributed over all

possible outcomes of a random variable.

## **Types of Probability Distributions**

There are **two main types** based on the type of random variable:

1. Discrete Probability Distribution

A discrete probability distribution deals with countable outcomes that is, outcomes you can list
one

by one.

A common example is the Binomial Distribution, which models the number of successes in a
fixed

number of independent trials, each with the same probability of success.

#### **General Examples:**

* The number of heads when tossing a coin three times

* The number of students present in a class

* The number of goals scored in a football match

* Rolling a die (possible outcomes: 1, 2, 3, 4, 5, 6)

**Example Table:**

| Outcome (X) | 1 | 2 | 3 | 4 | 5 | 6 |

| ---------------- | --- | --- | --- | --- | --- | --- |

| Probability P(X) | 1/6 | 1/6 | 1/6 | 1/6 | 1/6 | 1/6 |

Here, the probabilities are **non-negative** and add up to **1**.

2. Continuous Probability Distribution

A continuous probability distribution*deals with uncountable outcomes, usually measurements.

A

classical example is the Normal Distribution, also known as the Gaussian distribution. It is

described by its mean (μ) and standard deviation (σ).

The probabilities are represented by a **curve**, not individual points.

**General Examples:**

* Heights or weights of students

* Time taken to run a race

* Temperature of a city * Amount of rainfall in a month Example:

If the height of students in a class follows a normal distribution, most students have an average

height, while fewer are very tall or very short.

The area under the curve (representing all possible heights) equals 1.

Fitting a Model
Fitting a model is the process of selecting a suitable statistical model and estimating its
parameters so that it best represents the relationship between variables in a dataset.

In simple words:

Fitting a model means training a model using data so that it can explain
patterns and make predictions.

Why Model Fitting is Needed

●​ To understand the relationship between input and output variables
●​ To make predictions on new or unseen data
●​ To support decision-making
●​ To reduce errors between actual and predicted values

Steps Involved in Fitting a Model

1.​ Data Collection​
Collect relevant data required for analysis.
2.​ Data Preprocessing​
Clean the data by handling missing values, outliers, and errors.
3.​ Selecting a Model​
Choose a suitable statistical model such as:
●​ Linear Regression
●​ Logistic Regression
●​ Probability models
4.​ Estimating Parameters​
Calculate model parameters (coefficients) using data.​
Example: Using the least squares method in linear regression.
5.​ Model Evaluation​
Check how well the model fits the data using:
●​ Error measures
●​ Accuracy
●​ Residual analysis
6.​ Model Validation​
Test the model on new or unseen data to ensure reliability.

Example of Model Fitting

●​ Predicting house prices based on area and location
 ●​ Predicting student marks based on study hours

The model is adjusted until the difference between actual values and predicted values is
minimized.

-​ underfitting and overfitting are common problems that affect a model’s performance

Underfitting
Underfitting occurs when a model is too simple to capture the underlying pattern in the data.

Characteristics of Underfitting

●​ Model has high bias

●​ Performs poorly on training data
●​ Performs poorly on test data
●​ Fails to learn important patterns

Causes of Underfitting

●​ Using a very simple model

●​ Insufficient features
●​ Inadequate training

Example

Using a straight line to fit data that clearly follows a curved pattern.

Overfitting
Overfitting occurs when a model learns the training data too well, including noise and errors.

Characteristics of Overfitting

●​ Model has high variance

●​ Performs very well on training data
●​ Performs poorly on test data
●​ Does not generalize well​

Causes of Overfitting

●​ Very complex model

 ●​ Too many features
●​ Training for too long

Example

A highly complex curve that passes through every training data point but fails on new data.

Aspect Underfitting Overfitting

Definition Occurs when the model is too simple Occurs when the model is too
to capture the pattern in data complex and learns noise in data

Model Complexity Very low Very high

Learning Ability Fails to learn important patterns Learns patterns and noise
excessively

Bias High bias Low bias

Variance Low variance High variance

Training Accuracy Low Very high

Testing Accuracy Low Low (drops sharply)

Generalization Poor generalization Poor generalization

Error on Training Data High Very low

Error on Test Data High High

Reason Model is too simple or lacks features Model is too complex or trained too
long

Flexibility Inflexible model Over-flexible model

Sensitivity to Data Less sensitive to data changes Highly sensitive to small data
changes

Typical Models Linear model for nonlinear data Very high-degree polynomial model

Data Pattern Capture Misses true data trends Fits even random noise

Performance on New Poor Poor

Data

Solution Increase model complexity, add Reduce complexity, apply

features regularization

Visual Shape Straight line not fitting curve Highly curved line fitting all points