UNIT – 1

Data Science
Data science is the study of data used to extract meaningful insights
for business decisions. It combines mathematics, computing and
domain knowledge to solve real-world problems and uncover hidden
patterns.
It processes raw data to address business challenges and predict future
trends. For example, from large company datasets, data science can
help answer questions like:
• What do customer want?
• How can we improve our services?
• What will the upcoming trend in sales?
• How much stock they need for upcoming festival.

Increasing Demand of Data Science

Here are some key reasons why Data Science is important:
• Helps Business in Decision-Making: By analyzing data,
businesses can understand trends and make informed choices
that reduce risks and maximize profits.
• Improves Efficiency: Organizations can use data science to
identify areas where they can save time and resources.
 • Personalizes Experiences: Data science helps create
customized recommendations and offers that improve customer
satisfaction.
• Predicts the Future: Businesses can use data to forecast trends,
demand and other important factors.
• Drives Innovation: New ideas and products often come from
insights discovered through data science.
• Benefits Society: Data science improves public services like
healthcare, education and transportation by allocating resources
more effectively.

Real Life Example of Data Science

Some of the real life examples of Data Science are:
• E-Commerce: Recommender systems suggest products by
analyzing browsing history and user behavior patterns.
• Streaming Platforms: Machine learning models recommend
movies and videos based on watch duration and engagement
metrics.
• Banking and Finance: Fraud detection systems flag unusual
transaction activity using anomaly-based classification.
• Healthcare Diagnostics: Predictive models analyze medical
scans to assist doctors in early disease detection and treatment
decisions.
• Ride-Sharing: Algorithms estimate fares and optimal routes by
evaluating traffic conditions and supply-demand ratios.
• Social Media: Sentiment analysis examines user posts to
understand public opinion on brands, events and trends.
Here are some of the key data science job roles:
1. Data Scientist
• Responsibilities: Analyzing large datasets, developing
machine learning models, interpreting results and providing
insights to inform business decisions.
• Skills: Proficiency in programming languages like Python or
R, expertise in statistics and machine learning algorithms, data
visualization skills and domain knowledge in the relevant
industry.
2. Data Analyst
• Responsibilities: Collecting, cleaning and analyzing data to
identify trends, patterns and insights. Often involves creating
reports and dashboards to communicate findings to
stakeholders.
• Skills: Strong proficiency in SQL for data querying, experience
with data visualization tools like Tableau or Power BI, basic
statistical knowledge and familiarity with Excel or Google
Sheets.
3. Machine Learning Engineer
• Responsibilities: Building and deploying machine learning
models at scale, optimizing model performance and integrating
them into production systems.
• Skills: Proficiency in programming languages like Python or
Java, experience with machine learning frameworks like
TensorFlow or PyTorch, knowledge of cloud platforms like
AWS or Azure and software engineering skills for developing
scalable solutions.
4. Data Engineer
• Responsibilities: Designing and building data pipelines to
collect, transform and store large volumes of data. Ensuring
data quality, reliability and scalability.
• Skills: Expertise in database systems like SQL and NoSQL,
proficiency in programming languages like Python or Java,
experience with big data technologies like Hadoop or Spark
and knowledge of data warehousing concepts.
5. Business Intelligence (BI) Analyst
• Responsibilities: Gathering requirements from business
stakeholders, designing and developing BI reports and
dashboards and providing data-driven insights to support
strategic decision-making.
• Skills: Proficiency in BI tools like Tableau, Power BI or
Looker, strong SQL skills for data querying, understanding of
data visualization principles and ability to translate business
needs into technical solutions.

Exploratory Data Analysis (EDA)

Exploratory Data Analysis (EDA) is the process of examining and
investigating datasets to understand their main characteristics using
statistical summaries and visualizations. It helps data scientists
explore the structure of data, identify important variables, and detect
issues such as missing values or unusual patterns. EDA also guides
how to clean, transform, and prepare the data for deeper analysis.
EDA goes beyond formal modeling by revealing hidden patterns,
relationships between variables, and unexpected behaviors in the data.
It helps validate assumptions, choose suitable statistical techniques,
and refine hypotheses before applying advanced models. First
introduced by mathematician John Tukey in the 1970s, EDA remains
an essential step in modern data science to ensure accurate,
meaningful, and reliable analysis.
Types of EDA
• Univariate Graphical
• Univariate Non-Graphical
• Bivariate Graphical
• Bivariate Non-Graphical
• Multivariate Graphical
• Multivariate Non-Graphical
• Univariate Graphical EDA:
Univariate graphical EDA is used to visually study one variable at a
time. These graphs help us understand the shape of the data, its
spread, whether it has outliers, and how frequently different values
appear. Visualizing a single variable is the easiest and most common
starting point in data analysis.
Important graphical methods in Univariate EDA:
• Histogram:
o Shows how often values appear
o Helps identify distribution shape (normal, skewed, etc.)

• Boxplot:
o Shows median, quartiles, and outliers
o Very useful to detect extreme values
 • Bar Chart:
o Used for categorical data
o Shows counts or frequencies of categories

• Univariate non-graphical EDA is the simplest method of data

analysis because it focuses on only one variable at a time. It helps us
understand the basic properties of the data and the overall distribution
of values. This type of analysis is mainly used to study the sample
data and make observations about the larger group it represents. It
also plays an important role in identifying outliers, which are unusual
or extreme values that may affect analysis.
• Central Tendency (middle values):
o Mean → average value
o Median → middle value (very useful when data has
extreme values)
o Mode → most repeated value
• Spread (how far values are from the center):
o Variance: average of squared differences from the mean
 o Standard deviation: square root of variance; tells how
much the values vary
• Outlier detection: helps identify values that are too high or too
low compared to the rest

• Bivariate Graphical EDA

Bivariate graphical EDA is used to analyze two variables together.
These graphs help identify relationships, patterns, whether variables
move together, or how one variable changes with respect to another.
This type of visualization is essential when we want to understand
connections between factors.
Important graphical methods in Bivariate EDA:
• Scatter Plot:
o Best for two numerical variables
o Shows direction and strength of relationship

• Line Plot:
o Used when data has a sequence (like time series)
o Shows how one variable changes over time or steps
 • Boxplot (Category vs. Numeric):
o One axis = category, other axis = numeric variable
o Helps compare distributions across groups

• Bivariate Non-Graphical EDA

Bivariate non-graphical EDA is used when we analyze two variables
at the same time to understand how they are related. Instead of
looking at each variable separately, this type of analysis helps us
check whether one variable changes when the other changes. It is
useful for understanding relationships such as positive or negative
association, differences between groups, or how one factor might
affect another.
• Correlation
o Measures how strongly two variables move together
o Values range from –1 to +1
 o +1 → strong positive relation, –1 → strong negative
relation
• Covariance
o Shows whether two variables increase or decrease together
o Harder to interpret compared to correlation
• Multivariate Graphical EDA
Multivariate graphical EDA is used to explore three or more
variables at the same time. These visualizations help reveal complex
patterns, interactions, and combined effects that cannot be seen with
just univariate or bivariate plots. They are commonly used when
preparing data for machine learning models, which depend on multiple
input variables.
Important graphical methods in Multivariate EDA:
• Pair Plot (Scatterplot Matrix):
o Multiple scatter plots arranged in a grid
o Shows pairwise relationships between many variables
• Correlation Heatmap:
o Displays correlation between all numerical variables
o Helps find strong or weak relationships
• 3D Scatter Plot:
o Shows relationship among three numerical variables
o Useful for detecting clusters or patterns
• Grouped Boxplots / Violin Plots:
o Compare distribution across many categories
o Shows differences between groups using multiple
variables
• Facet Grids (Subplots):
 o Multiple small plots split by category
o Good for comparing many groups at once

• Multivariate EDA Non-Graphical

Multivariate EDA is used when we study three or more
variables together to understand complex relationships. In real-
world data, many factors influence each other at the same time,
so multivariate analysis helps identify patterns, interactions, and
combined effects of multiple variables. It is especially important
when preparing data for machine learning models, which
usually depend on many input features.
• Correlation Matrix:
o Shows correlation between many variables at once
o Usually displayed as a heatmap
• Multivariate statistics:
o Includes methods like ANOVA, PCA, multiple regression
o Used to understand combined influence of several
variables
• Detection of multicollinearity:
o Identifies variables that are strongly related to each other
o Important for regression models

Data Science Process

The Data Science Process is a structured workflow followed by data
scientists to extract useful knowledge and build machine learning
solutions from raw data. It ensures that every project moves in a
systematic way—from understanding the problem to deploying the
final model. Although different organizations use different versions,
the core steps are similar.
The commonly accepted stages include: Business Understanding,
Data Understanding, Data Preparation, Exploratory Data
Analysis, Data Modeling, Model Evaluation, and Model
Deployment.

1. Business Understanding
The process begins with clearly understanding the business problem.
The goal is to identify what needs to be solved and why it matters. At
this stage, a project charter is created which defines objectives,
timelines, the scope of work, and expected outcomes. This ensures
that both the data team and stakeholders share the same vision and
success criteria.
2. Data Understanding
After defining the problem, the next step is to collect and explore the
available data. Data may come from databases, data warehouses, data
lakes, APIs, surveys, or web scraping. Here, data scientists examine
the quality of the data, identify missing values, detect outliers, and
understand the nature of variables. This helps to estimate whether the
data is sufficient to solve the problem.
3. Data Preparation
Most real-world data is unstructured, noisy, and incomplete.
Therefore, data preparation becomes one of the most critical phases. It
involves:
• Data Cleaning: Handling missing values, correcting errors,
removing duplicates.
• Data Integration: Combining multiple datasets.
• Data Transformation: Feature scaling, encoding categorical
variables, and creating new variables (feature engineering).
This step converts raw data into a structured and usable format
suitable for analysis and modeling.
4. Exploratory Data Analysis (EDA)
EDA helps uncover patterns, trends, and relationships in the data.
Both graphical and non-graphical techniques are used—such as
histograms, scatter plots, box plots, and correlation analysis. EDA
helps identify key factors that influence the target variable, detect
anomalies, and decide which modeling techniques might work best. It
also provides direction for feature selection and transformation.
5. Data Modeling
In this stage, machine learning or statistical models are developed to
learn patterns from the cleaned data. Depending on the problem,
various algorithms like linear regression, logistic regression, decision
trees, random forests, SVM, clustering, or neural networks may be
used. Multiple models are trained and their performance is compared
to select the best one. This step relies heavily on algorithm selection,
parameter tuning, and validation techniques.
6. Model Evaluation
Before deployment, the selected model must be evaluated using test
datasets. Evaluation metrics vary based on problem type—accuracy,
precision, recall, F1-score for classification; RMSE and MAE for
regression. Cross-validation techniques are used to measure how well
the model generalizes to unseen data. If performance is unsatisfactory,
the model is refined or data preparation is revisited.
7. Model Deployment
Once the model achieves acceptable performance, it is deployed into a
production environment where real users or systems can utilize it.
Deployment may be done through APIs, dashboards, or integrated
software applications. The model’s performance is continuously
monitored to detect drift or degradation. Feedback from the real world
helps improve the next iteration of the data science life cycle.

Motivation for Using Python for Data Analysis

Python has become one of the most preferred programming languages
for data analysis. Students, researchers, and data professionals widely
use it because it is simple, powerful, and supported by a large
ecosystem of tools. The following points explain why Python is the
ideal choice for data analysis.
1. Easy to Learn and Use
Python’s syntax is simple and readable. Even beginners with limited
programming experience can quickly start writing data analysis code.
The language feels close to English, making it easier to understand,
debug, and maintain.
2. Rich Ecosystem of Data Analysis Libraries
Python provides powerful, ready-made libraries specifically designed
for data analysis, such as:
• NumPy for numerical computations
• Pandas for data cleaning, manipulation, and analysis
• Matplotlib and Seaborn for data visualization
• SciPy for scientific and statistical analysis
• Scikit-learn for machine learning
These libraries reduce development time and help perform
complex tasks with very little code.
3. Strong Support for Data Visualization
Data analysis requires charts and graphs to understand patterns.
Python makes this easy with libraries such as Matplotlib, Seaborn,
and Plotly. These tools can create line plots, histograms, box plots,
heatmaps, and interactive dashboards.
4. Excellent Community and Documentation
Python has one of the largest developer and data science
communities. Most problems already have solutions available online.
Continuous updates, tutorials, and forums make learning and
troubleshooting easy.
5. High Compatibility with Data Formats
Python can easily read and process different types of data, such as:
• CSV, Excel, JSON, SQL databases
• Web APIs and HTML pages
• Big data formats like Parquet
This makes Python extremely flexible for handling real-world
datasets.
6. Integration with Machine Learning and AI
Data analysis often leads to predictive modeling. Python seamlessly
connects data analysis with machine learning using libraries like
Scikit-learn, TensorFlow, and PyTorch. This makes Python a complete
end-to-end solution—from cleaning data to building AI models.
7. Platform Independence
Python works on Windows, macOS, Linux, and cloud systems. Code
written on one machine runs easily on another, making collaboration
easier for teams.
8. Free and Open Source
Python and most of its libraries are free to use. Students and
professionals can access all tools without paying for licenses.

Introduction to Python Jupyter Notebook and Its Architecture

Introduction to Jupyter Notebook
Jupyter Notebook is an open-source, interactive computing
environment widely used in data science, machine learning, scientific
computing, and education. It allows users to create and share
documents that contain:
• Live executable Python code
• Text written in Markdown
• Visualizations and plots
• Equations (LaTeX)
• Interactive widgets
Jupyter provides a highly interactive workflow that supports data
cleaning, transformation, visualization, simulation, and model
building—all in one place.
Key Features of Jupyter Notebook
1. Interactive Coding – Execute code cell-by-cell and view output
immediately.
2. Rich Media Output – Supports charts, images, audio, HTML,
and more.
3. Reproducible Analysis – Combines code, results, and
explanations in a single document.
4. In-browser Environment – Runs directly in a web browser.
5. Supports Multiple Languages – Though widely used for
Python, it also supports R, Julia, and more via kernels.
Architecture of Jupyter Notebook
The internal functioning of Jupyter Notebook follows a client–server
architecture, designed to separate user interaction from code
execution.
Core Components
Jupyter Notebook consists of three main components:
a) Notebook Server
• A Python-based backend running locally or on a remote
machine.
• Responsible for:
o Handling user requests
o Managing file storage (saving .ipynb files)
o Providing notebook directory
o Starting/stopping kernels
• Acts as the bridge between the browser interface and the kernel.
b) Web-Based Notebook Interface (Client)
• The interface that users interact with in the web browser.
 • Displays:
o Cells (code and markdown)
o Output of code execution
o Tables, graphs, and widgets
• Sends code entered by the user to the notebook server, which
forwards it to the kernel.
c) Kernel
• A computational engine that executes the code.
• The IPython kernel is used for Python execution.
• Role of the kernel:
o Runs Python statements sent from the client
o Returns results (values, visuals, errors) back to the browser
o Maintains session state (variables, imported modules, etc.)

How It Works (Execution Flow)

1. The user writes Python code in a cell.
2. The browser sends the code to the Notebook Server.
3. The server forwards the code to the Python Kernel.
4. The kernel executes the code and returns:
o numeric results
o visualizations
o error messages
5. The browser displays the output below the cell.
This interaction continues until all cells are executed.
File Format (.ipynb)
Jupyter Notebook files use the .ipynb extension.
• These are JSON-based documents.
• They store:
o Code cells
o Markdown cells
o Metadata
o Execution history
The format makes notebooks easy to share, reproduce, and
upload to GitHub or cloud systems.

ESSENTIAL PYTHON LIBRARIES

1. NumPy (Numerical Python)
NumPy is the foundation of scientific computing in Python. Almost
every data-science library—pandas, SciPy, scikit-learn—internally
depends on NumPy’s powerful data structure called the ndarray (N-
Dimensional Array). It replaces slow Python lists with extremely fast
arrays optimized in C. NumPy allows you to store large amounts of
numerical data compactly and perform mathematical operations
(addition, subtraction, logarithms, trigonometry, matrix
multiplication, statistics, etc.) in a vectorized manner, meaning
operations run on whole arrays at once instead of using Python loops.
This makes NumPy hundreds of times faster for numerical work.
NumPy also supports random number generation, linear algebra
functions (dot product, inverse, eigenvalues), Fourier transforms, and
reshaping of arrays. Because of its speed, efficiency, and
mathematical power, NumPy is the first building block of all data
manipulation and machine learning tasks in Python.
2. pandas
pandas is the primary library for data manipulation, data cleaning,
and structured data analysis. While NumPy handles numerical
arrays, pandas introduces two powerful data structures:
• Series – a one-dimensional labeled array
• DataFrame – a two-dimensional tabular structure similar to
Excel spreadsheets or SQL tables
pandas makes real-world data preparation extremely easy. You can
load data from files (CSV, Excel, JSON), SQL databases, or online
sources. It supports removing missing values, fixing data types,
filtering rows, selecting columns, grouping data, merging datasets,
reshaping tables (pivot, melt), and creating new calculated columns.
Its integration with NumPy makes operations fast because the
DataFrame internally uses NumPy arrays. pandas is essential for
exploratory data analysis, preprocessing before machine learning, and
cleaning messy datasets. Almost all data analysis workflows start with
pandas.

3. matplotlib
matplotlib is Python’s fundamental plotting and visualization
library, used for creating a wide range of 2D graphs and charts. It
provides extremely fine control over the appearance of plots,
including colors, labels, scales, axes, annotations, and figure sizes.
With matplotlib, you can generate line charts, bar charts, scatter plots,
histograms, box plots, pie charts, and many more. The pyplot
interface (imported as plt) behaves like MATLAB’s plotting style,
making it easy to quickly draw graphs. Every other visualization
library—Seaborn, Plotly, pandas plotting—relies on matplotlib as the
core engine. While matplotlib can feel complex due to its
customization capability, its flexibility makes it a standard for
scientific visualization, research papers, technical reports, and
analytics dashboards.

4. Seaborn
Seaborn is a high-level statistical visualization library built on top
of matplotlib. It simplifies the process of creating visually appealing
and informative graphs. Seaborn automatically applies themes, color
palettes, legends, and layout settings that produce clean, professional-
looking charts with minimal code. It works beautifully with pandas
DataFrames because Seaborn understands column names directly. It is
mainly used for statistical plots such as distribution plots
(histograms, KDE curves), box plots, violin plots, pair plots,
heatmaps, regression plots, and categorical comparisons. Seaborn is
especially powerful for exploring relationships between variables
during EDA because it can visually represent statistical patterns using
simple commands. For exploratory analysis, Seaborn is more
convenient than matplotlib because it reduces code complexity while
improving aesthetics.

5. SciPy (Scientific Python)

SciPy extends the capabilities of NumPy by providing advanced
mathematical, scientific, and engineering tools. It is organized into
sub-packages such as:
• [Link] – statistical distributions, hypothesis tests,
percentiles
• [Link] – numerical integration and differential
equations
• [Link] – signal processing tools (filters, convolution)
SciPy contains highly optimized algorithms for functions that would
be extremely difficult to implement manually. Its role is deeper
scientific computation—more sophisticated than basic NumPy math.
Although SciPy contains a statistics module, it is more focused on
computational tools, while Statsmodels focuses on statistical
modeling. SciPy is widely used in machine learning preprocessing,
engineering simulations, physics calculations, bioinformatics, and
numerical optimization.
6. Statsmodels
Statsmodels is a library specifically developed for statistical
modeling, inference, and hypothesis testing. It is more statistics-
focused than scikit-learn. The key feature of Statsmodels is that it
provides detailed statistical outputs such as:
• p-values
• t-statistics
• R² and adjusted R²
Statsmodels is used for:
• Linear Regression (with statistical interpretation)
• Logistic Regression
• ANOVA
• Hypothesis tests (t-test, chi-square test, normality tests)
While scikit-learn focuses on predictive accuracy, Statsmodels
focuses on understanding why variables affect the outcome. It is
useful in fields like economics, research, social sciences, finance, and
academic modeling.

7. scikit-learn
scikit-learn is Python’s most widely used machine learning library.
It provides a simple and consistent interface for building, training, and
evaluating machine-learning models. scikit-learn includes:
Supervised Learning
• Classification algorithms (Decision Trees, SVM, KNN, Logistic
Regression)
• Regression algorithms (Linear Regression, Random Forest
Regression)
Unsupervised Learning
• Clustering (K-Means, DBSCAN)
• Dimensionality reduction (PCA)
Model Evaluation and Validation
• Accuracy, precision, recall, R², confusion matrix
• Cross-validation techniques
scikit-learn is optimized, easy to use, and extremely reliable for
building ML models in industry and
Installation & Import Commands for Essential Python Libraries
Library Purpose Installation Import Command
Name (pip)
NumPy Arrays & pip install import numpy as np
numerical numpy
computations
pandas Data cleaning pip install import pandas as pd
& dataframes pandas
matplotlib Data pip install import
visualization matplotlib [Link] as plt
(basic plots)
Seaborn Statistical & pip install import seaborn as sns
stylish seaborn
visualizations
SciPy Scientific & pip install import scipy
mathematical scipy or from scipy import
computations stats, linalg, optimize
scikit-learn Machine pip install import sklearn
learning scikit-learn or specific modules
algorithms like:
from
sklearn.model_selection
import train_test_split
Statsmodels Statistical pip install import [Link]
modeling & statsmodels as sm
tests
10-MARK QUESTIONS
1. Explain the Data Science Process in detail and describe the role
of Exploratory Data Analysis (EDA) in it.
2. Describe the architecture and features of the Jupyter Notebook.
Why is it widely used for Python-based data analysis?
3. Discuss the core functionalities of NumPy, pandas, matplotlib,
SciPy, scikit-learn, statsmodels, and seaborn in data analysis.
5-MARK QUESTIONS
1. What is Data Science? Briefly explain its importance and
applications.
2. Why is Python preferred for Data Analysis? Give at least four
reasons.
3. Write short notes on NumPy and pandas with simple examples
of their use.
4. What is seaborn? How is it different from matplotlib? Give any
two advantages.