Introduction to Data Science

Data Science: Data Science is the study of data using statistical, computational,
and analytical techniques to extract insights and support decision-making.
Components of Data Science:
1. Data Collection
o Gathering raw data from various sources (databases, sensors, web,
social media).
2. Data Cleaning & Preparation
o Removing errors, handling missing values, and converting data
into usable formats.
3. Exploratory Data Analysis (EDA)
o Using statistics and visualization to understand patterns, trends,
and relationships in data.
4. Modelling & Algorithms
o Applying machine learning and statistical models to make
predictions or classifications.
5. Data Visualization
o Presenting results through charts, graphs, and dashboards for easy
interpretation.
6. Communication & Decision Making
o Explaining insights clearly to stakeholders and supporting business
or academic decisions.
7. Deployment & Monitoring
o Implementing models into real-world systems and continuously
checking their performance.
Need of Data Science:
1. Handling Big Data
o Modern businesses generate huge volumes of structured and
unstructured data. Data Science helps manage and analyze this
effectively.
2. Better Decision-Making
o Provides insights that guide strategic, financial, and operational
decisions.
3. Prediction & Forecasting
o Enables future trend prediction using machine learning models
(e.g., sales forecasting, weather prediction).
4. Personalization
o Powers recommendation systems (Netflix, Spotify, Amazon) by
analysing user preferences.
5. Problem-Solving Across Domains
o Applied in healthcare (diagnosis), finance (fraud detection),
education (student performance analysis), and many more.

Evolution of Data Science:

Data Science has developed gradually over centuries, evolving from basic
statistics into today’s advanced AI-driven discipline.
Key Stages of Evolution
1. Early Statistics (17th–18th Century)
o Foundations laid by pioneers like John Graunt (mortality tables)
and Carl Friedrich Gauss (Gaussian distribution).
o Statistics used for census, trade, and agriculture.

2. Birth of Data Science (1960s–1980s)

o The term Data Science first appeared in the 1960s.
o Computers enabled data storage, mining, and statistical modelling.
 3. Machine Learning Era (1990s–2000s)
o Rise of algorithms like decision trees, clustering, and neural
networks.
o Internet growth led to massive digital data generation.

4. Big Data & Modern Data Science (2010s–Present)

o Technologies like Hadoop and Spark enabled large-scale data
processing.
o Integration with AI, IoT, and cloud computing.
o Applications expanded to healthcare, finance, retail, and education.

Data Science Process: The Data Science Process is a structured workflow that
guides how raw data is transformed into meaningful insights. It ensures
systematic handling of data from collection to decision-making.
Types of Data Science process:
1. Predictive Data Science Process
o Focuses on predicting future outcomes using historical data.
o Example: Sales forecasting, weather prediction.
2. Descriptive Data Science Process
o Summarizes past data to understand patterns and trends.
o Example: Analyzing customer purchase history.
3. Diagnostic Data Science Process
o Explains why something happened by finding root causes.
o Example: Identifying reasons for a drop in website traffic.
4. Prescriptive Data Science Process
o Suggests actions or solutions based on data analysis.
o Example: Recommending marketing strategies to increase sales.
 5. Exploratory Data Science Process
o Used when data is new or unstructured, to discover hidden
patterns.
o Example: Social media sentiment analysis.

Data Science Process in Business Intelligence

Steps in the Data Science Process for BI
1. Data Collection
o Gathering business data from sources like sales records, customer
databases, social media, and financial systems.
2. Data Cleaning & Preparation
o Removing duplicates, correcting errors, and standardizing formats
to ensure reliable analysis.
3. Exploratory Data Analysis (EDA)
o Identifying trends, customer behaviour, and market patterns using
statistical summaries and visualizations.
4. Model Building & Analytics
o Applying predictive models (e.g., sales forecasting, churn
prediction) and descriptive analytics to understand performance.
5. Evaluation
o Checking accuracy of models and ensuring insights align with
business goals.
6. Visualization & Reporting
o Creating dashboards, charts, and reports for managers and
stakeholders to make informed decisions.
7. Deployment & Monitoring
o Integrating insights into BI tools (like Power BI, Tableau) and
continuously monitoring for updates.
Prerequisites for a Data Scientist:
To become a Data Scientist, certain skills and knowledge areas are considered
essential.
Some of the Key Prerequisites:
1. Mathematics & Statistics
o Strong foundation in probability, linear algebra, and statistical
methods.
o Essential for understanding data distributions, hypothesis testing,
and model evaluation.
2. Programming Skills
o Knowledge of languages like Python, R, SQL.
o Ability to write code for data manipulation, analysis, and machine
learning.
3. Data Handling Skills
o Understanding of databases, data cleaning, and preprocessing.
o Familiarity with tools like Excel, Pandas, NumPy.
4. Machine Learning Basics
o Knowledge of algorithms such as regression, classification,
clustering, and neural networks.
o Ability to apply models to real-world datasets.
5. Data Visualization
o Skills in presenting insights using charts, graphs, and dashboards.
o Tools: Matplotlib, Seaborn, Power BI, Tableau.
6. Domain Knowledge
o Understanding the business or academic context where data is
applied.
o Helps in interpreting results meaningfully.
7. Communication Skills
 o Ability to explain technical results in simple terms to non-technical
stakeholders.

Tools Required:
Category Tools/Technologies Purpose
Data analysis, modelling,
Programming Python, R, SQL
queries
Cleaning, preprocessing,
Data Handling Pandas, NumPy
manipulation
Scikit-learn, TensorFlow,
Machine Learning Building predictive models
PyTorch
Matplotlib, Seaborn, Tableau,
Visualization Charts, dashboards, reports
Power BI
Big Data & Storage Hadoop, Spark, MongoDB Large-scale data processing
Cloud Platforms AWS, Azure, Google Cloud Scalable data science solutions

Applications of Data Science in various fields:

Healthcare System:
 Disease prediction and diagnosis.
 Personalized medicine and drug discovery.
 Example: AI-based cancer detection.
Finance & Banking System:
 Fraud detection and risk management.
 Customer credit scoring.
 Example: Detecting unusual transactions in real-time.
Retail & E-Commerce:
 Recommendation systems (Amazon, Flipkart).
 Customer behaviour analysis.
  Example: Suggesting products based on browsing history.

Education System:
 Student performance analysis.
 Adaptive learning systems.
 Example: Predicting dropout risks.
Transportation & Logistics
 Route optimization and traffic prediction.
 Autonomous vehicles.
 Example: Uber using data for ride pricing and demand forecasting.

Data Security Issues in Data Science:

Data Science deals with massive amounts of sensitive data, and ensuring its
security is a major challenge.
Some of the major data security issues are:
Data Privacy
 Personal information (like health records, financial details) can be
exposed if not protected.
 Example: Leakage of customer data from e-commerce sites.
Data Breaches
 Large-scale theft of sensitive data due to cyberattacks.
 Example: Banking fraud through stolen transaction records.
Data Integrity
 Alteration or manipulation of data can lead to wrong analysis and
decisions.
 Example: Tampered medical records affecting diagnosis.
Data Storage & Transmission Risks
 Improper encryption during storage or transfer can expose data.
  Example: Unsecured cloud storage leading to leaks.

Insider Threats
 Employees misusing access to sensitive datasets.
 Example: Selling confidential company data.

Data Collection Strategies:

In Data Science, data collection is the first and most crucial step. The quality of
collected data directly affects the accuracy of analysis and decision-making.
Major Strategies
1. Surveys & Questionnaires
o Collecting structured responses from individuals.
o Example: Online Google Forms for customer feedback.
2. Observation Method
o Recording data by observing behaviours or events.
o Example: Tracking user activity on a website.
3. Transaction Records
o Using existing business or institutional records.
o Example: Sales invoices, student attendance logs.
4. Web Scraping:
o Extracting data from websites using automated tools.
o Example: Collecting product prices from e-commerce sites.
5. APIs (Application Programming Interfaces):
 Accessing data from platforms like Twitter, YouTube, or weather
services.
 Example: Twitter API for sentiment analysis.
6. Sensor & IoT Devices
 Collecting real-time data from connected devices.
  Example: Smartwatches recording health data.

Data Pre-Processing Overview.

Data pre-processing is the stage in Data Science where raw data is transformed
into a clean and usable format before analysis. Since real-world data is often
messy, incomplete, or inconsistent, pre-processing ensures accuracy and
reliability of results.

Key Steps in Data Pre-Processing

1. Data Cleaning
o Handling missing values, removing duplicates, correcting errors.
o Example: Filling missing student marks with average values.
2. Data Integration
o Combining data from multiple sources into a single dataset.
o Example: Merging student attendance records with exam scores.
3. Data Transformation
o Converting data into suitable formats (normalization, scaling,
encoding).
o Example: Converting categorical values like “Yes/No” into 1/0.
4. Data Reduction
o Minimizing data size without losing important information.
o Example: Selecting only relevant attributes for analysis.
5. Data Discretization
o Converting continuous data into discrete categories.
o Example: Grouping ages into ranges (18–25, 26–35, etc.).

Data Cleaning:
Data Cleaning is the process of detecting and correcting errors, inconsistencies,
and missing values in datasets to improve their quality and reliability before
analysis. It is a crucial step in Data Pre-processing.
Key Tasks in Data Cleaning
1. Handling Missing Values
o Techniques: Removing records, replacing with mean/median, or
using predictive methods.
o Example: Filling missing student marks with average scores.
2. Removing Duplicates
o Ensures each record is unique.
o Example: Eliminating repeated entries in a student database.
3. Correcting Inconsistencies
o Standardizing formats (e.g., date formats, spelling).
o Example: Converting “01-06-26” and “June 1, 2026” into a single
format.
4. Outlier Detection & Treatment
o Identifying extreme values that may distort analysis.
o Example: A student’s attendance recorded as 500 days in a year.
5. Data Validation
o Ensuring values fall within acceptable ranges.
o Example: Age field should not contain negative numbers.
6. Noise Removal
o Filtering irrelevant or random data.
o Example: Removing spam entries in survey responses.

Data Integration:
Data Integration is the process of combining data from multiple sources into a
single, unified view. It ensures that information is consistent, accurate, and
ready for analysis in Data Science.
Key Aspects of Data Integration
1. Combining Multiple Sources
o Merging data from databases, files, APIs, or sensors.
o Example: Integrating student attendance records with exam results.
2. Schema Integration
o Aligning different data formats and structures.
o Example: Converting “DOB” in one dataset and “Date of Birth” in
another into a common format.
3. Data Consistency
o Resolving conflicts (e.g., different spellings, units, or formats).
o Example: Standardizing “INR” and “Rs.” into one currency format.
4. Data Redundancy Removal
o Eliminating duplicate or overlapping records.
o Example: Removing repeated customer entries across branches.
5. Data Quality Improvement
o Ensuring integrated data is clean, accurate, and reliable.

Data Transformation:
Data Transformation is the process of converting data into suitable formats for
analysis. It ensures that data is consistent, standardized, and ready for modelling
in Data Science.
Key Techniques in Data Transformation
1. Normalization & Scaling
o Adjusting values to a common scale without distorting differences.
o Example: Converting student marks (0–100) into a scale of 0–1.
2. Encoding Categorical Data
o Converting non-numeric values into numeric form.
 o Example: Changing “Yes/No” into 1/0 or “Male/Female” into 0/1.
3. Aggregation
o Summarizing data into meaningful groups.
o Example: Calculating average sales per month from daily records.
4. Feature Engineering
o Creating new variables from existing data to improve model
performance.
o Example: Deriving “BMI” from height and weight data.
5. Data Standardization
o Ensuring consistent formats across datasets.
o Example: Converting dates into a single format (DD/MM/YYYY).
6. Log Transformation
o Applying mathematical transformations to reduce skewness.
o Example: Using log scale for income data to handle extreme
values.

Data Reduction:
Data Reduction is the process of minimizing the volume of data while
preserving its essential information. It helps improve efficiency, reduce storage
requirements, and speed up analysis in Data Science.
Key Techniques in Data Reduction
1. Dimensionality Reduction
o Reducing the number of attributes/features in a dataset.
o Example: Using Principal Component Analysis (PCA) to simplify
student performance data.
2. Numerosity Reduction
o Replacing large datasets with smaller, representative forms.
o Example: Representing sales data with histograms or regression
models instead of raw records.
 3. Data Compression
o Encoding data to reduce storage size.
o Example: Compressing image datasets using JPEG format.
4. Aggregation
o Summarizing data into higher-level groups.
o Example: Converting daily attendance records into monthly
averages.
5. Sampling
o Selecting a subset of data for analysis instead of the entire dataset.
o Example: Analysing 1,000 student responses instead of all 10,000.

Data Discretization:
Data Discretization is the process of converting continuous data into discrete
categories or intervals. It simplifies complex datasets, makes them easier to
analyse, and is often used in machine learning and data mining.
Key Techniques in Data Discretization
1. Binning (Equal-Width / Equal-Frequency)
o Divides continuous values into fixed intervals.
o Example: Grouping student marks into ranges (0–20, 21–40, 41–
60, etc.).

2. Histogram Analysis
o Uses frequency distribution to create categories.
o Example: Categorizing ages based on their frequency distribution.

3. Clustering-Based Discretization
o Groups values into clusters using algorithms like k-means.
 o Example: Grouping customers into “low-spending,” “medium-
spending,” and “high-spending”.
4. Decision Tree-Based Discretization
o Splits data into categories based on decision rules.
o Example: Splitting income data into “eligible” and “not eligible”
for a loan.

5. Manual Discretization
o Categories defined by domain experts.
o Example: Blood pressure classified as “Low,” “Normal,” “High”.

Data Munging:
Data Munging is the process of cleaning, restructuring, and enriching raw data
into a usable format for analysis. Since real-world data is often messy, munging
ensures it becomes consistent and ready for modelling.
Key Steps in Data Munging
1. Data Extraction
o Collecting raw data from multiple sources (databases, APIs, files).
2. Data Cleaning
o Handling missing values, duplicates, and inconsistencies.
o Example: Correcting spelling errors in student names.
3. Data Transformation
o Converting data into suitable formats (normalization, encoding).
o Example: Changing “Yes/No” into 1/0.
4. Data Enrichment
o Adding external or derived information to make data more
meaningful.
o Example: Adding “BMI” column using height and weight data.
5. Data Validation
 o Ensuring accuracy, consistency, and reliability of the dataset.
o Example: Checking if age values fall within a valid range.
Data Filtering:
Filtering in Data Science is the process of selecting specific data from a dataset
based on defined conditions or criteria. It helps reduce irrelevant information and
focus only on the data that is useful for analysis.
Key Aspects of Filtering
1. Row Filtering (Record Selection)
o Selecting rows that meet certain conditions.
o Example: Filtering student records where marks > 75.
2. Column Filtering (Attribute Selection)
o Choosing only relevant attributes/features.
o Example: Selecting only “Name” and “Marks” columns from a
student dataset.
3. Conditional Filtering
o Applying logical conditions to extract data.
o Example: Filtering employees with salary between 30,000 and
50,000.
4. Noise Filtering
o Removing irrelevant or random data.
o Example: Eliminating spam responses in survey data.
5. Time-Based Filtering
o Selecting data within a specific time range.
o Example: Filtering sales records from January to March 2026.