Data Science – Exam Ready Notes

Definition

• Data Science is an interdisciplinary field that uses scientific methods, processes, algorithms, and systems to
extract knowledge and insights from structured and unstructured data.

• It combines statistics, mathematics, programming, machine learning, and domain knowledge.

Steps in Data Science Process

1. Problem Definition – Understanding the business/research question.

2. Data Collection – Gathering data from multiple sources (databases, APIs, web scraping, sensors, logs).

3. Data Cleaning & Preprocessing – Handling missing values, removing duplicates, scaling, normalization.

4. Exploratory Data Analysis (EDA) – Using statistical & visualization techniques to understand data patterns.

5. Feature Engineering – Creating new variables/features to improve model performance.

6. Model Building – Applying ML/DL algorithms (Regression, Classification, Clustering, etc.).

7. Model Evaluation – Checking accuracy, precision, recall, F1-score, RMSE.

8. Deployment – Integrating the model into real-world applications.

9. Monitoring & Maintenance – Ensuring the model works correctly over time.

How Data Scientists Work

Key Roles of a Data Scientist

• Data Collection → Gather raw data from multiple sources.

• Data Wrangling → Clean and preprocess data for analysis.

• Statistical Analysis → Identify patterns, correlations, and trends.

• Machine Learning & AI → Build predictive and prescriptive models.

• Visualization & Communication → Use tools like Tableau, PowerBI, Matplotlib, Seaborn to explain results.

Real-World Applications of Data Science

1. Healthcare

• Disease prediction (e.g., cancer detection, diabetes risk).

• Medical image analysis (X-ray, MRI using Deep Learning).

• Personalized medicine & drug discovery.

Example: IBM Watson Health helps doctors in diagnosis and treatment recommendation.
2. Finance & Banking

• Fraud detection in transactions.

• Credit scoring and loan approval.

• Algorithmic trading and risk management.

Example: Banks use machine learning to flag suspicious credit card activity.

3. E-Commerce & Retail

• Product recommendation systems (Amazon, Flipkart).

• Customer segmentation and personalized offers.

• Inventory demand forecasting.

Example: Netflix & Amazon use recommendation engines to suggest products/movies.

5. Social Media & Marketing

• Sentiment analysis of tweets, reviews, posts.

• Targeted digital marketing (Facebook, Instagram Ads).

• Fake news and spam detection.

Example: Twitter uses AI to detect hate speech and spam accounts.

Web Scraping

Definition

• Web Scraping is the process of automatically extracting data from websites using tools, scripts, or libraries.

• It helps to collect large amounts of data for analysis, research, and machine learning applications.

Steps of Web Scraping

1. Identify Target Website & Data

• Decide which website to scrape.

• Example: Scraping product details from Amazon (name, price, reviews).

2. Inspect the Web Page Structure

• Use HTML tags, CSS selectors, XPath to locate data.

• Example: <div class="price">₹499</div> contains price info.

• Use a library (e.g., requests in Python) to send a request to the website and fetch the HTML source code.

4. Parse the HTML Content

• Extract the required data from HTML using tools like:

o BeautifulSoup (Python)

o lxml

o Scrapy Framework

5. Extract Data

• Locate and extract required information (text, images, links).

• Example: Extract product name, price, rating.

6. Store the Data

• Save data in CSV, Excel, JSON, or Database for analysis.

7. Data Cleaning & Analysis

• Remove duplicates, missing values, or irrelevant data.

• Analyze the dataset using Pandas, NumPy, ML models.

Big Data

Definition

• Big Data refers to extremely large and complex datasets that cannot be managed, processed, or analyzed using
traditional data processing tools.

• It is often described by the 5 V’s:

o Volume (huge amount of data)

o Velocity (speed of data generation)

o Variety (different types of data)

o Veracity (uncertainty/accuracy of data)

Types of Data in Big Data

1. Structured Data

• Data stored in rows and columns (organized, relational).

• Easy to store, query, and analyze.

• Example:

o Banking transactions

o Student records (Name, Roll No., Marks)

o SQL Databases

2. Unstructured Data

• Data without predefined format or organization.

• Difficult to analyze with traditional tools.

• Example:

o Social media posts (Facebook, Twitter)

o Emails, videos, images, audio files

o Sensor data

3. Semi-Structured Data

• Data that is not in a strict tabular format but has some organizational tags/structure.

• Falls between structured and unstructured.

• Example:

o XML, JSON files

o Log files

o HTML pages

Traits of Big Data

Big Data is commonly described using the 5 V’s (sometimes extended to 7 V’s).

1. Volume
 • Refers to the huge amount of data generated every second.

• Example: Facebook generates petabytes of data daily.

2. Velocity

• The speed at which data is generated, collected, and processed.

• Example: Stock market data, online transactions, IoT sensor data in real-time.

3. Variety

• Different forms of data: structured, semi-structured, and unstructured.

• Example: Text, images, audio, video, social media posts, GPS signals.

4. Veracity

• Refers to the quality, accuracy, and reliability of data.

• Example: Fake news or incorrect entries reduce data trustworthiness.

5. Value

• The usefulness of data in decision-making.

• Example: Amazon uses customer purchase data to recommend products.

1. Matplotlib

Definition:
Matplotlib is a data visualization library in Python used for creating static, interactive, and animated plots.

Explanation & Features:

• Provides plots such as line, bar, scatter, histogram, pie charts.

• Highly customizable (labels, colors, styles).

• Often used with NumPy and Pandas for plotting.

Example (Exam Answer):

import [Link] as plt

x = [1,2,3,4,5]

y = [2,4,6,8,10]

[Link](x,y)
[Link]("X-axis")

[Link]("Y-axis")

[Link]("Line Plot Example")

[Link]()

2. NumPy (Numerical Python)

Definition:
NumPy is a fundamental scientific computing library in Python, mainly used for numerical operations, linear algebra,
and arrays.

Explanation & Features:

• Provides multi-dimensional arrays (ndarray) that are faster than Python lists.

• Supports mathematical operations (addition, multiplication, matrix operations).

• Backbone for libraries like Pandas, SciPy, Scikit-learn.

Example (Exam Answer):

import numpy as np

arr = [Link]([1,2,3,4,5])

print("Array:", arr)

print("Mean:", [Link](arr))

print("Standard Deviation:", [Link](arr))

3. Scikit-learn (sklearn)

Definition:
Scikit-learn is a machine learning library in Python that provides tools for data mining, analysis, and predictive
modeling.

Explanation & Features:

• Supports supervised learning (classification, regression).

• Supports unsupervised learning (clustering, dimensionality reduction).

• Provides utilities for data preprocessing, model evaluation, and feature selection.

Example (Exam Answer):

from sklearn.linear_model import LinearRegression

import numpy as np

X = [Link]([[1],[2],[3],[4],[5]])
y = [Link]([2,4,6,8,10])

model = LinearRegression()

[Link](X,y)

print("Prediction for 6:", [Link]([[6]]))

Data Visualization

Definition

• Data Visualization is the graphical representation of data using charts, graphs, and plots to make patterns,
trends, and insights easier to understand.

1. Bar Chart

• Use: Compare categories of data.

• Example: Comparing sales of products A, B, C.

Python Example

import [Link] as plt

# Data

products = ['A', 'B', 'C']

sales = [150, 230, 180]

# Bar Chart

[Link](products, sales, color='skyblue')

[Link]("Product Sales")

[Link]("Products")

[Link]("Sales")

[Link]()

2. Line Chart
 • Use: Show trends over time.

• Example: Temperature variation across days.

Python Example

import [Link] as plt

# Data

days = [1, 2, 3, 4, 5]

temperature = [30, 32, 33, 31, 29]

# Line Chart

[Link](days, temperature, marker='o', color='green')

[Link]("Temperature over Days")

[Link]("Days")

[Link]("Temperature (°C)")

[Link]()

3. Scatter Plot

• Use: Show relationship between two variables.

• Example: Hours studied vs Exam score.

Python Example

import [Link] as plt

# Data

hours = [2, 4, 6, 8, 10]

scores = [50, 55, 65, 70, 85]

# Scatter Plot

[Link](hours, scores, color='red')

[Link]("Hours Studied vs Exam Score")

[Link]("Hours Studied")

[Link]("Exam Score")

[Link]()
Python Program (Exam-Ready)

import [Link] as plt

# Sample data

x = [1, 2, 3, 4, 5]

y1 = [10, 20, 25, 30, 40] # for bar chart

y2 = [5, 15, 20, 25, 35] # for line chart

y3 = [7, 18, 22, 28, 38] # for scatter plot

# Create plots

[Link](figsize=(15, 5))

# Bar Chart

[Link](1, 3, 1)

[Link](x, y1, color='skyblue')

[Link]("Bar Chart Example")

[Link]("X-axis")

[Link]("Y1 Values")

# Line Chart

[Link](1, 3, 2)

[Link](x, y2, marker='o', linestyle='-', color='green')

[Link]("Line Chart Example")

[Link]("X-axis")

[Link]("Y2 Values")

# Scatter Chart

[Link](1, 3, 3)

[Link](x, y3, color='red', marker='x')

[Link]("X-axis")

[Link]("Y3 Values")

plt.tight_layout()

[Link]()