Python Libraries for Data

Science
December 2023

- by Ashish Gurjar
 Python's Role in Data Science
Overview
Significance of Python Libraries

Scrapy: Overview
Data Mining Libraries Beautiful Soup: Overview
Comparison: Scrapy vs. Beautiful Soup

Data Processing & Modeling Overview of NumPy, SciPy, Pandas,

TensorFlow, Keras, and their roles in data
Libraries analysis and machine learning (ML)

AGENDA Data Visualization Libraries

Introduction to the functionalities of
Matplotlib, Plotly, and Pydot for data
visualization

Discussing the features and uses of

Additional Libraries Seaborn and Scikit-learn in data science

Conclusion and Appendix

Python's Role in Data Science
• User-Friendly Language:
• Analogy: Python is like a versatile Swiss Army knife for data scientists — easy to use and adaptable for
various tasks.
• Example: Its syntax is intuitive and readable, making coding more approachable for beginners and
efficient for experts.
• Versatility in Applications:
• Analogy: Think of Python as a multi-genre artist, adept in various data science fields from analytics to
machine learning.
• Example: Used in a broad range of applications, including web development, automation, and scientific
modeling.
• Community and Collaboration:
• Analogy: Python's community is a thriving ecosystem, similar to a bustling marketplace where ideas and
solutions are freely exchanged.
• Example: A vast array of forums, documentation, and open-source projects available for learning and
collaboration.
• Integration Capabilities:
• Analogy: Python acts as a bridge, connecting various data sources and tools, much like a universal
adapter.
• Example: Seamless integration with other languages and tools, such as R for statistical analysis or SQL for
database management.
Significance of Python Libraries
• Rich Library Ecosystem:
• Analogy: Python libraries are like specialized tools in a data scientist's toolkit, each designed for specific
tasks.
• Example: Libraries like NumPy and Pandas for data manipulation, Matplotlib and Seaborn for data
visualization.
• Streamlining Data Analysis:
• Analogy: Using Python libraries for data analysis is akin to having a personal assistant who takes care of
repetitive tasks efficiently.
• Example: Pandas simplify data cleaning and exploration, allowing more time for data interpretation and
insights.
• Facilitating Advanced Machine Learning:
• Analogy: Libraries like TensorFlow and Scikit-learn are akin to building blocks for constructing
sophisticated machine learning models.
• Example: TensorFlow enables building and training complex neural networks, while Scikit-learn offers
tools for various machine learning algorithms.
• Open-Source Advantage:
• Analogy: Python libraries are akin to a community garden, nurtured and grown by contributions from
users worldwide.
• Example: Continuous improvements and updates from a diverse and active community, ensuring
robustness and cutting-edge features.
Data Mining Libraries
• Scrapy:
• Beautiful Soup
• Comparison:
Scrapy vs. Beautiful Soup
 Scrapy is an open-source web
crawling framework for Python

Introduction Designed for data mining and

automated web scraping
to Scrapy
Enables extraction of
structured data from websites
 Creating new projects and
spiders

Basic
Operations Defining how to follow links
and extract data
of Scrapy
Storing scraped data in various
formats
 Ideal for large-scale web scraping
and complex data extraction tasks

Real-Life Used in market research to gather

Applications consumer and competitor insights
of Scrapy
Employed for automated monitoring
of product prices and availability
across different websites
 import scrapy

Scrapy class QuotesSpider([Link]):

name = "quotes"
start_urls = [
Code ]
'[Link]

Example def parse(self, response):

for quote in [Link]('[Link]'):
yield {
This spider scrapes quotes and authors
from a quote's website.
'text': [Link]('[Link]::text').get(),
'author': [Link]('span/small/text()').get(),
}
 • {'text': '“The world as we have created it is a
process of our thinking. It cannot be changed
without changing our thinking.”', 'author': 'Albert
Scrapy Einstein'}
• {'text': '“It is our choices, Harry, that show what we
Output truly are, far more than our abilities.”', 'author':
'J.K. Rowling'}
Example
• Output Sample: Displays scraped quotes and their
authors
Scrapy in the Industry

Companies: Widely
Departments/Roles: Used
adopted in tech
primarily by Data Analysts,
companies, marketing
Web Developers, and
agencies, and academic
Researchers
research for data collection
 Beautiful Soup is a Python library for
parsing HTML and XML documents

Introduction to It creates parse trees that are

Beautiful Soup helpful to extract the data easily

Often used for web scraping

Basic Operations of Beautiful Soup

01 02 03
Navigating the parse Modifying the tree Parsing and printing
tree (e.g., searching (e.g., changing, documents.
by tags, navigating adding, and deleting
using tag names). tags).
 Extracting data from HTML content
for web scraping
Real-Life
Applications of
Beautiful Soup Screen scraping, which involves
extracting data from websites

Useful in gathering social media

content, news articles, and historical
data from websites
 from bs4 import BeautifulSoup
import requests

Beautiful URL = "[Link]

page = [Link](URL)
Soup Code soup = BeautifulSoup([Link], "[Link]")

Example quotes = soup.find_all("div", class_="quote")

for quote in quotes:
This example scrapes quotes and authors text = [Link]("span", class_="text").text
from the same quote's website.
author = [Link]("small", class_="author").text
print(f"{text} - {author}")
 • “The world as we have created it is
a process of our thinking. It cannot
Beautiful be changed without changing our
thinking.” - Albert Einstein

Soup • “It is our choices, Harry, that show

what we truly are, far more than

Output our abilities.” - J.K. Rowling

Example • Displays extracted quotes and their

authors.
 Feature Scrapy Beautiful Soup
Comparison: Library for parsing HTML and
Scrapy vs. Type Web crawling framework
XML documents

Beautiful Soup Use Case

Ideal for large-scale data Best for small-scale projects
extraction and web scraping and simple scraping tasks

• Scrapy vs. Beautiful Soup Learning Curve

Steeper, due to its framework
Easier to learn for beginners
nature

• Note: The comparison Data Handling

Built-in support for output Requires manual handling of
highlights formats like JSON, CSV data extraction
fundamental
differences to help Commonly used in
Can be integrated with other
Integration conjunction with requests for
choose the right tool Python libraries
web requests
based on the project's
scale and complexity More suited for complex and
More flexibility in parsing
Flexibility and extraction of data from
vast web scraping projects
HTML/XML
Data Processing & Modelling Libraries
• NumPy
• SciPy
Comparison:
NumPy vs. SciPy
• Pandas
• TensorFlow
• Keras
Comparison:
Keras vs. TensorFlow
 NumPy is a fundamental
package for scientific
computing in Python

Introduction Provides support for large,

multi-dimensional arrays and
to NumPy matrices

Includes a large collection of

mathematical functions to
operate on these arrays
 Array creation, manipulation,
and indexing

Basic Mathematical operations like

Operations linear algebra, statistics, and
of NumPy trigonometry
Random number generation
 Essential in numerical
computations and data analysis

Real-Life Widely used in fields like physics,

Applications engineering, and machine learning
for modeling and simulations

of NumPy
Key in image and signal processing
applications
NumPy Code Example
input
output
import numpy as np

arr = [Link]([1, 2, 3, 4, 5]) Array: [1 2 3 4 5]

print("Array:", arr) Mean of the array: 3.0
print("Mean of the array:", [Link](arr))

import numpy as np

# Creating an array
arr = [Link]([1, 2, 3, 4, 5])
Addition Result: [3 4 5 6 7]
# Element-wise addition
add_result = arr + 2
print("Addition Result:", add_result)
NumPy Code Example
input output
# Matrix multiplication
mat1 = [Link]([[1, 2], [3, 4]]) Matrix Product:
mat2 = [Link]([[5, 6], [7, 8]]) [[19 22]
product = [Link](mat1, mat2) [43 50]]
print("Matrix Product:\n", product)

# Statistical operations
data = [Link]([1, 2, 3, 4, 5]) Mean: 3.0
print("Mean:", [Link](data)) Standard Deviation: 1.4142135623730951
print("Standard Deviation:", [Link](data))

# Random number generation

random_arr = [Link](5)
Random Array: [0.42, 0.65, 0.78, 0.34, 0.93] # Example output
print("Random Array:", random_arr)
 Departments/Roles: Primarily
used by Data Scientists,
Engineers, and Researchers
NumPy in
the
Companies: Extensively used
Industry in technology, finance, and
academic sectors for complex
data analysis and modeling
 SciPy is a Python-based ecosystem of open-
source software for mathematics, science,
and engineering

It builds on NumPy arrays and provides

Introduction many higher-level functions
to SciPy
Commonly used for tasks in data science
such as optimization, linear algebra,
integration, interpolation, special functions,
FFT, signal and image processing
 Linear algebra operations

Basic
Optimization and fit
Operations of
algorithms
SciPy

Signal processing tools

 Integral in solving scientific and
mathematical problems

Real-Life
Employed extensively in engineering,
Applications of physics, and computational biology
SciPy
Used for modeling and simulating
complex systems in various scientific
domains
 from scipy import integrate

SciPy # Defining a simple function

Code f = lambda x: x**2

# Integrating function f from 0 to 1
Example integration_result = [Link](f,
0, 1)
This example print("Integration Result:",
demonstrates the
integration of a simple
integration_result[0])
function.
 Output Sample:

SciPy Integration Result:

Output 0.33333333333333337
Example
Result of integrating the
function x power 2 from 0 to 1.
 Departments/Roles: Predominantly
used by Researchers, Data Scientists,
and Engineers

SciPy in the
Industry Companies: Utilized in sectors such
as aerospace, automotive,
telecommunications, and
environmental science for complex
analyses and modeling
 • Note: Both libraries are integral to scientific computing in
Comparison: Python, with NumPy providing the foundational array
structure and SciPy building upon it with more
NumPy vs. SciPy specialized functions

Feature NumPy SciPy

Array processing and basic mathematical Advanced mathematical functions and scientific
Primary Focus
operations computing
Basic operations like array manipulation and Extensive functions for optimization, linear algebra,
Functionality
indexing integration, etc.
Fundamental for any computation using arrays Ideal for more complex scientific calculations and
Use Case
and matrices advanced computations
Dependency Standalone, does not require SciPy Builds on and extends NumPy; often used together
Provides additional functionality suited for specific
Performance Optimized for array operations
scientific computations
Community & Strong community, particularly in scientific and research
Wide usage and strong community support
Support sectors
 Pandas is a powerful data manipulation and
analysis tool using its two primary data
structures: DataFrames and Series

Ideal for handling structured data and offers

Introduction extensive operations for data manipulation,
merging, reshaping, and aggregation
to Pandas
Widely used for data cleaning, preparation,
and analysis
 Data import and export (e.g.,
CSV, Excel).

Basic Data indexing, selection, and

Operations filtering.
of Pandas
Handling missing data, data
transformation, and aggregation.
 Essential in data preprocessing
for Machine Learning

Real-Life Used extensively in financial

Applications analysis, statistics, and analytics
of Pandas
Suitable for time-series data
analysis and manipulation
Pandas Code Examples
input output
Example 1: Creating and Manipulating a DataFrame
Name Age City
import pandas as pd 0 Anna 25 New York
1 Bob 30 Paris
data = {'Name': ['Anna', 'Bob', 'Charlie'], 2 Charlie 35 London
'Age': [25, 30, 35],
'City': ['New York', 'Paris', 'London']}
df = [Link](data)
print(df)

Example 2: Data Selection and Filtering

# Selecting rows based on condition Name Age City

older_than_30 = df[df['Age'] > 30] 2 Charlie 35 London
print(older_than_30)
Pandas Code Examples
input output

Example 3: Handling Missing Data

Name Age City Salary
# Adding a column with missing values 0 Anna 25 New York 70000.0
df['Salary'] = [Link]([70000, 80000, None]) 1 Bob 30 Paris 80000.0
# Filling missing values 2 Charlie 35 London 75000.0
df['Salary'].fillna(value=df['Salary'].mean(), inplace=True)
print(df)
Age
Example 4: Data Aggregation City
# Grouping and aggregating data London 35
grouped_data = [Link]('City').agg({'Age': 'mean'}) New York 25
print(grouped_data) Paris 30
 Pandas Code Examples
input output

Example 5: Time-Series Data Analysis A B C D

2023-01-01 -0.282863 1.213460 -1.083207 -0.128210
# Creating a time-series DataFrame 2023-01-02 0.176715 0.052541 -0.309056 1.049605
time_data = pd.date_range('20230101', periods=6) ... (4 more rows)
ts_df = [Link]([Link](6,4), index=time_data,
columns=list('ABCD'))
print(ts_df)

Example 6: Merging/Joining DataFrames

# Merging two DataFrames Name Age City Salary Population

other_data = [Link]({'City': ['New York', 'Paris'], 0 Anna 25 New York 70000.0 8000000
'Population': [8000000, 2148000]}) 1 Bob 30 Paris 80000.0 2148000
merged_df = [Link](df, other_data, on='City')
print(merged_df)
 Departments/Roles: Utilized by
Data Analysts, Business Analysts,
and Data Scientists
Pandas in
the Companies: Employed across
Industry various industries including
finance, healthcare, retail, and
technology for data analysis tasks
 TensorFlow is an open-source library
developed by Google for numerical
computation and machine learning
Introduction It uses data flow graphs for scalable
to machine learning tasks
TensorFlow
TensorFlow excels in handling large-
scale, multi-dimensional arrays
 Creating and manipulating tensors

Basic
Operations Implementing machine learning
algorithms, especially neural
of networks
TensorFlow Automatic differentiation for
optimizing machine learning
models
 Tensor: A multi-dimensional array,
similar to NumPy arrays but can
run on GPU for faster computation
Understanding Graph: A series of TensorFlow
TensorFlow operations arranged into a graph
Jargon of nodes
Session: A TensorFlow mechanism
to execute operations in the graph
 Widely used in deep learning models like
image recognition, natural language
processing, and predictive analytics
Real-Life
Preferred in complex neural network
Applications applications
of
TensorFlow Suitable for both research and production
with robust capabilities in training and
deploying machine learning models
 import tensorflow as tf

TensorFlow # Creating a constant tensor

Code tensor1 = [Link]([[1, 2], [3, 4]])
# Addition operation
Example tensor2 = tensor1 + tensor1
print(tensor2)
 [Link](
[[2 4]
TensorFlow [6 8]], shape=(2, 2), dtype=int32)
Output
Example Shows the result of adding two tensors.
 Departments/Roles: Primarily used
by Data Scientists, AI Engineers, and
Researchers
TensorFlow
in the Companies: Adopted in various
Industry sectors such as technology,
healthcare, finance, and
entertainment for advanced
machine learning projects
 Keras is a high-level neural networks API,
written in Python and capable of running
on top of TensorFlow, CNTK, or Theano

It is designed for fast experimentation

Introduction with deep neural networks
to Keras
It focuses on being user-friendly,
modular, and extensible
 Building and training neural
network models

Basic Supports convolutional and

Operations recurrent neural networks, as well
as combinations of the two
of Keras
Easy to define and train models
due to its high-level nature
 Ideal for prototyping deep
learning models quickly and
efficiently
Real-Life Used in applications ranging from
Applications image and text classification to
generative models
of Keras
Popular in both academic research
and industry applications for its
ease of use and flexibility
 from [Link] import Sequential
from [Link] import Dense

# Creating a simple neural network model

model = Sequential([
Dense(32, activation='relu', input_shape=(784,)),
Keras Code
Dense(10, activation='softmax')
Example ])
[Link](optimizer='adam',
loss='categorical_crossentropy', metrics=['accuracy'])

This example creates a basic neural

network model for classification.
Keras in the Industry

Companies: Extensively
Departments/Roles:
used in startups as well as
Favored by Data Scientists
large companies for
and Machine Learning
developing commercial
Engineers for rapid
and research applications
prototyping
in AI
 • Comparison Slide
Comparison: • Note: While Keras provides an easier entry point,
Keras vs. TensorFlow especially for beginners, TensorFlow offers more
advanced capabilities

Feature Keras TensorFlow

Level of API High-level API, focusing on usability Low-level API, offering more control and flexibility
Easier for beginners due to its simplicity and user-
Ease of Use Requires deeper understanding of machine learning concepts
friendliness
More abstract and less flexible; good for standard neural
Flexibility Highly flexible; suitable for custom and complex models
networks
Can run on top of TensorFlow (also supports other
Integration A comprehensive framework that Keras can integrate with
backends)
Rapid prototyping and experimentation with deep neural Developing and training complex and large-scale machine learning
Use Case
networks models
Community and Extensive community; backed by Google with continuous updates
Popular in the community for its simplicity and ease of use
Support and support
Data Visualization Libraries

• Matplotlib
• Plotly
• Pydot
• Comparison:
Matplotlib vs. Plotly vs. Pydot
 Matplotlib is a comprehensive library
for creating static, animated, and
interactive visualizations in Python
Introduction It offers an object-oriented API for
to embedding plots into applications
Matplotlib
Popular for its versatility in creating a
wide range of graphs and plots
Basic Operations of Matplotlib

Creating a wide variety of plots and charts, like line plots, scatter
Creating plots, bar charts, and histograms

Customizing Customizing plots with labels, axes, legends, and annotations

Integrating Integrating with Pandas for streamlined plotting of DataFrames

 Essential for data exploration and
analysis
Real-Life
Used in academic and scientific
Applications research for visual representation
of of data
Matplotlib Commonly used in business for
generating insights from data
Matplotlib Code Example and Visual Output
import [Link] as plt
import numpy as np

# Sample data
x = [Link](0, 10, 100)
y = [Link](x)

# Create a plot
[Link](figsize=(8, 4))
[Link](x, y, '-b', label='Sine Wave')
[Link]('Sample Matplotlib Plot')
[Link]('X Axis')
[Link]('Y Axis')
[Link](loc='upper right')
[Link](True)
[Link]()
 Departments/Roles: Used by Data
Analysts, Scientists, and Engineers
for data visualization
Matplotlib
in the Companies: Employed across
Industry various sectors like finance,
healthcare, marketing, and
technology for data-driven decision
making
 Plotly is an interactive graphing
library for Python

Specializes in creating high-quality,

Introduction interactive, and browser-based
to Plotly graphs
Ideal for making complex plots
intuitive and accessible
 Creating a variety of interactive
charts, such as line charts, scatter
plots, and 3D charts
Basic Customizable and interactive
Operations elements, such as hover effects,
zooming, and panning
of Plotly
Integration with Pandas for
efficient data manipulation and
visualization
 Widely used in business intelligence for
creating interactive dashboards

Real-Life Useful in academic research for dynamic

data visualization
Applications
of Plotly
Employed in finance, weather forecasting,
and engineering for its dynamic and
responsive charting capabilities
Plotly Code Example and Visual Output
import plotly.graph_objs as go
import numpy as np

# Sample data
x = [Link](0, 10, 100)
y = [Link](x)

# Create a plotly figure

fig = [Link](data=[Link](x=x, y=y, mode='lines',
name='Sine Wave'))
fig.update_layout(title='Sample Plotly Plot',
xaxis_title='X Axis',
yaxis_title='Y Axis')

# Show the plot

[Link]()
 Departments/Roles: Popular among
Data Analysts, Business Analysts,
and UX/UI Designers
Plotly in
the Companies: Used in diverse
Industry industries like e-commerce,
environmental services, and
healthcare for interactive data
presentations
 Pydot is a Python interface to
Graphviz, which allows the creation of
both directed and non-directed graphs

It's used to visualize complex

Introduction structures like networks, workflows,
to Pydot and hierarchies

Suitable for graph representation of

data structures, state diagrams, and
other networked information
 •Creating and manipulating
graph objects
Basic •Adding nodes and edges to
represent connections and
Operations relationships
of Pydot •Exporting graphs to various
image formats for
visualization
Real-Life Applications of Pydot

Often used in software engineering for visualizing code

dependencies and architectures

Applied in data science to represent decision trees and neural

network structures

Useful in academic research for diagramming complex systems and

processes
Pydot Code Example and Visual Output
import pydot
from [Link] import Image, display

# Sample graph with pydot

graph = [Link](graph_type='graph')
edge = [Link]("Node 1", "Node 2")
graph.add_edge(edge)

# Create another node and add an edge

node3 = [Link]("Node 3", style="filled", fillcolor="yellow")
graph.add_node(node3)
edge = [Link]("Node 2", "Node 3")
graph.add_edge(edge)

# Generate and display the graph

png_str = graph.create_png(prog='dot')
display(Image(data=png_str))
 Departments/Roles: Utilized by
Software Engineers, Data
Scientists, and System
Pydot in Architects
the
Companies: Adopted in sectors
Industry like technology, academia, and
engineering for visualizing
complex networked systems
Comparison • Note: While Matplotlib is excellent for static and
Matplotlib vs. Plotly simple interactive plots, Plotly excels in creating
vs. Pydot more dynamic and interactive visualizations

Feature/Aspect Matplotlib Plotly Pydot

Interface to Graph viz for network and
Type Static and interactive plotting library Interactive graphing library
graph diagrams

Advanced 2D and 3D interactive

Visualization 2D and basic 3D visualizations Graphs and network diagrams
visualizations

Limited (enhanced with additional High (interactive plots with hover, zoom, Static (focuses on structure rather than
Interactivity
libraries) and update features) interactivity)

Moderate to high (depending on usage Moderate (requires understanding of

Learning Curve Moderate
complexity) graph theory)

General-purpose plotting (scientific, Business intelligence, dashboards, Visualizing structured data, workflows,
Use Cases
statistical) complex data exploration hierarchies

PNG, PDF, SVG, and others through

Output Formats PNG, PDF, SVG, EPS, and others HTML, PNG, JPEG, Webgl, and others
Graphviz

Integrates with Pandas and Dash for Can be used alongside other
Integration Works well with Pandas and NumPy
web applications visualization libraries

Customization High (detailed control over plots) High (with interactive elements) Moderate (focused on graph structures)
Additional Libraries

• Seaborn
• Scikit-learn
• Statsmodels
• NLTK:
 Seaborn is a Python visualization library
based on Matplotlib that provides a high-
level interface for drawing attractive
statistical graphics
It's particularly suited for visualizing
Introduction complex datasets
to Seaborn
Offers built-in themes for stylish and
informative statistical graphics
 Scikit-learn is a library for machine
learning that provides simple and efficient
tools for data mining and data analysis

Introduction It's built on NumPy, SciPy, and Matplotlib

to Scikit-
learn
Widely used for various machine learning
tasks, including classification, regression,
clustering, and dimensionality reduction
 Statsmodels is a Python module that provides
classes and functions for the estimation of
many different statistical models, as well as for
conducting statistical tests and exploring data

Introduction Ideal for statistical modeling and hypothesis

testing
to
Statsmodels
Extensively used in academia and research
 The Natural Language Toolkit is a leading
platform for building Python programs to
work with human language data

It provides easy-to-use interfaces to over

Introduction 50 corpora and lexical resources
to NLTK
Commonly used for text processing and
analysis, especially in linguistics and NLP
 The Python ecosystem offers a rich set
of libraries for various data science
needs

Python in Encouraging continuous learning and

exploration to harness these tools
Data effectively
Science
Emphasis on the importance of practical
application and staying updated with
new developments in the field
Insights

 If we have the right mindset,

learning python is a piece of cake
 Python can help me get any IT
enabled job
 Data Science Jobs are prestigious
and gives me more MONEY
 Our generation should lead a
lavish life

75
Thank You • THANK YOU for you participation

& • Invitation for any questions or

Questions discussions