Unit 4

Web Scraping And Numerical Analysis

What is Web Scraping?
• Web scraping, also called web data extraction, is an automated
process of collecting publicly available web data from targeted
websites.
• Instead of gathering data manually, web scraping software can be
used to acquire a vast amount of information automatically, making
the process much faster.
Why is Web Scraping Important?
• Some websites can contain a very large amount of invaluable data such as
stock prices, product details, sports stats, you name it.
• If you want to access this information, you either have to use whatever format
the website uses or copy and paste the information manually into a new
document.
• This can be pretty tedious when you want to extract a lot of information from
a website and here is where web scraping can help.
• Instead of scraping this data manually, in most cases, software tools called web
scrapers are preferred because they are less expensive compared to human
labor and they work at a faster rate.
• Web scrapers can run on your PC or in a data center.
How Does Web Scraping Work?
• Step 1: Retrieving content from a website

• Step 2: Extracting the required data

• Step 3: Store parsed data

•
Web Scraping
• Web scraping is the process of extracting structured data from
websites for analysis, automation, or integration.
• For example, you might scrape product details from an e-commerce
website or collect weather data from a forecasting site.
1. Concepts in Web Scraping
1. HTTP Requests
• Web scraping relies on the HTTP protocol to fetch webpages.
• Key request types include:
• GET: Retrieves data (e.g., loading a webpage).
• POST: Sends data to the server (e.g., form submission).
2. HTML Structure
• Websites are built using HTML. Scraping involves parsing this structure to extract the
desired elements like tags (<h1>, <p>, <table>), attributes, or classes.
3. Ethical Considerations
• Always follow a website's Terms of Service and check for a [Link] file, which
specifies the allowed scraping practices.
Fetching Web Pages
• Fetching a webpage is the first step in scraping.
• This involves sending an HTTP GET request to the URL and retrieving
the server's response.
• Python’s requests library simplifies HTTP communication.
• The server's response includes metadata (headers) and the content
(HTML).
• This content can then be parsed and processed.
Example Program: Fetching a Web Page
• import requests

• # Fetch a web page

• url = "[Link]
• response = [Link](url)

• # Print the HTML content of the page

• print([Link])
Submitting Forms
• Some web applications require user input via forms (e.g., search
queries, login forms).
• Scraping these pages involves automating form submissions.
• A form submission uses an HTTP POST request.
• Form data is sent as key-value pairs, representing input field names
and values.
Example Program: Submitting a Form
import requests
# URL of the form submission endpoint
url = "[Link]
# Data to submit
form_data = {
"username": "mca_student",
"password": "secure_password"
}
# Submit the form using POST
response = [Link](url, data=form_data)
# Print the server's response
print([Link]())
Downloading Web Pages After Form
Submission
• Some pages are only accessible after form submission (e.g.,
dashboards). Maintaining a session is critical for these scenarios.
• A session preserves cookies and authentication details across
requests.
• Use [Link]() for login and subsequent requests.
Example Program: Downloading Pages After Login
import requests
# Form submission URL
login_url = "[Link]
download_url = "[Link]
# Form data
login_data = {"username": "user", "password": "pass"}
# Create a session to maintain cookies
session = [Link]()
# Submit the form
[Link](login_url, data=login_data)
# Access another page after login
response = [Link](download_url)
# Save the downloaded content
with open("downloaded_page.html", "w") as file:
[Link]([Link])
Parsing HTML with CSS Selectors
• After fetching a webpage, you need to extract relevant data.
• CSS selectors are powerful tools to identify HTML elements based on
tags, classes, or IDs.
• CSS selectors can target:
• Tags: div, p, h1
• Classes: .class-name
• IDs: #id-name
• The BeautifulSoup library in Python parses HTML and enables CSS
selector-based extraction.
Example Program: Using CSS Selectors
from bs4 import BeautifulSoup
import requests

# Fetch the page

url = "[Link]
response = [Link](url)
soup = BeautifulSoup([Link], '[Link]')

# Extract elements using CSS selectors

titles = [Link]("h1, h2, h3")
for title in titles:
print([Link])
Numerical Analysis with NumPy
• Numerical analysis is a branch of mathematics focused on algorithms for
solving problems involving continuous variables.
• Python's NumPy library is a cornerstone for such computations.
• 1. Introduction to NumPy
• NumPy (Numerical Python) is a Python library designed for efficient numerical
computations. It supports:
• Multi-dimensional arrays and matrices.
• High-level mathematical functions like linear algebra, Fourier transforms, and
statistical operations.
• Key Features:
• Speed: NumPy is faster than Python lists due to its underlying implementation in C.
• Memory Efficiency: Stores data compactly.
2. Creating Arrays
• Arrays are central to NumPy. Unlike Python lists, NumPy arrays allow
for vectorized operations.
• A NumPy array is a grid of values of the same type, indexed by a tuple
of non-negative integers.
• Array dimensions are referred to as axes.
Example Program: Creating Arrays
import numpy as np

# 1D Array
arr = [Link]([1, 2, 3, 4, 5])
print("1D Array:", arr)

# 2D Array
arr_2d = [Link]([[1, 2, 3], [4, 5, 6]])
print("2D Array:\n", arr_2d)
Array Operations
• NumPy enables element-wise operations.
• For example:
• Adding 10 to each element: arr + 10
• Multiplying each element by 2: arr * 2
Example Program: Array Operations

import numpy as np

arr = [Link]([1, 2, 3, 4, 5])

# Element-wise addition
print("Add 10 to each element:", arr + 10)

# Element-wise multiplication
print("Multiply each element by 2:", arr * 2)
4. Statistical Analysis
• NumPy provides built-in functions for statistical analysis, including
mean, median, standard deviation, and variance.
• Mean: Average of elements.
• Standard Deviation: Measures the dispersion of data points.
Example Program: Statistics with NumPy
import numpy as np

arr = [Link]([1, 2, 3, 4, 5])

# Mean and Median

print("Mean:", [Link](arr))
print("Median:", [Link](arr))

# Standard Deviation
print("Standard Deviation:", [Link](arr))
5. Matrix Operations
• Matrices are 2D arrays used for linear algebra computations.
• Dot Product: Computes the product of two matrices.
• Transpose: Swaps rows and columns.
Example Program: Matrix Operations
import numpy as np

# Define matrices
matrix_a = [Link]([[1, 2], [3, 4]])
matrix_b = [Link]([[5, 6], [7, 8]])

# Matrix multiplication
result = [Link](matrix_a, matrix_b)
print("Matrix Multiplication:\n", result)

# Transpose
print("Transpose of matrix_a:\n", matrix_a.T)
 Unit 5
Data Visualization
• Data visualization transforms raw data into graphical representations,
enabling easier interpretation and insights.
• Python's libraries like NumPy, Matplotlib, Seaborn, and Pandas provide
powerful tools for creating meaningful visualizations.
• Data visualization is an easier way of presenting the data, however
complex it is, to analyze trends and relationships amongst variables with
the help of pictorial representation.
• The following are the advantages of Data Visualization
• Easier representation of compels data
• Highlights good and bad performing areas
• Explores relationship between data points
• Identifies data patterns even for larger data points
Best Practices to be followed during Data
Visualization
• Ensure appropriate usage of shapes, colors, and size while building
visualization

• Plots/graphs using a co-ordinate system are more pronounced

• Knowledge of suitable plot with respect to the data types brings more
clarity to the information

• Usage of labels, titles, legends and pointers passes seamless

information the wider audience
Data Visualization with NumPy Arrays
• NumPy arrays are a foundational data structure for numerical
computation in Python.
• They integrate seamlessly with visualization libraries like Matplotlib
and Seaborn, enabling efficient and flexible data visualization.
• Visualizing data stored in NumPy arrays provides a better
understanding of patterns, distributions, and relationships.
Why Use NumPy Arrays for Visualization?
1. Efficient Data Handling: NumPy arrays are faster and more memory-
efficient than Python lists.
2. Vectorized Operations: Operations on NumPy arrays are element-
wise, making data manipulation easy before visualization.
3. Integration with Visualization Libraries: NumPy arrays are natively
supported by libraries like Matplotlib and Seaborn.
1. Setting Up NumPy for Visualization
• Install the necessary libraries
• pip install numpy matplotlib
• Import the libraries in your Python script
• import numpy as np
• import [Link] as plt
2. Basic Visualization with NumPy and
Matplotlib
• 2.1 Plotting NumPy Array Data
• Example: Line Plot
import numpy as np
import [Link] as plt

# Create a NumPy array

x = [Link](0, 10, 100) # 100 points between 0 and 10
y = [Link](x)

# Plot
[Link](x, y, label="Sine Wave")
[Link]("Line Plot of NumPy Array")
[Link]("X values")
[Link]("Y values")
[Link]()
[Link]()
2.2 Scatter Plot with NumPy Data
• Scatter plots visualize relationships between two variables.
• Example: Scatter Plot
# Random data using NumPy
x = [Link](50) # 50 random values
between 0 and 1
y = [Link](50)

# Scatter plot
[Link](x, y, color='blue', alpha=0.7)
[Link]("Scatter Plot of NumPy Array")
[Link]("X values")
[Link]("Y values")
[Link]()
3. Visualizing Array Transformations
• You can perform mathematical operations on NumPy arrays and
visualize the results.
• Example: Multiple Trigonometric Functions
# Create x values
x = [Link](0, 10, 100)
# Compute y values
y1 = [Link](x)
y2 = [Link](x)
# Plot both functions
[Link](x, y1, label="Sine")
[Link](x, y2, label="Cosine", linestyle="--")
[Link]("Sine and Cosine Functions")
[Link]("X-axis")
[Link]("Y-axis")
[Link]()
[Link]()
4. Statistical Visualization with NumPy Arrays
• NumPy provides functions for statistical analysis, such as mean, std,
and percentile, which are useful for creating visualizations.
• Example: Histogram of Random DatapythonCopy code
# Generate random data
data = [Link](1000) # 1000 random
values from a normal distribution

# Plot histogram
[Link](data, bins=30, color='green',
edgecolor='black')
[Link]("Histogram of Random Data")
[Link]("Value")
[Link]("Frequency")
[Link]()
5. Advanced Visualizations with NumPy Arrays
• Bar Charts with Aggregated Data
• Bar charts are useful for comparing categorical data.

# Categories and values

categories = [Link](['A', 'B', 'C', 'D'])
values = [Link]([23, 45, 56, 78])

# Plot bar chart

[Link](categories, values, color='orange')
[Link]("Bar Chart Example")
[Link]("Categories")
[Link]("Values")
[Link]()
Practical Workflow Example: Real-World Visualization
• Task: Compare Monthly Sales Data Using Line and Bar Charts
import numpy as np
import [Link] as plt
# Bar chart overlay
# Simulated sales data bar_width = 0.3
months = [Link](['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun']) x_indices = [Link](len(months))
sales_2023 = [Link]([100, 150, 200, 250, 300, 350]) [Link](x_indices - bar_width / 2, sales_2023, width=bar_width,
sales_2024 = [Link]([120, 160, 210, 260, 310, 370]) label="2023 Sales")
[Link](x_indices + bar_width / 2, sales_2024, width=bar_width,
# Line plot label="2024 Sales")
[Link](months, sales_2023, label="2023 Sales",
marker='o') [Link]("Monthly Sales Comparison")
[Link](months, sales_2024, label="2024 Sales", [Link]("Months")
marker='s', linestyle='--') [Link]("Sales")
[Link]()
[Link](x_indices, months) # Align bar labels with months
[Link]()
Data Visualization with Matplotlib
• Matplotlib is the foundational library for data visualization in Python,
offering a wide range of customizable plots.
• Matplotlib: A versatile library for creating static, animated, and
interactive visualizations.
• pyplot: A module in Matplotlib that mimics MATLAB-like plotting
functionalities.
• Basic Workflow:
1. Create data (e.g., using NumPy).
2. Use [Link]() or other functions to create plots.
3. Customize the graph with titles, labels, legends, etc.
1.2 Plotting Graphs
• Line Plot
• A line plot is used to visualize trends over a continuous range.
• Example Program: Line Plot
import [Link] as plt
import numpy as np

# Data
x = [Link](0, 10, 100) # 100 points between 0 and 10
y = [Link](x)

# Plot
[Link](x, y, label="Sine Wave")
[Link]("Line Plot Example")
[Link]("X-axis")
[Link]("Y-axis")
[Link]()
[Link]()
1.3 Controlling Graph Appearance
• Matplotlib allows control over graph styles, line properties, and
marker attributes.
• Example: Customizing Line Style and Markers
[Link](x, y, color='red', linestyle='--', linewidth=2, marker='o')
[Link]("Customized Line Plot")
[Link]()
1.4 Adding Text to Graphs
• You can annotate plots by adding titles, labels, and custom text.
• Example: Adding Annotations
[Link](x, y)
[Link]("Adding Annotations")
[Link]("X-axis")
[Link]("Y-axis")
[Link](5, 0, "Midpoint", fontsize=12, color='blue')
[Link]()
1.5 More Graph Types
• Bar Chart
• Bar charts are used to represent categorical data.
categories = ['A', 'B', 'C']
values = [10, 15, 7]

[Link](categories, values, color='green')

[Link]("Bar Chart Example")
[Link]()
1.5 More Graph Types
• Histogram
• Histograms are used to represent data distributions.
data = [Link](1000)

[Link](data, bins=30, color='purple', edgecolor='black')

[Link]("Histogram Example")
[Link]()
1.5 More Graph Types
• Scatter Plot
• Scatter plots visualize relationships between two variables.
x = [Link](50)
y = [Link](50)

[Link](x, y, color='orange')
[Link]("Scatter Plot Example")
[Link]()
1.6 Patches
• Patches in Matplotlib allow you to add shapes like circles, rectangles,
and polygons to plots.
• Example: Adding a Rectangle
from [Link] import Rectangle

fig, ax = [Link]()
ax.add_patch(Rectangle((0.1, 0.2), 0.5, 0.3, color='cyan'))
[Link](0, 1)
[Link](0, 1)
[Link]("Rectangle Patch Example")
[Link]()
2. Advanced Data Visualization with Seaborn
• Seaborn is a higher-level library built on Matplotlib that provides a
more aesthetic interface for creating advanced visualizations.
• Seaborn is a powerful Python library built on top of Matplotlib,
designed to simplify the creation of attractive and informative
statistical graphics.
Key Features of Seaborn
• High-Level Interface: Seaborn provides a more convenient and user-
friendly interface compared to Matplotlib, enabling users to create
complex visualizations with fewer lines of code.
• Built-in Themes and Color Palettes: The library includes aesthetically
pleasing default styles and color palettes, which can be customized to
enhance visual appeal.
• Integration with Pandas: Seamless integration with Pandas
DataFrames allows for easy manipulation and visualization of
structured datasets.
 Advanced Visualization Techniques
1. Pair Plots
• Pair plots are an effective way to visualize relationships between multiple
variables in a dataset.
• They create a matrix of scatter plots for each pair of variables, allowing
quick identification of correlations.
import seaborn as sns
import pandas as pd

# Load sample dataset

iris = sns.load_dataset('iris')

# Create pair plot

[Link](iris, hue='species')
[Link]('Pair Plot of Iris Dataset')
[Link]()
2. Heatmaps
• Heatmaps are ideal for visualizing matrix-like data where values are
represented by colors.
• They are particularly useful for displaying correlations between multiple
variables.

# Compute correlation matrix

correlation_matrix = [Link]()

# Create heatmap
[Link](correlation_matrix, annot=True, cmap='coolwarm')
[Link]('Heatmap of Correlations in Iris Dataset')
[Link]()
3. Facet Grids
• Facet grids allow the creation of a grid of plots based on subsets of your
dataset.
• This technique is useful for visualizing the distribution of data across different
categories.
g = [Link](iris, col='species')
[Link]([Link], 'sepal_length')
[Link]()
• Statistical Visualization
• Seaborn simplifies the process of performing and visualizing statistical
analyses:
• Regression Plots: Use regplot() or lmplot() to visualize linear relationships between
variables along with regression lines.
[Link](x='sepal_length', y='sepal_width', data=iris)
[Link]('Regression Plot of Sepal Length vs Width')
[Link]()

• Box Plots and Violin Plots: These plots provide insights into the distribution
and frequency of data points across different categories.
Customization in Seaborn
• Customization Options
• Seaborn offers extensive customization options to enhance the aesthetics and
clarity of visualizations:
• Custom Color Palettes: Users can define custom color palettes using
sns.set_palette().
• Style Settings: Adjust overall styles using sns.set_style() for a polished look.
• You can enhance Seaborn plots by adding themes and color palettes.
• Example: Using Themes
• sns.set_theme(style="darkgrid")
• [Link](x=x, y=y, color="red")
• [Link]("Scatter Plot with Seaborn Theme")
• [Link]()
3. Time Series Analysis with Pandas
• Time series analysis involves statistical techniques to analyze time-
ordered data points, often collected at regular intervals.
• The Pandas library in Python provides robust tools for manipulating
and analyzing time series data, making it a popular choice for data
scientists and analysts.
• Time series analysis is used to examine data points collected over
time intervals. Pandas makes time series analysis easier with its
datetime capabilities.
Creating Time Series Data
• To create a time series in Pandas, you can use the pd.date_range()
function to generate a range of dates and then create a Series or
DataFrame.
import pandas as pd

# Create a date range

date_range = pd.date_range(start='2020-01-01', periods=10,
freq='D')

# Create a time series

ts = [Link](range(len(date_range)), index=date_range)
print(ts)
Resampling Time Series Data
• Resampling involves changing the frequency of the time series data.
• This can be useful for aggregating data over different time periods
(e.g., from daily to monthly).

# Resample to get the mean for every 3 days

resampled_ts = [Link]('3D').mean()
print(resampled_ts)
Time Series Manipulation

• Pandas provides several functions to manipulate time series data:

• Shifting Data: The shift() function allows you to shift the data forward or backward in
time, which is useful for calculating lagged values.
ts_shifted = [Link](1) # Shift by one period
• Rolling Windows: Use rolling windows to compute statistics over a specified window
size, such as moving averages.
• rolling_mean = [Link](window=3).mean()
• Handling Missing Data
• Pandas allows you to handle missing values in time series using methods like fillna(),
which can fill missing values with specified values or methods (e.g., forward fill).
•
Visualization of Time Series Data
• Visualizing time series data is essential for understanding trends and
patterns.
• You can use Matplotlib or Seaborn alongside Pandas for effective
visualization.
import [Link] as plt

# Plot the original time series

[Link](title='Time Series Example')
[Link]('Date')
[Link]('Values')
[Link]()
Practical Application: Combining Libraries
• Task: Create a multi-layered plot using Seaborn and Matplotlib
import seaborn as sns
import [Link] as plt
import numpy as np

# Data
x = [Link](0, 10, 100)
y = [Link](x)
noise = y + [Link](scale=0.2, size=100)

# Seaborn plot
[Link](x=x, y=noise, label="Noisy Sine", color="blue")

# Add a Matplotlib line

[Link](x, y, label="True Sine", color="red", linestyle="--")

[Link]("Multi-Layered Plot with Seaborn and Matplotlib")

[Link]()
[Link]()