XML & HTML: Web Scraping
Web Scraping
5
Web scraping is the automated process of extracting data from websites. Instead of manually
4
3
copying information, a program (usually written in Python or another language) collects data
2
directly from web pages and converts it into a structured format like tables, CSV files, or
1
databases.
Why Web Scraping is Used
Websites contain a large amount of useful data, but it is usually in HTML format, which is not
directly suitable for analysis. Web scraping helps to:
● Collect data quickly and efficiently
● Automates repetitive tasks
● Convert unstructured web data into structured formats
● Enable data analysis and insights
Examples:
● Price comparison (e-commerce sites)
● News aggregation
● Stock market data collection
● Job listings extraction
Components of Web Scraping
Web scraping mainly consists of two core components:
1. Crawler (Navigator)
Definition
A crawler is a program that browses and navigates through web pages automatically by
following links.
What it does
● Starts from a URL (called a seed URL)
● Visits the webpage
● Finds links (<a href="...">)
● Follows those links to other pages
● Repeats the process
How it works (Flow)
. Start with a URL
. Download the page
. Extract links
. Visit new links
. Continue recursively
Example
If you start from a homepage:
● It visits the homepage
● Then follows links to product pages
● Then category pages
● Then sub-pages
Tools for Crawling
● Scrapy (built-in crawler)
● Custom Python scripts using requests
Real-world Example
A crawler in an e-commerce site:
● Opens homepage
● Navigates to “Electronics”
● Then to “Mobiles”
● Then to each product page
2. Scraper (Data Extractor)
Definition
A scraper is a tool that extracts specific data from a webpage.
What it does
● Reads HTML content
● Finds required elements
● Extracts useful information
How it works
. Receive HTML from crawler
. Parse HTML
. Locate required tags
. Extract data
. Store it
Example
From a product page, scraper extracts:
● Product name
● Price
● Rating
● Description
End-to-End Flow
Input → URLs + required data
● The website URL(s)
● The data you want to extract
Examples
● URL: [Link]
● Required data:
○ Product name
○ Price
○ Rating
This step is important because it defines the goal of scraping.
Request → Send HTTP request
The program sends a request to the website (like a browser).
Types of Requests
● GET → Fetch data (most common)
● POST → Send data to server
Example
import requests
url = "[Link]
response = [Link](url)
Output
● Server responds with HTML content
Load → Get HTML content
● The response contains full HTML of the webpage
● This includes:
○ Tags (<html>, <body>, <div>, <table>)
○ Content (text, links, images)
Example
<h1>Product A</h1>
<p>Price: $100</p>
This is still raw data, not structured.
Parse → Convert HTML into structure
What happens
HTML is converted into a tree-like structure so Python can navigate it.
Tools
● BeautifulSoup
● lxml
● [Link]
Example
from bs4 import BeautifulSoup
soup = BeautifulSoup([Link], "[Link]")
Result
● HTML becomes a structured object
● We can now search elements easily
Extract → Select Required Data
What happens
You locate and extract only the needed information.
Methods
● find() → first match
● find_all() → all matches
Example
title = [Link]('h1').text
price = [Link]('p').text
Now we have only useful data, not the whole page.
Clean → Format Data
Why needed
Extracted data is often messy:
● Extra spaces
● Symbols ($, ,)
● Wrong data types
Cleaning tasks
● Remove unwanted characters
● Convert types (string → int/date)
● Handle missing values
Example
price = "$100"
price = int([Link]("$", ""))
Clean data = ready for analysis
Paginate → Follow Next Pages
What happens
Many websites have multiple pages:
● Page 1 → Page 2 → Page 3
Goal
Scrape all pages, not just one.
How
● Find "Next" button or page links
● Loop through pages
Example
for page in range(1, 5):
url = f"[Link]
response = [Link](url)
Ensures complete data collection
Save → Store Data
What happens
Store cleaned data in a structured format.
Common Formats
● CSV
● Excel
● JSON
● Database
Example (CSV)
import pandas as pd
df = [Link](data)
df.to_csv("[Link]", index=False)
Example (Excel)
df.to_excel("[Link]", index=False)
Python Libraries for HTML/XML Parsing
Common Libraries
Library Features
lxml Fast, best for structured HTML/XML
BeautifulSoup Handles messy HTML well
html5lib Parses like a web browser
Installation
pip install lxml
pip install beautifulsoup4
pip install html5lib
# or all together
pip install lxml beautifulsoup4 html5lib
Extracting Tables from HTML using Pandas
Function
pd.read_html()
● pandas.read_html() is a function that automatically extracts tables (<table> tags) from
an HTML page and converts them into DataFrames.
● No need to manually parse HTML using BeautifulSoup.
● Note: Even if we don’t explicitly import lxml or BeautifulSoup, pandas will use them
internally if installed.
Basic Example (Step-by-Step)
<!DOCTYPE html>
<html>
<head>
<title>Failed Banks List</title>
</head>
<body>
<h2>FDIC Failed Bank List</h2>
<table border="1">
<thead>
<tr>
<th>Bank Name</th>
<th>City</th>
<th>State</th>
<th>Closing Date</th>
</tr>
</thead>
<tbody>
<tr>
<td>First National Bank</td>
<td>New York</td>
<td>NY</td>
<td>2020-05-01</td>
</tr>
<tr>
<td>Trust Bank</td>
<td>Los Angeles</td>
<td>CA</td>
<td>2019-08-15</td>
</tr>
<tr>
<td>Security Bank</td>
<td>Chicago</td>
<td>IL</td>
<td>2018-11-30</td>
</tr>
</tbody>
</table>
</body>
</html>
Step 1: Import pandas
import pandas as pd
Step 2: Read HTML file
tables = pd.read_html('fdic_failed_bank_list.html')
Step 3: Check number of tables
print(len(tables))
Output:
Meaning: There is only one table in the HTML file.
Step 4: Access the first table
df = tables[0]
print(df)
print([Link])
tables[0] → First table converted into a DataFrame
Bank Name City State Closing Date
0 First National Bank New York NY 2020-05-01
1 Trust Bank Los Angeles CA 2019-08-15
2 Security Bank Chicago IL 2018-11-30
Bank Name object
City object
State object
Closing Date object
dtype: object
Convert Column to Datetime
close_timestamps = pd.to_datetime(df['Closing Date'])
print(close_timestamps)
print([Link])
Why this is important:
● Enables date calculations (differences, sorting)
● Allows filtering by date ranges
● Required for using .dt accessor
● Essential for time-series analysis
Extract Year
years = close_timestamps.[Link]
print(years)
Output
0 2020
1 2019
2 2018
Name: Closing Date, dtype: int64
Explanation
● .dt → Access datetime-specific properties
● .year → Extracts only the year part from each date
Add Year as a New Column
df['Year'] = pd.to_datetime(df['Closing Date']).[Link]
print(df)
Output:
Bank Name Closing Date Year
0 First National Bank 2020-05-01 2020
1 Trust Bank 2019-08-15 2019
2 Security Bank 2018-11-30 2018
Other Useful .dt Properties
df['Month'] = pd.to_datetime(df['Closing Date']).[Link]
● .dt → access datetime properties of a pandas Series.
● month → extracts month as an integer (1-12).
● Adds a new column Month to the DataFrame.
Resulting DataFrame:
Bank Name Closing Date Month
0 First National Bank 2020-05-01 5
1 Trust Bank 2019-08-15 8
2 Security Bank 2018-11-30 11
Explanation:
● May → 5
● August → 8
● November → 11
df['Day'] = pd.to_datetime(df['Closing Date']).[Link]
● .day → extracts day of the month (1–31).
● Adds a new column Day.
Bank Name Closing Date Month Day
0 First National Bank 2020-05-01 5 1
1 Trust Bank 2019-08-15 8 15
2 Security Bank 2018-11-30 11 30
Explanation:
● 1st of May → 1
● 15th of August → 15
● 30th of November → 30
df['Weekday'] = pd.to_datetime(df['Closing Date']).dt.day_name()
● .day_name() → extracts name of the weekday (Monday, Tuesday, etc.)
● Adds a new column Weekday.
Bank Name Closing Date Month Day Weekday
0 First National Bank 2020-05-01 5 1 Friday
1 Trust Bank 2019-08-15 8 15 Thursday
2 Security Bank 2018-11-30 11 30 Friday
Explanation:
● May 1, 2020 → Friday
● August 15, 2019 → Thursday
● November 30, 2018 → Friday
Count Bank Failures Per Year
<!DOCTYPE html>
<html>
<head>
<title>Bank Failures</title>
</head>
<body>
<h2>FDIC Failed Banks</h2>
<table border="1">
<thead>
<tr>
<th>Bank Name</th>
<th>Closing Date</th>
</tr>
</thead>
<tbody>
<tr>
<td>Bank A</td>
<td>2020-03-15</td>
</tr>
<tr>
<td>Bank B</td>
<td>2020-07-20</td>
</tr>
<tr>
<td>Bank C</td>
<td>2019-05-10</td>
</tr>
<tr>
<td>Bank D</td>
<td>2018-11-30</td>
</tr>
</tbody>
</table>
</body>
</html>
failures_per_year = close_timestamps.[Link].value_counts()
print(failures_per_year)
Example Output:
2020 2
2019 1
2018 1
Name: Closing Date, dtype: int64
Explanation:
● close_timestamps.[Link] → extracts the year from each date
● .value_counts() → counts how many times each year appears
So:
● 2020 → 2 banks failed
● 2019 → 1 bank
● 2018 → 1 bank
By default, value_counts() sorts results by frequency (highest first).
Sort Results by Year (Chronological Order)
failures_per_year_sorted = failures_per_year.sort_index()
print(failures_per_year_sorted)
Output:
2018 1
2019 1
2020 2
Explanation:
● .sort_index() sorts based on the index (years)
● This gives a time-order view, which is better for analysis and plotting
Important Parameters of read_html()
Parameter Description
io URL / file path / HTML string
match Extract table containing specific
text
flavor Parser (lxml, bs4)
header Row to use as column names
index_col Set a column as index
skiprows Skip rows at top
attrs Filter table using HTML attributes
parse_dates Convert columns to datetime
encoding Encoding type
<!DOCTYPE html>
<html>
<head>
<title>Bank Data</title>
</head>
<body>
<h2>Failed Banks Table</h2>
<table id="failed_banks" border="1">
<tr>
<th>Bank Name</th>
<th>City</th>
<th>State</th>
<th>Closing Date</th>
</tr>
<tr>
<td>First National Bank</td>
<td>New York</td>
<td>NY</td>
<td>2020-05-01</td>
</tr>
<tr>
<td>Trust Bank</td>
<td>Los Angeles</td>
<td>CA</td>
<td>2019-08-15</td>
</tr>
<tr>
<td>Security Bank</td>
<td>Chicago</td>
<td>IL</td>
<td>2018-11-30</td>
</tr>
</table>
</body>
</html>
import pandas as pd
tables = pd.read_html(
io='[Link]', # File path / URL / HTML string
match='Failed Banks', # Find table containing this text
# Extracts only tables containing this text
# Useful when page has multiple tables
flavor='lxml', # Parser: 'lxml' or 'bs4'
# Parser used to read HTML:
● 'lxml' → faster (recommended)
● 'bs4' → more flexible
header=0, # First row as column names, Row to use as column names
● 0 → first row becomes header
● [0,1] → multi-level header
index_col=None, # No index column set
skiprows=0, # Skip 0 rows
attrs={'id': 'failed_banks'}, # Match table by HTML attribute
parse_dates=['Closing Date'], # Convert to datetime
encoding='utf-8' # File encoding-Handles special characters correctly.
)
df = tables[0]
print(df)
Filter Table Using Text
url = "[Link]
tables = pd.read_html(url, match="Company")
df = tables[0]
print(df)
Extracts only the table containing the word "Company"
Company Contact Country
0 Alfreds Futterkiste Maria Anders Germany
1 Centro comercial Moctezuma Francisco Chang Mexico
2 Ernst Handel Roland Mendel Austria
3 Island Trading Helen Bennett UK
Use Parser and Header
tables = pd.read_html(url, flavor="lxml", header=0)
df = tables[0]
print(df)
header=0 → First row becomes column names
flavor="1xml"
● Tells pandas to use the Ixml parser for reading HTML.
● 'Ixml' is usually faster than 'bs4' .
'bs4' can handle more messy HTML.
Advanced Options
tables = pd.read_html(
url,
flavor="lxml",
header=0,
index_col=1,
skiprows=2
)
df = tables[0]
print(df)
Explanation:
● index_col=1 → Second column becomes index
● skiprows=2 → Skip first 2 rows
Scraping
We’ll use this page:
W3Schools HTML Tables Page
Step 1: Import Libraries
import requests
import bs4 as bs
Explanation:
● requests → sends HTTP request to fetch webpage
● requests is a Python library that lets you communicate with websites.
● Specifically, it can send HTTP requests like GET, POST, PUT, DELETE, etc.
● bs4 (BeautifulSoup) → parses HTML so you can extract data
● bs4 stands for BeautifulSoup 4, a Python library for parsing HTML or XML.
● as bs → gives it a shorter alias so you can write [Link]() instead of
[Link]().
Think of it like:
● requests = download the page
● BeautifulSoup = understand the page structure
Step 2: Get HTML Content
url = "[Link]
response = [Link](url)
Explanation:
● [Link](url) sends a GET request to the website
● response contains:
○ [Link] → HTML source code
○ response.status_code → request status (200 = success)
Example:
print(response.status_code) # 200 means OK
Step 3: Parse HTML
soup = [Link]([Link], "[Link]")
Explanation:
●
Converts raw HTML into a structured tree
●
"[Link]" → built-in parser
Now we can search elements like:
● <table>
● <tr> (rows)
● <td> (cells)
Step 4: Find Table
table = [Link]('table')
Explanation:
● .find('table') → gets the first table in the page
● If multiple tables exist, use:
soup.find_all('table')
Step 5: Extract Data
rows = table.find_all('td')
for row in rows:
print([Link])
Explanation:
● .find_all('td') → finds all table data cells
● Each <td> contains one value
Output (from W3Schools table)
Alfreds Futterkiste
Maria Anders
Germany
Centro comercial Moctezuma
Francisco Chang
Mexico
Ernst Handel
Roland Mendel
Austria
...
Important Insight
This method:
table.find_all('td')
extracts all data in one flat list (not structured)
So you lose:
● Row grouping
● Column structure
Better Approach (Structured Data)
rows = table.find_all('tr') # find all rows
for row in rows:
cols = row.find_all(['td', 'th']) # get all cells, including headers
cols = [[Link]() for col in cols] # clean text
print(cols)
. .find_all('tr') → gets all rows. Each <tr> contains multiple <td> or <th>.
. .find_all(['td','th']) → gets all cells in that row. Includes header cells <th>.
. [[Link]() for col in cols] → removes extra spaces/newlines for cleaner data.
. Each row is now a list, preserving the column structure.
Output:
['Company', 'Contact', 'Country']
['Alfreds Futterkiste', 'Maria Anders', 'Germany']
['Centro comercial Moctezuma', 'Francisco Chang', 'Mexico']
...
Now data is row-wise and structured
Complete program
# Step 1: Import libraries
import requests
import bs4 as bs
import pandas as pd
# Step 2: Download the webpage
url = "[Link]
response = [Link](url)
# Optional: Check if request was successful
if response.status_code == 200:
print("Page downloaded successfully")
else:
print("Failed to download page, status code:", response.status_code)
# Step 3: Parse HTML using BeautifulSoup
soup = [Link]([Link], "[Link]")
# Step 4: Find the first table
table = [Link]("table")
# Step 5: Extract all rows
rows = table.find_all("tr")
# Step 6: Extract headers and data
data = []
for row in rows:
cols = row.find_all(["td", "th"]) # get all cells (header or data)
cols = [[Link]() for col in cols] # remove extra spaces/newlines
[Link](cols)
# Step 7: Convert to DataFrame
# First row is header
df = [Link](data[1:], columns=data[0])
# Step 8: Display DataFrame
print(df)