Scribd
Upload
172 views6 pages

Web Scraping with Python Guide

This document provides an introduction to web scraping using Python. It discusses how web scraping works by sending requests to servers and extracting specific data elements from pages. The steps involved in web scraping include sending HTTP requests, parsing HTML responses, and traversing the parse trees. It also covers installing and importing the BeautifulSoup and Requests libraries for scraping and making requests. As an example, it describes scraping and analyzing COVID-19 case data from the Worldometer website.

Uploaded by

Anand Sharma
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
172 views6 pages

Web Scraping with Python Guide

This document provides an introduction to web scraping using Python. It discusses how web scraping works by sending requests to servers and extracting specific data elements from pages. The steps involved in web scraping include sending HTTP requests, parsing HTML responses, and traversing the parse trees. It also covers installing and importing the BeautifulSoup and Requests libraries for scraping and making requests. As an example, it describes scraping and analyzing COVID-19 case data from the Worldometer website.

Uploaded by

Anand Sharma
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Web Scraping using Python

Topics Covered:
● Introduction to Web Scraping
● How Does Web Scraping Work?
● Steps involved in web scraping
● Installing BeautifulSoup
● Installing Requests
● Scraping and Analyzing data from Worldometer website

Introduction to Web Scraping


What is Web Scraping? Why do we use Web Scraping?

Web scraping, web harvesting, or web data extraction is an


automated process of collecting large data(unstructured) from
websites. It is the process of gathering information from the Internet.
Even copying and pasting the lyrics of your favorite song is a form of
web scraping! However, the words “web scraping” usually refer to a
process that involves automation. Some websites don’t like it when
automatic scrapers gather their data, while others don’t mind. The
data collected can be stored in a structured format for further
analysis.

If you’re scraping a page respectfully for educational purposes, then


you’re unlikely to have any problems. Still, it’s a good idea to do some
research on your own and make sure that you’re not violating any
Terms of Service before you start a large-scale project.

1
How Does Web Scraping Work?

When we scrape the web, we write code that sends a request to the
server that’s hosting the page we specified. The server will return the
source code — HTML, mostly — for the page (or pages) we requested.
So far, we’re essentially doing the same thing a web browser does —
sending a server request with a specific URL and asking the server to
return the code for that page.
But unlike a web browser, our web scraping code won’t interpret the
page’s source code and display the page visually. Instead, we’ll write
some custom code that filters through the page’s source code looking
for specific elements we’ve specified, and extracting whatever content
we’ve instructed it to extract.
For example, if we wanted to get all of the data from inside a table that
was displayed on a web page, our code would be written to go
through these steps in sequence:

● Request the content (source code) of a specific URL from the


server
● Download the content that is returned
● Identify the elements of the page that are part of the table we
want
● Extract and (if necessary) reformat those elements into a dataset
we can analyze or use in whatever way we require.

Steps involved in web scraping

● Send an HTTP request to the URL of the webpage you want to


access. The server responds to the request by returning the
HTML content of the webpage.
● Once we have accessed the HTML content, we are left with the
task of parsing the data. Since most of the HTML data is nested,
we cannot extract data simply through string processing. One
needs a parser which can create a nested/tree structure of the
HTML data.

2
● Now, all we need to do is navigate and search the parse tree that
we created, i.e. tree traversal. For this task, we will be using
another third-party python library, Beautiful Soup. It is a Python
library for pulling data out of HTML and XML files.

Installing BeautifulSoup

BeautifulSoup is one of the most prolific Python libraries in existence,


in some part having shaped the web as we know it. BeautifulSoup is a
lightweight, easy-to-learn, and highly effective way to
programmatically isolate information on a single webpage at a time.
It's common to use BeautifulSoupin conjunction with the requests
library, where requests will fetch a page, and BeautifulSoup will extract
the resulting data.

● For installing Pandas Type pip install beautifulsoup4 in the


Command prompt/ terminal.

3
● Or type !pip install beautifulsoup4 or %pip install beautifulsoup4
in a Jupyter notebook cell.

● Then type from bs4 import BeautifulSoup to import pandas.

For more information on BeautifulSoup please refer to the official


Beautiful Soup Documentation.

Installing Requests

The first thing we’ll need to do to scrape a web page is to download


the page. We can download pages using the Python requests library.
The requests library will make a GET request to a web server, which
will download the HTML contents of a given web page for us. There are
several different types of requests we can make using requests, of
which GET is just one.

4
● For installing Pandas Type pip install requests in the Command
prompt/ terminal.

● Or type !pip install requests or %pip install requests in a Jupyter


notebook cell.

● Then type import requests to import pandas.

For more information on Requests please refer to the official Requests


Documentation.

5
Scraping and Analyzing data from Worldometer
website

We have scrapped Covid19. Confirmed cases, Deaths according to


Country and continent from the Worldometer website, this is for
purely educational purposes. The Website-link with the Reference
Notebook and scrapped data set with analysis is given below

● Worldometer website link


● Jupyter Notebook Download Link
● Scrapped Covid19 Dataset Download Link

Common questions

Powered by AI

The `requests` library is used to send an HTTP request to the web server to fetch the HTML content of a webpage. Following this, the `BeautifulSoup` library is employed to parse and navigate through the HTML or XML data, creating a nested/tree structure of the data which allows for efficient data extraction. These libraries complement each other as `requests` handles data retrieval, while `BeautifulSoup` handles data parsing and extraction .

By using `requests` to access and retrieve the HTML of a webpage, and then using `BeautifulSoup` to parse this HTML, specific elements of interest can be efficiently extracted and stored in a structured format. This organized dataset can then be further processed and analyzed for insights, making the combination of these tools effective for data extraction and preparation .

Parsing HTML data into a tree structure is significant because HTML data is often nested and complex, and simple string processing would fail to accurately capture the hierarchical organization of elements. The tree structure allows for efficient navigation and manipulation of specific elements of interest, enabling precise data extraction necessary for creating structured datasets .

To install BeautifulSoup in a command prompt or terminal, you can use the command `pip install beautifulsoup4`. In a Jupyter notebook, the setup might differ slightly using `!pip install beautifulsoup4` or `%pip install beautifulsoup4` within a cell. Importing is consistent, using `from bs4 import BeautifulSoup` in both environments .

Large-scale web scraping can significantly impact a website's server and its performance by overwhelming it with numerous, frequent requests that mimic regular browsing traffic. This can lead to slower load times or even crashes, affecting the service provider’s ability to serve other users. Consequently, websites may implement rate limiting or block IPs associated with scraping to protect their resources and maintain performance stability .

Automated web scraping offers the benefit of timely and comprehensive data collection, crucial during evolving situations like the COVID-19 pandemic. However, risks include potential legal implications if sites like Worldometer do not allow scraping, as well as data integrity issues if scraped data is not verified against reliable sources. Ethical considerations regarding user consent and the impact on site performance also exist .

Researching the Terms of Service is crucial to avoid potential legal issues, as many websites explicitly prohibit automated data extraction techniques like web scraping. Ensuring compliance with these terms helps in mitigating risks of legal actions and aligns with ethical standards expected when interacting with third-party web data .

The web scraping process begins with sending an HTTP request to the server hosting the target webpage, utilizing libraries like `requests` to download the HTML content. Once retrieved, the HTML content needs to be parsed to create a navigable structure, typically using a library like `BeautifulSoup`. The parsed data then allows for tree traversal, enabling the identification and extraction of specified elements for analysis and use .

When engaging in web scraping, it's essential to consider whether you are violating the Terms of Service of the website being scraped. Some websites may not explicitly allow automated scraping, which can lead to ethical and legal challenges. For educational purposes and when scraping pages respectfully, issues are unlikely, but it's critical to research and ensure that the site's rules are not being breached .

Web scraping is powerful because it automates the collection of large volumes of data, enabling detailed analysis and insights that would be time-consuming to gather manually. However, it poses privacy concerns as the data might include sensitive information or be used without the website owners' consent, raising ethical issues regarding data ownership and user privacy .

You might also like