Web Scraping in Python
1. How Websites Work
When you open a website:
● The browser sends a request to the website
● The website sends back HTML code
● The browser displays that HTML as a webpage
Web scraping works by:
● Sending the same request using Python
● Reading the HTML
● Finding and extracting required data from HTML tags
2. Libraries Used for Web Scraping in Python
(a) requests
Used to send HTTP requests to a website and get its HTML.
Purpose:
● Connect to a website
● Download page content
(b) BeautifulSoup (bs4)
Used to parse and search HTML content easily.
Purpose:
● Read HTML tags
● Extract text, links, headings, tables, etc.
3. Installing Required Libraries
pip install requests
pip install beautifulsoup4
4. Basic Web Scraping Steps
1. Import libraries
2. Send request to website
3. Get HTML response
4. Parse HTML using BeautifulSoup
5. Extract required data
5. Example Website for Practice
[Link]
This site is specially made for learning web scraping.
6. Simple Working Web Scraping Code
Example: Scraping Quotes from a Website
import requests
from bs4 import BeautifulSoup
# Step 1: Website URL
url = "[Link]
# Step 2: Send request to website
response = [Link](url)
# Step 3: Get HTML content
html_content = [Link]
# Step 4: Parse HTML using BeautifulSoup
soup = BeautifulSoup(html_content, "[Link]")
# Step 5: Find all quotes on the page
quotes = soup.find_all("span", class_="text")
# Step 6: Print extracted quotes
for q in quotes:
print([Link])
7. Explanation of the Code
● [Link](url)
Sends a request to the website and gets HTML
● [Link]
Contains the HTML source code of the webpage
● BeautifulSoup(html, "[Link]")
Converts raw HTML into a searchable structure
● find_all("span", class_="text")
Finds all <span> tags with class text (quotes)
● [Link]
Extracts readable text from HTML tags
8. Example Output
“The world as we have created it is a process of our thinking.”
“It is our choices that show what we truly are.”
“There are only two ways to live your life.”
9. Scraping Links (Extra Example)
links = soup.find_all("a")
for link in links:
print([Link]("href"))
This extracts all links (href) from the webpage.
10. Important Rules & Ethics
● Do not scrape private or restricted websites
● Always check [Link] of a website
● Avoid sending too many requests (can overload servers)
● Web scraping is for learning, research, and public data only
Testing in Python
1. What is Testing?
Testing means checking whether your code works correctly or not.
When we write a program, we assume it works — but testing confirms it.
Example:
If you write a function to add two numbers:
● Input: 2 and 3
● Expected Output: 5
Testing checks:
● Does the function always return correct results?
● Does it break for wrong input?
2. Why Do We Need Testing?
Testing is important because:
● It finds bugs (errors) early
● It ensures correct output
● It makes code reliable
● It is used in real software companies
● It prevents breaking old code when new code is added
Without testing:
● Small errors can cause big failures
● Fixing bugs becomes difficult later
3. What is Unit Testing?
Unit Testing means:
● Testing small individual parts (units) of code
● Usually a single function
Example:
● Function add(a, b) → test only this function
● Not the whole program
4. Testing in Python
Python provides two popular ways to write tests:
1. unittest (built-in, official)
2. pytest (external, simpler, more powerful)
PART A: Testing using unittest
5. What is unittest?
● Built-in Python testing framework
● Inspired by Java’s JUnit
● Uses classes and methods
● Tests are written inside test classes
6. Basic Rules of unittest
● Import unittest
● Create a class that inherits from [Link]
● Test method names must start with test_
● Use assert methods to check results
7. Example Code
Step 1: Function to test
def add(a, b):
return a + b
Step 2: Test Code
import unittest
class TestMathOperations([Link]):
def test_addition(self):
[Link](add(2, 3), 5)
[Link](add(-1, 1), 0)
[Link](add(0, 0), 0)
if __name__ == "__main__":
[Link]()
8. Explanation of unittest Code
● TestMathOperations → test class
● test_addition() → test method
● assertEqual() → checks expected vs actual result
● If result is correct → test passes
● If result is wrong → test fails
9. Common unittest Assertions
Assertion Method Purpose
assertEqual(a, b) a == b
assertNotEqual(a, a != b
b)
assertTrue(x) x is True
assertFalse(x) x is False
assertIsNone(x) x is None
assertRaises() checks exception
10. Testing Exceptions (unittest)
def divide(a, b):
return a / b
class TestDivide([Link]):
def test_divide_by_zero(self):
with [Link](ZeroDivisionError):
divide(10, 0)
This test passes only if an error occurs.
PART B: Testing using pytest
11. What is pytest?
● External testing library
● Very simple syntax
● No need for test classes
● Uses plain functions
● Very popular in industry
Installation:
pip install pytest
12. Basic Rules of pytest
● Test files start with test_
● Test functions start with test_
● Uses simple assert keyword
13. Example Code
Function
def multiply(a, b):
return a * b
Test File (test_math.py)
def test_multiply():
assert multiply(2, 3) == 6
assert multiply(0, 5) == 0
assert multiply(-1, 3) == -3
14. Running pytest
From terminal:
pytest
pytest will:
● Automatically find tests
● Run them
● Show pass/fail results
15. Testing Exceptions in pytest
import pytest
def divide(a, b):
return a / b
def test_divide_by_zero():
with [Link](ZeroDivisionError):
divide(10, 0)
16. unittest vs pytest
Feature unittest pytest
Built-in Yes No
Syntax More code Simple
Classes needed Yes No
Assertion style Methods asser
t
Beginner friendly Medium High
17. Real-Life Example
In a banking app:
● Test deposit function
● Test withdrawal function
● Test balance calculation
In a website:
● Test login function
● Test signup function
● Test payment processing
Testing ensures no wrong behavior reaches users.