Web Scraping for Beginners
Using Python in VS Code – December 2025
Introduction
Web scraping is the process of automatically extracting data from websites. It’s a powerful
skill for data science, research, journalism, and automation projects.
Important Ethical Guidelines
• Always check a website’s [Link] (e.g., [Link]
• Respect the site’s Terms of Service
• Do not overload servers (add delays between requests)
• Use scraping only for public data and educational purposes
• Practice on sites designed for scraping (like the examples below)
Tools You Need
• VS Code (free code editor)
• Python 3
• Required libraries: requests, beautifulsoup4, selenium, scrapy, pandas, webdriver-
manager
Step 1: Set Up Your Project in VS Code (MacBook)
1. Open VS Code
2. Create/Open a folder for your project (e.g., web-scraping-lab)
3. Open Terminal in VS Code → Terminal → New Terminal
4. Create a virtual environment named dslab
python3 -m venv dslab
5. Activate the environment
source dslab/bin/activate
(You’ll see (dslab) appear in your terminal prompt)
6. Install required packages
pip install requests beautifulsoup4 selenium scrapy pandas webdriver-
manager
7. Tell VS Code to use this environment
o Press Cmd + Shift + P
o Type “Python: Select Interpreter”
o Choose the one inside dslab/bin/python
Example 1: Simple Static Website Scraping
Goal: Scrape book titles and prices from [Link]
Create a file: simple_scrape.py
import requests
from bs4 import BeautifulSoup
import pandas as pd
# Send request to the website
url = "[Link]
response = [Link](url)
if response.status_code == 200:
print("Page fetched successfully!")
else:
print("Error fetching page")
exit()
# Parse HTML
soup = BeautifulSoup([Link], '[Link]')
# Extract data
books = []
for article in soup.find_all('article', class_='product_pod'):
title = article.h3.a['title']
price = [Link]('p', class_='price_color').text
[Link]({'Title': title, 'Price': price})
# Save to DataFrame and CSV
df = [Link](books)
print([Link](10))
df.to_csv('books_scraped.csv', index=False)
print("Data saved to books_scraped.csv")
Run: In terminal → python simple_scrape.py
Example 2: Scraping Multiple Pages
Create a file: multi_page_scrape.py
import requests
from bs4 import BeautifulSoup
import pandas as pd
import time
all_books = []
base_url = "[Link]
for page in range(1, 4): # Scrape first 3 pages
if page == 1:
url = "[Link]
else:
url = base_url.format(page)
response = [Link](url)
soup = BeautifulSoup([Link], '[Link]')
for article in soup.find_all('article', class_='product_pod'):
title = article.h3.a['title']
price = [Link]('p', class_='price_color').text
all_books.append({'Title': title, 'Price': price, 'Page': page})
print(f"Page {page} completed")
[Link](1) # Be respectful to the server
df = [Link](all_books)
df.to_csv('books_multiple_pages.csv', index=False)
print(f"Total books: {len(df)} → Saved!")
Example 3: Scraping JavaScript-Loaded Content (Using Selenium)
Goal: Scrape quotes from an infinite-scroll page
Create a file: selenium_scrape.py
from selenium import webdriver
from [Link] import Service
from [Link] import Options
from webdriver_manager.chrome import ChromeDriverManager
import pandas as pd
import time
# Setup browser (headless = no visible window)
options = Options()
options.add_argument("--headless") # Remove this line to see browser
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")
driver = [Link](service=Service(ChromeDriverManager().install()),
options=options)
[Link]("[Link]
[Link](3)
quotes = []
last_height = driver.execute_script("return [Link]")
while True:
# Extract current quotes
elements = driver.find_elements("css selector", ".quote")
for elem in elements:
text = elem.find_element("css selector", ".text").text
author = elem.find_element("css selector", ".author").text
[Link]({'Quote': text, 'Author': author})
# Scroll down
driver.execute_script("[Link](0,
[Link]);")
[Link](2)
new_height = driver.execute_script("return [Link]")
if new_height == last_height:
break
last_height = new_height
[Link]()
# Remove duplicates and save
df = [Link](quotes)
df.drop_duplicates(inplace=True)
df.to_csv('quotes_selenium.csv', index=False)
print(f"Scraped {len(df)} unique quotes!")
Practice Assignments
1. Modify Example 1 to scrape only books under £20
2. Scrape all 50 pages from [Link]
3. Visit [Link] and scrape quotes by a specific author
4. Try scraping a real-world site (e.g., news headlines) – with teacher approval
Deactivating Environment
When done working:
deactivate