UNIT – III

Reading and Writing Files | Organizing Files

SECTION A: Reading and Writing Files

1. Files and File Paths

1.1 Definition
A file is a named location on disk used to store related information permanently. Unlike
variables (which lose data when the program ends), files provide persistent storage. A file
path is a string that specifies the exact location of a file within the file system hierarchy.

1.2 Theory Explanation

Why Files Are Used:
• Data persists beyond program execution — closing a program does not erase file
data
• Files enable data sharing between multiple programs
• Files are used for logging, configuration, input/output in real applications
• Large datasets cannot be held in RAM; files allow streaming of data

Types of Files in Python:

File Type Description & Examples
Text Files (.txt, .csv, .log) Store human-readable characters; each line ends with \n
Binary Files (.jpg, .mp3, .pdf) Store data as raw bytes; not human-readable
JSON Files (.json) Structured text for data interchange
Python Files (.py) Source code files

1.3 Absolute vs. Relative Paths

Absolute Path: A full path from the root of the file system.
# Absolute path (Windows)
C:\\Users\\Student\\Documents\\[Link]

# Absolute path (Linux/Mac)

/home/student/documents/[Link]

Relative Path: A path relative to the current working directory (CWD).

# Relative path examples
[Link] # file in current directory
reports/[Link] # file in subdirectory 'reports'
 ../[Link] # file one level up

Key Concept — Current Working Directory (CWD): When a Python script runs, it has an
associated CWD. All relative paths are resolved from this location.

2. The [Link] Module

2.1 Definition
The [Link] module is a sub-module of Python's os module that provides functions for
manipulating file paths in a platform-independent manner. It abstracts away the
differences between Windows (backslash) and Unix/Mac (forward slash) path separators.

2.2 Why [Link]?

• Writing paths as raw strings ('C:\\Users\\[Link]') is OS-specific — code
breaks on other platforms
• [Link] functions generate correct separators automatically
• Provides safe methods to check existence, join, split and analyze paths

2.3 Key Functions of [Link]

Function Purpose & Behavior
[Link](p1, p2, Joins path components using the correct OS separator. Always
...) use this instead of string concatenation.
[Link](path) Returns the absolute path of a given relative or partial path.
[Link](path) Returns True if the path (file or directory) exists on disk.
[Link](path) Returns True only if path points to an existing file.
[Link](path) Returns True only if path points to an existing directory.
[Link](path) Returns the directory component of the path.
[Link](path) Returns the filename component of the path.
[Link](path) Returns a tuple (dirname, basename).
[Link](path) Returns a tuple (root, extension). Useful for checking file type.
[Link](path) Returns the size of the file in bytes.
[Link]() Returns the current working directory (from os module).
[Link](path) Changes the current working directory.

2.4 Code Examples

import os
 # --- Joining paths (platform-safe) ---
path = [Link]('C:\\Users', 'Student', '[Link]')
print(path) # C:\Users\Student\[Link] (Windows)

# --- Checking existence ---

if [Link]('[Link]'):
print('File found!')
else:
print('File not found!')

# --- Getting file info ---

print([Link]('/home/user/[Link]')) # /home/user
print([Link]('/home/user/[Link]')) # [Link]
print([Link]('[Link]')) # ('report', '.pdf')

# --- Current directory ---

print([Link]()) # Prints current working directory
[Link]('/home/user') # Changes working directory

# --- List all files in a directory ---

for filename in [Link]('.'):
print(filename)

3. The File Reading/Writing Process

3.1 Definition
File reading/writing in Python is done through file objects. A file object is created by the
open() function and provides methods to read from or write to a file. The process follows
three steps: Open → Read/Write → Close.

3.2 The open() Function — Syntax and Modes

Syntax: open()
open(file, mode='r', encoding=None)

file : Path to the file (string)

mode : How to open the file (see table below)

encoding : Character encoding (use 'utf-8' for Unicode support)

Mode Full Name Behavior

'r' Read (default) Opens for reading. Error if file does not exist.
'w' Write Opens for writing. Creates file if absent; OVERWRITES if present.
'a' Append Opens for appending. Creates if absent; adds to end if present.
'r+' Read+Write Opens for both reading and writing. File must exist.
 'x' Exclusive Create Creates new file. Error if file already exists.
'b' Binary (modifier) Use with r/w/a for binary files: 'rb', 'wb'.
't' Text (modifier) Default; use for text files. Can be combined: 'rt'.

3.3 How File Operations Work Internally

Internal Flow of File I/O
1. OPEN: Python requests OS to open the file → OS returns a file descriptor (handle)

2. READ: Data is transferred from disk to memory buffer → Python reads from buffer

3. WRITE: Python writes to an internal buffer → OS flushes buffer to disk

4. CLOSE: Buffer is flushed, file descriptor is released back to OS

KEY: Always close a file after use. Unclosed files may:

- Cause data loss (buffer not flushed)

- Consume OS resources (file descriptor leak)

- Lock the file on Windows (preventing other access)

3.4 Reading Files — Methods

Method What It Does Returns
read() Reads the ENTIRE file as one string str
read(n) Reads n characters from current position str
readline() Reads one line (up to and including \n) str
readlines() Reads ALL lines into a list list of str

3.5 Code Examples — Reading

# Method 1: read() — entire file as one string
file = open('[Link]', 'r')
content = [Link]()
print(content)
[Link]()

# Method 2: readline() — one line at a time

file = open('[Link]', 'r')
line = [Link]() # reads first line
print(line)
[Link]()

# Method 3: readlines() — all lines as list

file = open('[Link]', 'r')
 lines = [Link]() # ['Line 1\n', 'Line 2\n', ...]
for line in lines:
print([Link]()) # strip() removes trailing \n
[Link]()

# Method 4: Loop directly (memory efficient, PREFERRED)

file = open('[Link]', 'r')
for line in file:
print([Link]())
[Link]()

3.6 Code Examples — Writing

# Writing to a file (creates or overwrites)
file = open('[Link]', 'w')
[Link]('Hello, World!\n') # write() does NOT add newline
[Link]('Python is great!\n')
[Link]()

# Appending to an existing file

file = open('[Link]', 'a')
[Link]('This is appended.\n')
[Link]()

# writelines() — writes a list of strings

lines = ['Line 1\n', 'Line 2\n', 'Line 3\n']
file = open('[Link]', 'w')
[Link](lines) # No separators added automatically
[Link]()

3.7 The with Statement (Context Manager) — PREFERRED Approach

The with statement automatically closes the file when the block exits, even if an exception
occurs. This is the recommended way to handle files in Python.

# RECOMMENDED: Using 'with' — auto-closes file

with open('[Link]', 'r', encoding='utf-8') as file:
content = [Link]()
print(content)
# file is automatically closed here — even if error occurs

# Writing with 'with'

with open('[Link]', 'w', encoding='utf-8') as file:
[Link]('Hello from Python!\n')

# Reading and writing simultaneously

with open('[Link]', 'r') as infile, open('[Link]', 'w') as outfile:
for line in infile:
[Link](line)

Important: write() vs print()

 write() : Writes a string EXACTLY as given. Does NOT add newline.

[Link]('Hello') → file contains: Hello

print() : Can write to file using 'file' parameter. Adds newline automatically.
print('Hello', file=f) → file contains: Hello\n

3.8 File Pointer and seek() / tell()

• File pointer: A marker that tracks the current read/write position within the file
• tell(): Returns the current byte position of the file pointer
• seek(offset): Moves the file pointer to the specified byte position

with open('[Link]', 'r') as f:

print([Link]()) # 0 — at start
[Link](5) # reads 5 characters
print([Link]()) # 5 — pointer moved
[Link](0) # go back to beginning
print([Link]()) # reads from start again

4. Saving Variables with the shelve Module

4.1 Definition
The shelve module provides a persistent, dictionary-like object that can store Python
data structures (lists, dicts, objects) directly to disk. Unlike text files, shelve handles
serialization automatically — converting Python objects to bytes for storage and back again
on retrieval.

4.2 Theory Explanation

Why shelve?
• Text files store strings only. To save a list or dictionary, you'd need to manually
convert
• shelve acts like a mini-database — stores any Python object by key
• Uses the pickle protocol internally for object serialization
• Data persists between program runs — survives program restarts

How shelve Works Internally:

• When you write to a shelf, Python pickles (serializes) the object to bytes
• The bytes are stored in a DBM database file (creates 3 files: .db, .dir, .bak on some
systems)
• When you read, Python unpickles the bytes back into the original Python object
4.3 Syntax and Usage
import shelve

# Opening a shelf (creates files if they don't exist)

shelf = [Link]('mydata') # creates [Link] (or similar)

# WRITING: Store any Python object

shelf['name'] = 'Alice'
shelf['scores'] = [95, 87, 92, 78]
shelf['config'] = {'debug': True, 'version': 2}

# READING: Retrieve by key

print(shelf['name']) # Alice
print(shelf['scores']) # [95, 87, 92, 78]

# Checking keys (like a dictionary)

print(list([Link]())) # ['name', 'scores', 'config']

[Link]() # Always close!

# RECOMMENDED: Using 'with'

with [Link]('mydata') as shelf:
shelf['students'] = ['Bob', 'Carol', 'Dave']
data = shelf['students']
print(data) # ['Bob', 'Carol', 'Dave']

4.4 Key Features of shelve

• Dictionary-like API: Supports keys(), values(), items(), len(), del
• Any picklable object: Strings, numbers, lists, dicts, tuples, class instances
• Persistent: Data survives between program executions
• Not thread-safe: Should not be accessed by multiple processes simultaneously
• Platform-dependent file format: Shelf files may not be portable across OS

shelve vs Text File Notes

Data types shelve: any Python object | Text: strings only
Complexity shelve: automatic | Text: manual conversion needed
Readability shelve: binary (not human-readable) | Text: readable
Use case shelve: storing Python state | Text: logs, config, user data

5. Saving Variables with [Link]()

5.1 Definition
The pprint (Pretty Printer) module provides functions to display Python data structures in a
readable, well-formatted manner. The pformat() function converts a Python object to a
formatted string that can be saved to a file. Unlike shelve, this saves data as human-
readable Python code.
5.2 Theory Explanation
Why [Link]()?
• When printing large nested dicts/lists, print() produces a single unreadable line
• pprint formats data with proper indentation, line breaks, and sorting
• pformat() returns the formatted representation as a string (unlike pprint() which
prints directly)
• Saving pformat() output to a .py file creates human-readable persistent storage
of Python objects

5.3 Code Examples

import pprint

data = {'name': 'Alice', 'scores': [95, 87, 92],

'address': {'city': 'Delhi', 'pin': '110001'}}

# pprint() — prints formatted output

[Link](data)
# Output:
# {'address': {'city': 'Delhi', 'pin': '110001'},
# 'name': 'Alice',
# 'scores': [95, 87, 92]}

# pformat() — returns formatted string (for saving)

formatted = [Link](data)
print(type(formatted)) # <class 'str'>

# SAVING to file as readable Python code

with open('data_backup.py', 'w') as f:
[Link]('data = ' + [Link](data))

# data_backup.py will contain readable Python code:

# data = {'address': {'city': 'Delhi', 'pin': '110001'},
# 'name': 'Alice',
# 'scores': [95, 87, 92]}

pformat() vs shelve Comparison

Human-readable? pformat(): YES (plain text) | shelve: NO (binary)
Editable? pformat(): YES | shelve: NO
Supports all objects? pformat(): No (only printable) | shelve: Yes (all picklable)
Best for pformat(): Configs, debugging | shelve: App state, complex data

SECTION B: Organizing Files

6. The shutil Module
6.1 Definition
The shutil (Shell Utilities) module provides high-level file and directory operations —
copying, moving, renaming, and deleting files and entire directory trees. While os handles
low-level path operations, shutil handles bulk file management tasks that mirror shell
commands.

6.2 Why shutil?

• The os module can only delete empty directories and individual files
• shutil can copy/move entire directory trees recursively with one call
• Provides cross-platform file operations that work on Windows, Linux, and Mac
• Essential for file automation scripts: backup systems, batch renaming, deployment

6.3 Key Functions

Function Description
[Link](src, dst) Copies file from src to dst. Copies content only (not metadata like
timestamps).
shutil.copy2(src, dst) Copies file AND metadata (timestamps, permissions). Like cp -p
in Unix.
[Link](src, dst) Recursively copies an ENTIRE directory tree from src to dst. dst
must NOT exist.
[Link](src, dst) Moves file or directory. Can rename if dst is in same location.
[Link](path) PERMANENTLY deletes an entire directory tree. ⚠️ No Recycle
Bin!
shutil.make_archive(base, Creates a ZIP or TAR archive of a directory.
fmt, root)
shutil.disk_usage(path) Returns disk usage statistics (total, used, free) in bytes.

6.4 Code Examples

import shutil
import os

# --- Copying files ---

[Link]('[Link]', 'backup/source_copy.txt')
shutil.copy2('[Link]', 'archive/') # copies to directory, keeps filename

# --- Copying entire directory ---

# 'backup_folder' must NOT already exist
[Link]('project_files', 'backup_folder')

# --- Moving / Renaming ---

[Link]('old_name.txt', 'new_name.txt') # rename
 [Link]('[Link]', 'archive/') # move to directory

# --- Deleting entire directory tree ---

# WARNING: This is PERMANENT! Cannot be undone.
[Link]('temp_folder') # deletes temp_folder and everything in it

# --- Disk usage ---

usage = shutil.disk_usage('C:\\')
print(f'Total: {[Link] // (1024**3)} GB')
print(f'Free: {[Link] // (1024**3)} GB')

⚠️ Warning: [Link]() is DANGEROUS

[Link]() permanently deletes directories without any warning.

There is NO Recycle Bin or Undo.

Safe practice: always check before deleting:

if [Link](folder):

[Link](folder)

Alternative: Use send2trash module (third-party) to send to Recycle Bin.

7. Walking a Directory Tree

7.1 Definition
Directory tree walking is the process of traversing all directories and subdirectories
starting from a root folder, visiting every file and folder along the way. Python's [Link]()
function generates a tuple for each directory in the tree, making it easy to process all files
recursively.

7.2 Theory Explanation

Why Walk a Directory Tree?
• Find all files with a specific extension (e.g., all .py files in a project)
• Batch rename or process thousands of files without manually opening each folder
• Build backup systems that replicate a folder structure
• Calculate total size of a directory and all its contents

How [Link]() Works:

• Starts at the top directory (the path you provide)
• For each directory visited, yields a 3-tuple: (dirpath, dirnames, filenames)
• dirpath: String — the current folder's full path
◦ dirnames: List of sub-folder names inside dirpath
 ◦ filenames: List of filenames (not paths) inside dirpath
• Continues recursively into each subdirectory until all are visited

7.3 Visual Representation

Directory Tree Walk — Text Diagram
Folder Structure:
project/

├── [Link]

├── data/
│ ├── [Link]

│ └── [Link]

└── docs/

└── [Link]

[Link]('project') yields (in order):

1st yield: ('project', ['data', 'docs'], ['[Link]'])

2nd yield: ('project/data', [], ['[Link]', '[Link]'])

3rd yield: ('project/docs', [], ['[Link]'])

7.4 Code Examples

import os

# --- Basic walk: print all files and folders ---

for dirpath, dirnames, filenames in [Link]('project'):
print(f'Current folder: {dirpath}')
print(f' Subfolders: {dirnames}')
print(f' Files: {filenames}')

# --- Finding all .txt files in a tree ---

for dirpath, dirnames, filenames in [Link]('C:\\Users\\Student'):
for filename in filenames:
if [Link]('.txt'):
full_path = [Link](dirpath, filename)
print(full_path)

# --- Calculating total size of all files ---

total_size = 0
for dirpath, dirnames, filenames in [Link]('project'):
for filename in filenames:
filepath = [Link](dirpath, filename)
total_size += [Link](filepath)
print(f'Total size: {total_size} bytes')
 # --- Backup: copy all .py files to backup folder ---
import shutil
for dirpath, dirnames, filenames in [Link]('project'):
for filename in filenames:
if [Link]('.py'):
src = [Link](dirpath, filename)
[Link](src, 'backup/')

8. Compressing Files with the zipfile Module

8.1 Definition
The zipfile module provides tools to create, read, write, append, and extract ZIP archives
in Python. ZIP is a lossless compression format that reduces file size and bundles
multiple files into one archive, making it ideal for backup and distribution.

8.2 Why Use ZIP Files?

• Reduce storage space — ZIP compresses files to a fraction of their original size
• Bundle multiple files into a single archive for easy transfer
• Widely compatible — ZIP files can be opened on all operating systems without
extra tools
• Used in automated backup systems, software distribution, and data archiving

8.3 Key Classes and Methods

Method / Class Description
[Link](path, Opens or creates a ZIP file. Modes: 'r' (read), 'w' (write/create),
mode) 'a' (append).
.write(filename) Adds a file to the archive.
.extractall(path) Extracts ALL files from archive to given path.
.extract(member, path) Extracts a SINGLE file from archive.
.namelist() Returns list of all filenames in the archive.
.getinfo(name) Returns ZipInfo object for a specific file (size, date, etc.).
.infolist() Returns list of ZipInfo objects for all files.
zipfile.is_zipfile(path) Returns True if the file is a valid ZIP archive.

8.4 Code Examples

import zipfile

# --- CREATING a ZIP archive ---

with [Link]('[Link]', 'w') as zf:
 [Link]('[Link]') # add file to archive
[Link]('[Link]')
[Link]('images/[Link]') # preserves folder structure
print('Archive created!')

# --- READING: Listing contents of a ZIP ---

with [Link]('[Link]', 'r') as zf:
print([Link]()) # ['[Link]', '[Link]', 'images/[Link]']
for info in [Link]():
print(f'{[Link]}: {info.file_size} bytes')

# --- EXTRACTING all files ---

with [Link]('[Link]', 'r') as zf:
[Link]('extracted_folder')

# --- EXTRACTING a single file ---

with [Link]('[Link]', 'r') as zf:
[Link]('[Link]', 'output/')

# --- ZIP an entire directory tree ---

import os
with [Link]('project_backup.zip', 'w') as zf:
for dirpath, dirnames, filenames in [Link]('project'):
for filename in filenames:
filepath = [Link](dirpath, filename)
[Link](filepath) # add each file
print('Full project backed up!')

UNIT III — Summary

Concept-Focused Summary
File I/O is a 3-step process: Open → Read/Write → Close. Always prefer with statement
for automatic resource management.
[Link] provides platform safety: Never hardcode path separators — use
[Link]() for cross-platform code.
shelve = persistent dictionary: Saves any Python object to disk by key. Internally uses
pickle serialization.
[Link]() = human-readable save: Converts Python data to readable formatted
strings for saving as .py files.
shutil = high-level file operations: Copying, moving, and deleting files/directories.
rmtree() is permanent — use with caution.
[Link]() = recursive directory traversal: Yields (dirpath, dirnames, filenames) for every
directory in a tree.
zipfile = compression and archiving: Create, read, and extract ZIP archives. Essential for
backup and distribution.

Quick Revision Points

Key Definitions
File Path : Location of a file in the filesystem (absolute or relative)

[Link] : Module for platform-safe path manipulation

open() : Built-in function to open a file and return a file object

shelve : Module for dictionary-like persistent storage of Python objects

pformat() : Returns pretty-printed string representation of a Python object

shutil : Module for high-level file/directory operations

[Link]() : Generator that yields (dirpath, dirnames, filenames) for each directory

zipfile : Module for creating and reading ZIP archives

Important Syntax — Quick Reference

open(file, mode, encoding='utf-8') # Open a file

[Link](p1, p2) # Platform-safe path joining

[Link](path) # Check if path exists

[Link]('filename') # Open/create a shelf

[Link](data) # Pretty-format to string

[Link](src, dst) # Copy a file

[Link](src, dst) # Copy entire directory

[Link](src, dst) # Move/rename file

[Link](path) # Delete directory tree (⚠️ permanent)

[Link](top) # Walk directory tree

[Link](path, mode) # Open/create ZIP file

.write(file) / .extractall(dest) # Add / extract from ZIP

Frequently Asked Exam Concepts

Q: What is the difference between 'w' and 'a' mode?
'w' OVERWRITES the file from the beginning; 'a' APPENDS to the end.

Q: Why should we use 'with' for file handling?

It auto-closes the file even if an exception occurs (context manager).

Q: What does [Link]() return?

A generator yielding (dirpath, dirnames, filenames) for each directory.

Q: What is the difference between [Link]() and shutil.copy2()?

copy() copies content only; copy2() copies content + metadata (timestamps).

Q: What does shelve use internally to serialize objects?

The pickle protocol.

Q: How do you read all lines of a file as a list?

Use [Link]() or list(file).

 UNIT – IV
Web Scraping | Modules | HTTP | HTML

SECTION A: Web Scraping Introduction

1. What is Web Scraping?

1.1 Definition
Web Scraping (also called Web Harvesting or Web Data Extraction) is the automated
process of extracting data from websites using a program. Instead of manually copying data
from web pages, a scraper programmatically fetches HTML, parses it, and extracts
relevant information.

1.2 Theory Explanation

Why Web Scraping?
• Websites present data in HTML pages — not directly accessible as a structured
dataset
• Scraping enables automated collection of prices, news, job listings, research data
• Used when no official API (Application Programming Interface) exists for the data
• Can collect thousands of records in seconds that would take days manually

How Web Scraping Works — Flow:

Web Scraping Flow Diagram (Text)
Step 1: SEND REQUEST
Python program → HTTP GET Request → Web Server

Step 2: RECEIVE RESPONSE

Web Server → HTTP Response (HTML content) → Python program

Step 3: PARSE HTML

Python parses HTML using BeautifulSoup or regex

Identifies and extracts target tags/data

Step 4: STORE DATA

Extracted data → CSV / JSON / Database / File

 Key Libraries:

webbrowser : Opens URLs in browser (no HTML access)

requests : Sends HTTP requests, receives HTML

bs4 : BeautifulSoup — parses and navigates HTML

1.3 Legal and Ethical Considerations

• [Link]: Websites publish rules about what may be scraped at /[Link]
• Terms of Service: Many websites prohibit scraping in their ToS — always check
• Rate limiting: Don't send hundreds of requests per second — can constitute a DoS
attack
• Personal data: Scraping personal information raises privacy and legal concerns
• Ethical use: Scrape only public data, respect server load, credit data sources

2. Project: [Link] with the webbrowser Module

2.1 Definition
The webbrowser module provides a high-level interface to open URLs in the user's
default web browser. It doesn't scrape or parse HTML — it simply launches the browser.
The classic example is [Link]: a script that takes an address and opens it in Google
Maps automatically.

2.2 Theory Explanation

Why webbrowser?
• Automates the task of opening specific URLs — no need to manually type addresses
• Quick integration with web services (Google Maps, searches, documentation)
• Lightweight — no installation required (part of Python standard library)
• Ideal for simple automation where you don't need to read the page content

2.3 Key Functions

Function Description
[Link](url) Opens URL in the default browser. Returns True if successful.
webbrowser.open_new(url) Opens URL in a NEW browser window.
webbrowser.open_new_tab(url) Opens URL in a NEW browser tab.

2.4 [Link] — Project Explanation

The [Link] project opens Google Maps for an address from:
• Command-line arguments — when you type the address after the script name
• Clipboard — when address is already copied, script reads it automatically
 # [Link] — Opens Google Maps for a given address

import webbrowser
import sys
import pyperclip # Third-party module to access clipboard

# Check if address was passed as command-line argument

if len([Link]) > 1:
# Arguments: python [Link] 123 Main St Delhi
address = ' '.join([Link][1:])
else:
# No argument: read address from clipboard
address = [Link]()

# URL encode the address and open in Google Maps

url = '[Link] + address
[Link](url)
print(f'Opening Google Maps for: {address}')

# Usage:
# python [Link] 1600 Amphitheatre Pkwy Mountain View CA
# OR: Copy an address, then run: python [Link]

2.5 [Link] Explained

[Link] is a list that contains the command-line arguments passed to the script:
• [Link][0]: Always the script name (e.g., '[Link]')
• [Link][1:]: All additional arguments passed after the script name
# Command: python [Link] 123 Main St
# [Link] = ['[Link]', '123', 'Main', 'St']
# [Link][1:] = ['123', 'Main', 'St']
# ' '.join([Link][1:]) = '123 Main St'

3. Downloading Files from the Web with the requests Module

3.1 Definition
The requests module is a third-party Python library that simplifies making HTTP
requests. It allows Python to send GET/POST requests to web servers and receive
responses, including the full HTML content of web pages or any other downloadable file
(images, PDFs, etc.).

3.2 HTTP Fundamentals

HTTP (HyperText Transfer Protocol): The communication protocol used between web
browsers (clients) and web servers.
HTTP Concept Explanation
Request Message sent by client (browser/Python) TO the server asking for a resource
 Response Message sent BY the server back to the client containing data + status
GET Most common request type — retrieve data from server (read-only)
POST Send data TO server (login forms, form submissions)
Status Code 3-digit number indicating result of request (200=OK, 404=Not Found,
500=Error)
URL Uniform Resource Locator — address of a resource on the web

Status Code Meaning When It Occurs

200 OK Success Request succeeded, data returned
301 Moved Redirect URL has permanently moved
400 Bad Request Client Error Malformed request
403 Forbidden Access Denied Not authorized to access resource
404 Not Found Missing URL does not exist on server
500 Server Error Server Crash Server-side problem

3.3 Installation
# Install requests using pip
pip install requests

# Import in your script

import requests

3.4 How [Link]() Works Internally

Step-by-step process:
1. Python calls [Link](url) with the target URL
2. The requests library builds an HTTP GET request
3. Request is sent to the server via the network (TCP/IP)
4. Server processes the request and returns an HTTP Response
5. Response is stored in a Response object — contains status, headers, content
6. Your code accesses .text (HTML), .content (bytes), or .json()

3.5 Key Attributes of the Response Object

Attribute / Method Description Data Type
response.status_code HTTP status code (200=OK, 404=Not int
Found)
[Link] Response body as a decoded string str
(HTML text)
 [Link] Response body as raw bytes (for binary bytes
files)
[Link] Response headers as a dictionary dict
[Link] The actual URL that was requested str
[Link] Character encoding of the response str
[Link]() Parses response body as JSON (for APIs) dict/list
response.raise_for_status() Raises HTTPError if status code is 4xx or None/Exception
5xx

3.6 Code Examples — Downloading Web Pages

import requests

# --- Basic GET request ---

response = [Link]('[Link]

# --- Check if request succeeded ---

print(response.status_code) # 200
print([Link]) # [Link]

# --- Get HTML content ---

html_text = [Link]
print(html_text[:500]) # Print first 500 characters of HTML

# --- raise_for_status(): crash on error (best practice) ---

try:
response = [Link]('[Link]
response.raise_for_status() # Raises exception if 404/500
print('Success!')
except [Link] as e:
print(f'HTTP Error: {e}')
except [Link]:
print('Could not connect to server.')
except [Link]:
print('Request timed out.')

# --- Request with headers (pretend to be a browser) ---

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)'}
response = [Link]('[Link] headers=headers)

4. Saving Downloaded Files to the Hard Drive

4.1 Theory Explanation
When downloading binary files (images, PDFs, videos, ZIP archives), you must:
7. Use [Link] (not .text) to get raw bytes
8. Open the output file in binary write mode ('wb')
9. Write in chunks to avoid memory overflow with large files
Why write in chunks?
• A 1 GB file loaded with .content at once would consume 1 GB of RAM
• Writing in chunks of 100 KB keeps memory usage constant regardless of file size
• The iter_content(chunk_size) method returns a generator that yields chunks

4.2 Code Examples — Downloading and Saving

import requests

# --- Download and save a text/HTML file ---

response = [Link]('[Link]
response.raise_for_status()

with open('pride_and_prejudice.txt', 'w', encoding='utf-8') as f:

[Link]([Link])
print('Text file saved!')

# --- Download and save a binary file (image) ---

image_url = '[Link]
response = [Link](image_url)
response.raise_for_status()

with open('python_logo.png', 'wb') as f:

[Link]([Link]) # .content returns bytes
print('Image saved!')

# --- Download large file in CHUNKS (memory efficient) ---

url = '[Link]
response = [Link](url, stream=True) # stream=True prevents loading all at
once
response.raise_for_status()

with open('video.mp4', 'wb') as f:

for chunk in response.iter_content(chunk_size=100000): # 100 KB per chunk
if chunk: # filter out keep-alive chunks
[Link](chunk)
print('Large file downloaded!')

Chunk Download — How iter_content() Works

stream=True : Tells requests NOT to download the entire response at once

Instead, the response body streams in as needed

iter_content(chunk_size=N) : Yields N bytes at a time as a generator

Each iteration: write chunk to disk → free memory

Memory usage: CONSTANT (only chunk_size bytes in RAM at any time)

vs. [Link]: FULL file loaded into RAM (dangerous for large files)
5. HTML — HyperText Markup Language
5.1 Definition
HTML (HyperText Markup Language) is the standard language for creating web pages.
It describes the structure and content of a web page using tags (markup). When web
scraping, understanding HTML structure is essential because we need to identify which
tags contain the data we want to extract.

5.2 HTML Structure

HTML Document Structure
<!DOCTYPE html>

<html>

<head>

<title>Page Title</title> ← not visible on page

</head>

<body>

<h1>Main Heading</h1> ← largest heading

<p>A paragraph of text.</p> ← paragraph

<a href='url'>Link text</a> ← hyperlink

<img src='[Link]'> ← image

</body>

</html>

5.3 Key HTML Concepts for Scraping

HTML Concept Description Scraping Relevance
Tags Keywords in angle brackets: <p>, Identify what type of content it is
<div>, <a>
Attributes Properties within tags: href, class, id, Used to target specific elements
src
id attribute Unique identifier — one element per Most precise selector for scraping
page
class attribute Groups elements — multiple per Used to find all elements of a type
page
Nesting Tags inside tags form a tree (DOM) Navigate tree to find target data
<a> tag Anchor/hyperlink — href contains the Extract all links from page
URL
 <table> tag HTML table with <tr> rows and <td> Common source of structured data
cells

5.4 BeautifulSoup — Parsing HTML

After downloading HTML with requests, we use BeautifulSoup (from the bs4 library) to
parse and navigate the HTML tree to find specific elements.

# Install: pip install beautifulsoup4

from bs4 import BeautifulSoup
import requests

# --- Step 1: Download the page ---

response = [Link]('[Link]
response.raise_for_status()

# --- Step 2: Parse HTML ---

# '[Link]' is Python's built-in parser (no extra install)
soup = BeautifulSoup([Link], '[Link]')

# --- Step 3: Find elements ---

# Find FIRST <h1> tag

heading = [Link]('h1')
print([Link]) # text content of the tag

# Find ALL <a> tags (links)

links = soup.find_all('a')
for link in links:
print([Link]('href')) # get href attribute

# Find by CSS class

items = soup.find_all('div', class_='product-title')

# Find by id
price = [Link](id='product-price')
print([Link])

# --- CSS Selectors (advanced) ---

# [Link]() uses CSS selector syntax
prices = [Link]('.price') # class selector
header = [Link]('#main-header') # id selector
cells = [Link]('table tr td') # nested selector

5.5 Key BeautifulSoup Methods

Method Description
find(tag, attrs) Returns FIRST matching tag. Returns None if not found.
find_all(tag, attrs) Returns a LIST of all matching tags.
[Link] Returns all text content inside the tag (strips HTML).
[Link]('attr') Returns the value of an attribute (e.g., href, src).
[Link] Returns a dictionary of all attributes of the tag.
[Link] Returns the parent tag.
[Link] Iterates over direct children.
[Link](css) Returns list of elements matching CSS selector.
soup.get_text() Returns ALL text from the entire document.

5.6 Complete Web Scraping Example

# Complete scraper: Extract all article headlines from a news page

import requests
from bs4 import BeautifulSoup
import csv

# Step 1: Download
url = '[Link] # Hacker News
response = [Link](url)
response.raise_for_status()

# Step 2: Parse
soup = BeautifulSoup([Link], '[Link]')

# Step 3: Extract data

headlines = []
articles = soup.find_all('span', class_='titleline')

for article in articles:

link_tag = [Link]('a')
if link_tag:
title = link_tag.text
url = link_tag.get('href', 'N/A')
[Link]({'title': title, 'url': url})

# Step 4: Save to CSV

with open('[Link]', 'w', newline='', encoding='utf-8') as f:
writer = [Link](f, fieldnames=['title', 'url'])
[Link]()
[Link](headlines)

print(f'Saved {len(headlines)} headlines to [Link]')

5.7 Advantages and Limitations of Web Scraping

Advantages Limitations
Automates data collection — saves hours of Websites change structure, breaking scrapers
manual work
Collects large datasets quickly Some sites block scrapers (CAPTCHA, IP bans)
Works when no API is available Legal/ethical issues — must check ToS
 Can be scheduled for regular updates JavaScript-rendered pages need Selenium (not
just requests)
Free and open-source tools (requests, bs4) Rate limiting — too many requests = ban

UNIT IV — Summary

Concept-Focused Summary
Web Scraping is a 4-step pipeline: Send Request → Receive HTML → Parse HTML →
Store Data
[Link](): Opens URLs in the default browser — no HTML access. Used for
automation of browser actions.
[Link](): Fetches HTML or binary content from a URL. Returns a Response object
with .text, .content, .status_code.
raise_for_status(): Always call this after [Link]() to detect HTTP errors gracefully.
Binary vs Text download: Use .text for HTML/text, .content (with 'wb' mode) for
images/PDFs/binaries.
iter_content(chunk_size): Download large files in memory-efficient chunks instead of
loading entire file into RAM.
HTML = structure of web pages: Tags, attributes, id, class — all used by BeautifulSoup to
locate and extract data.
BeautifulSoup: Parses HTML into a navigable tree. find() for first match, find_all() for all
matches, select() for CSS selectors.

Quick Revision Points

Key Definitions
Web Scraping : Automated extraction of data from websites

HTTP : Protocol for communication between client and web server

GET Request : HTTP request to retrieve a resource from server

Response : Server's reply containing status, headers, and body

Status 200 : OK — request successful

Status 404 : Not Found — URL doesn't exist

HTML : Markup language defining structure of web pages

BeautifulSoup : Python library for parsing and navigating HTML

Tag : HTML element like <p>, <div>, <a>

Attribute : Property of a tag (href, class, id, src)

[Link] : File specifying what parts of site can be scraped

Important Syntax — Quick Reference

import webbrowser

[Link](url) # Open URL in browser

import requests

res = [Link](url, headers=headers) # GET request

res.raise_for_status() # Raise on HTTP error

[Link] # HTML as string

[Link] # Raw bytes

res.status_code # HTTP status code

[Link]() # Parse JSON response

from bs4 import BeautifulSoup

soup = BeautifulSoup([Link], '[Link]') # Parse HTML

[Link]('tag', class_='name') # Find first match

soup.find_all('tag', id='myid') # Find all matches

[Link]('.class-name') # CSS selector

[Link] # Inner text content

[Link]('href') # Get attribute value

for chunk in res.iter_content(100000): # Chunked download

[Link](chunk)

Frequently Asked Exam Concepts

Q: What is the role of [Link]() in web scraping?

It sends an HTTP GET request to the server and returns a Response object

containing the HTML content ([Link]) of the web page.

Q: What is raise_for_status() and why use it?

It raises an HTTPError exception if the status code is 4xx or 5xx.

Used for error checking — detects failed requests gracefully.

Q: Difference between [Link] and [Link]?

.text: decoded string (for HTML, text files)

.content: raw bytes (for images, PDFs, binary files)

Q: What does BeautifulSoup do?

Parses HTML into a tree structure and provides methods to navigate

and search the tree (find, find_all, select).

Q: How do you download a large file without running out of memory?

Use stream=True in [Link]() and write with iter_content(chunk_size).

Q: What is [Link]?

A file at the root of a website that specifies which pages/directories

web scrapers and bots are allowed or not allowed to access.

Q: What is [Link] used for in [Link]?

To read command-line arguments passed when running the script.

[Link][0] = script name, [Link][1:] = remaining arguments.

— End of Notes: Unit III & IV —

Python Programming | File Handling & Web Scraping