PYTHON PROGRAMMING

MASTERY GUIDE
From Zero to Professional Developer

PART 9
File Handling

Reading & Writing • CSV • JSON • XML • pathlib

Large Files • File System Operations • Practical Projects
PART 9: FILE HANDLING
Almost every real-world application works with files. Web servers read configuration files. Data
pipelines ingest CSV exports. APIs exchange JSON documents. Log analysers parse gigabytes of log
files. Backup utilities copy entire directory trees. Knowing how to work with the file system fluently is a
core Python skill that you will use daily.

In this part you will master Python's complete file-handling toolkit: opening and closing files safely,
reading and writing text and binary data, working with structured formats (CSV, JSON, XML),
processing files too large to fit in memory, and navigating the file system with the modern pathlib
module.

🎯 What You Will Learn in Part 9

9.1 Opening and Closing Files — open(), modes, encoding, and the with statement
9.2 Reading Files — read(), readline(), readlines(), and iteration
9.3 Writing Files — write(), writelines(), append mode
9.4 File Modes Reference — r, w, a, x, b, t, + combinations explained
9.5 CSV Files — reading and writing with the csv module
9.6 JSON Files — reading and writing with the json module
9.7 XML Basics — parsing XML with ElementTree
9.8 The pathlib Module — modern, object-oriented file system operations
9.9 Working with Large Files — streaming, chunking, and memory efficiency
9.10 OS and shutil — file system operations (copy, move, delete, walk)
9.11 Real-World Programs and Projects
9.12 Exercises, Knowledge Check, Common Mistakes, Professional Tips

9.1 Opening and Closing Files

Every file operation in Python starts with opening the file using the built-in open() function. This creates
a file object that lets you read from or write to the file. When you are finished, the file must be closed to
release the system resource.

The open() Function

# Opening files
# Syntax:
# open(file, mode='r', encoding=None, errors=None, newline=None)

# Basic usage
file = open('[Link]', 'r') # open for reading
content = [Link]() # read the entire file
[Link]() # MUST close when done
 # The problem: if an exception occurs before close(), file stays open!
file = open('[Link]', 'r')
try:
content = [Link]()
process(content) # if this raises, close() is never called
finally:
[Link]() # this guarantees close()

# The Pythonic solution: always use 'with'

with open('[Link]', 'r') as file:
content = [Link]() # [Link]() called automatically
# File is guaranteed closed here — even if an exception occurred

Character Encoding — Always Specify It

Text files store characters as bytes. The encoding tells Python how to convert those bytes into Python
strings. Always specify encoding='utf-8' explicitly — relying on the system default causes bugs when
your code runs on different operating systems.
# Character encoding
# Always specify encoding for text files
with open('[Link]', 'r', encoding='utf-8') as f:
content = [Link]()

# Writing with encoding

with open('[Link]', 'w', encoding='utf-8') as f:
[Link]('Hello, World! Olá Mundo! ‫)'مرحبا‬

# Common encodings:
# 'utf-8' — universal, handles all languages (use this!)
# 'ascii' — English only, 7-bit
# 'latin-1' — Western European languages
# 'utf-16' — Windows default for some files

# Handling encoding errors

with open('[Link]', 'r', encoding='utf-8', errors='replace') as f:
# errors='replace' — replace bad bytes with ?
# errors='ignore' — silently skip bad bytes
# errors='strict' — raise UnicodeDecodeError (default)
content = [Link]()

9.2 Reading Files

Python provides multiple ways to read file contents, each suited for different situations. Choosing the
right method matters enormously for performance, especially with large files.
read() — Read Entire File into a String
# read()
with open('[Link]', 'r', encoding='utf-8') as f:
content = [Link]() # entire file as one string
print(type(content)) # <class 'str'>
print(len(content)) # total number of characters

# Read only N characters

with open('[Link]', 'r', encoding='utf-8') as f:
first_100 = [Link](100) # read first 100 characters
next_100 = [Link](100) # read the next 100 characters
# File position advances automatically

# Tell current position and seek to a position

with open('[Link]', 'r', encoding='utf-8') as f:
[Link](50) # read 50 chars
print([Link]()) # 50 — current position
[Link](0) # go back to start
print([Link]()) # 0

readline() — Read One Line at a Time

# readline()
with open('[Link]', 'r', encoding='utf-8') as f:
line1 = [Link]() # 'First line\n'
line2 = [Link]() # 'Second line\n'
line3 = [Link]() # 'Third line\n'
end = [Link]() # '' (empty string = end of file)

# Reading until end of file with readline()

with open('[Link]', 'r', encoding='utf-8') as f:
while True:
line = [Link]()
if not line: # empty string means EOF
break
print([Link]()) # .strip() removes trailing newline

readlines() — Read All Lines into a List

# readlines()
with open('[Link]', 'r', encoding='utf-8') as f:
lines = [Link]() # ['line1\n', 'line2\n', 'line3\n']
print(type(lines)) # <class 'list'>
print(len(lines)) # number of lines

# Strip newlines from all lines

lines_clean = [[Link]() for line in lines]
Iterating Over Lines — The Best Practice
The most Pythonic and memory-efficient way to read a file line by line is to iterate over the file object
directly. This reads one line at a time, never loading the whole file into memory — critical for large files.
# Iterating over file — best practice
# BEST PRACTICE for most situations
with open('[Link]', 'r', encoding='utf-8') as f:
for line in f: # iterates one line at a time
line = [Link]() # remove trailing newline
if line: # skip empty lines
process(line)

# Example: count lines and find specific content

error_count = 0
total_lines = 0

with open('[Link]', 'r', encoding='utf-8') as f:

for line in f:
total_lines += 1
if 'ERROR' in line:
error_count += 1

print(f'Total: {total_lines} lines, Errors: {error_count}')

Reading Methods Comparison

Method Returns Loads Whole Best For
File?
[Link]() str (entire file) Yes Small files you need as one
string
[Link](n) str (n characters) No Reading chunks of a specific
size
[Link]() str (one line) No Reading line by line with manual
control
[Link]() list of str Yes When you need all lines as a list
for line in f: str (one line per iter) No Best practice — memory-
efficient line iteration

9.3 Writing Files

Python provides two main methods for writing to files: write() for writing a single string, and writelines()
for writing a sequence of strings. The file mode you open with determines whether existing content is
overwritten or preserved.

# Writing files
# write() — write a single string
 with open('[Link]', 'w', encoding='utf-8') as f:
[Link]('Hello, World!\n') # returns number of chars written
[Link]('This is line 2.\n')
[Link]('This is line 3.\n')

# writelines() — write a list of strings (no auto newlines!)

lines = ['First line\n', 'Second line\n', 'Third line\n']
with open('[Link]', 'w', encoding='utf-8') as f:
[Link](lines)

# Appending to an existing file (mode='a')

with open('[Link]', 'a', encoding='utf-8') as f:
[Link]('New log entry at 14:25\n') # adds to end, does not
overwrite

# Write numbers and other types — must convert to string first

data = [1, 2, 3, 4, 5]
with open('[Link]', 'w', encoding='utf-8') as f:
for number in data:
[Link](str(number) + '\n') # convert int to str

# Using print() to write to a file

with open('[Link]', 'w', encoding='utf-8') as f:
print('Report Header', file=f) # print() has a 'file' argument
print('=' * 40, file=f)
print(f'Total: {sum(data)}', file=f)

9.4 File Modes — Complete Reference

Mode Name Description File Exists? File Missing?
'r' Read Open for reading (default) Reads from start FileNotFoundError
'w' Write Open for writing OVERWRITES Creates new file
content
'a' Append Open for writing at end Preserves Creates new file
content, adds to
end
'x' Exclusive Create and write (fails if FileExistsError Creates new file
exists)
'r+' Read+Write Open for reading and writing Reads/writes FileNotFoundError
from start
'w+' Write+Read Open for writing and reading OVERWRITES Creates new file
content
'a+' Append+Read Open for reading and Preserves Creates new file
appending content
'rb' Read Binary Read in binary mode Reads bytes FileNotFoundError
 'wb' Write Binary Write in binary mode Overwrites as Creates new file
bytes
'ab' Append Binary Append in binary mode Adds bytes to Creates new file
end

# File mode examples

# Text mode (default) — strings
with open('[Link]', 'r') as f: # 't' is implicit: 'rt'
data = [Link]() # returns str

# Binary mode — bytes (for images, PDFs, executables, etc.)

with open('[Link]', 'rb') as f:
data = [Link]() # returns bytes
print(type(data)) # <class 'bytes'>
print(data[:4]) # b'\x89PNG' — PNG magic bytes

# Copy a binary file

with open('[Link]', 'rb') as src:
with open('photo_copy.jpg', 'wb') as dst:
[Link]([Link]())

# Exclusive creation — raise error if file already exists

try:
with open('[Link]', 'x') as f: # only creates, never overwrites
[Link]('{}')
except FileExistsError:
print('Config already exists — not overwriting.')

9.5 CSV Files — The csv Module

CSV (Comma-Separated Values) is the most common format for exchanging tabular data between
applications — spreadsheets, databases, and analytics tools all export and import CSV. Python's csv
module handles the complexity of quoting, escaping, and different delimiters for you.

Reading CSV Files

# Reading CSV
import csv

# ── [Link] — reads rows as lists ─────────────────────

with open('[Link]', 'r', encoding='utf-8', newline='') as f:
reader = [Link](f) # each row is a list of strings
header = next(reader) # read the header row
print('Columns:', header) # ['Name', 'Age', 'Score', 'Grade']
for row in reader:
name, age, score, grade = row
print(f'{name}: {score} ({grade})')
 # ── [Link] — reads rows as dicts (better!) ────────
with open('[Link]', 'r', encoding='utf-8', newline='') as f:
reader = [Link](f) # each row is an OrderedDict
for row in reader:
# access by column name — no index guessing
print(f"{row['Name']}: {row['Score']} ({row['Grade']})")

# ── Read all rows into a list of dicts ────────────────────

with open('[Link]', 'r', encoding='utf-8', newline='') as f:
data = list([Link](f))

# Now data is a list of dicts

avg_score = sum(float(row['Score']) for row in data) / len(data)
print(f'Class average: {avg_score:.1f}')

Writing CSV Files

# Writing CSV
import csv

# ── [Link] — writes rows from lists ──────────────────

students = [
['Alice', 22, 88.5, 'B'],
['Bob', 19, 72.0, 'C'],
['Charlie', 21, 95.5, 'A'],
['Diana', 20, 60.0, 'D'],
]

with open('[Link]', 'w', encoding='utf-8', newline='') as f:

writer = [Link](f)
[Link](['Name', 'Age', 'Score', 'Grade']) # header
[Link](students) # all data rows

# ── [Link] — writes rows from dicts ───────────────

students_dicts = [
{'Name': 'Alice', 'Age': 22, 'Score': 88.5, 'Grade': 'B'},
{'Name': 'Bob', 'Age': 19, 'Score': 72.0, 'Grade': 'C'},
{'Name': 'Charlie', 'Age': 21, 'Score': 95.5, 'Grade': 'A'},
]

with open('students_out.csv', 'w', encoding='utf-8', newline='') as f:

fieldnames = ['Name', 'Age', 'Score', 'Grade']
writer = [Link](f, fieldnames=fieldnames)
[Link]() # writes the header row
[Link](students_dicts) # writes all data rows

# ── Custom delimiters ─────────────────────────────────────

# Some files use ; or \t instead of comma
with open('[Link]', 'r', encoding='utf-8', newline='') as f:
 reader = [Link](f, delimiter='\t') # tab-separated

Practical CSV: Data Processing Pipeline

# CSV data pipeline
import csv

def process_sales_csv(input_file, output_file):

'''Read sales data, calculate totals, write enriched CSV.'''
processed = []

with open(input_file, 'r', encoding='utf-8', newline='') as f:

for row in [Link](f):
qty = int(row['Quantity'])
price = float(row['UnitPrice'])
subtotal = qty * price
discount = subtotal * 0.1 if subtotal > 100 else 0
[Link]({
'Product': row['Product'],
'Qty': qty,
'Price': price,
'Subtotal': round(subtotal, 2),
'Discount': round(discount, 2),
'Total': round(subtotal - discount, 2)
})

# Write enriched data

fields = ['Product','Qty','Price','Subtotal','Discount','Total']
with open(output_file, 'w', encoding='utf-8', newline='') as f:
w = [Link](f, fieldnames=fields)
[Link]()
[Link](processed)

grand_total = sum(row['Total'] for row in processed)

print(f'Processed {len(processed)} rows. Grand total: $
{grand_total:.2f}')
return processed

9.6 JSON Files — The json Module

JSON (JavaScript Object Notation) is the universal data exchange format for web APIs, configuration
files, and data storage. Python's json module makes reading and writing JSON seamless — Python
dicts and lists map directly to JSON objects and arrays.

JSON ↔ Python Type Mapping

JSON Type Python Type Example
object {} dict { "name": "Alice" } ↔ {'name': 'Alice'}
array [] list [1, 2, 3] ↔ [1, 2, 3]
string str "hello" ↔ 'hello'
number (int) int 42 ↔ 42
number (float) float 3.14 ↔ 3.14
true / false bool true ↔ True, false ↔ False
null None null ↔ None

Reading JSON
# Reading JSON
import json

# ── [Link]() — read JSON from a file ───────────────────

with open('[Link]', 'r', encoding='utf-8') as f:
config = [Link](f) # returns a Python dict or list

print(config['database']['host']) # 'localhost'
print(config['debug']) # True

# ── [Link]() — parse JSON from a string ───────────────

json_string = '{"name": "Alice", "age": 25, "scores": [88, 92, 79]}'
data = [Link](json_string) # 'loads' = load string
print(data['name']) # Alice
print(data['scores']) # [88, 92, 79]

# ── Real-world: reading API response ──────────────────────

import [Link]

url = '[Link]
with [Link](url) as response:
data = [Link]([Link]().decode('utf-8'))
print(data['name']) # Python
print(data['public_repos']) # number of repos

Writing JSON
# Writing JSON
import json

config = {
'database': {
'host': 'localhost',
'port': 5432,
'name': 'myapp_db'
},
 'debug': False,
'allowed_hosts': ['localhost', '[Link]'],
'version': 1.0
}

# ── [Link]() — write Python object to file ─────────────

with open('[Link]', 'w', encoding='utf-8') as f:
[Link](config, f, indent=4) # indent=4 for pretty formatting

# Output file ([Link]):

# {
# "database": {
# "host": "localhost",
# "port": 5432,
# "name": "myapp_db"
# },
# "debug": false,
# ...
# }

# ── [Link]() — convert to JSON string ─────────────────

json_str = [Link](config, indent=2, sort_keys=True)
print(json_str)

# ── [Link]() options ───────────────────────────────────

import datetime
# indent=4 — pretty print with 4-space indentation
# sort_keys=True — sort dictionary keys alphabetically
# ensure_ascii=False — allow non-ASCII (for Unicode text like
Arabic/Chinese)
data = {'name': '‫'عربي‬, 'city': '‫}'القاهرة‬
print([Link](data, ensure_ascii=False)) # {"name": "‫"عربي‬, ...}

Handling Non-Serialisable Types

# Non-serialisable types
import json
from datetime import datetime, date
from decimal import Decimal

# These types are NOT JSON-serialisable by default:

# datetime, date, Decimal, set, bytes, custom objects

# Solution 1: Custom encoder class

class AppJSONEncoder([Link]):
def default(self, obj):
if isinstance(obj, (datetime, date)):
return [Link]() # '2024-01-15T14:23:01'
if isinstance(obj, Decimal):
return float(obj)
 if isinstance(obj, set):
return list(obj)
return super().default(obj)

data = {
'timestamp': [Link](),
'price': Decimal('19.99'),
'tags': {'python', 'programming'}
}

json_str = [Link](data, cls=AppJSONEncoder, indent=2)

print(json_str)

# Solution 2: default function parameter

def serialiser(obj):
if isinstance(obj, datetime): return [Link]()
raise TypeError(f'Type {type(obj)} not serialisable')

[Link](data, default=serialiser)

9.7 XML Basics — ElementTree

XML (eXtensible Markup Language) is used in enterprise systems, configuration files, SOAP APIs,
Microsoft Office documents, and RSS feeds. Python's built-in [Link] module provides
a clean API for parsing and generating XML.

# XML with ElementTree

import [Link] as ET

# ── Sample XML ─────────────────────────────────────────────

xml_data = '''
<students>
<student id="001" grade="A">
<name>Alice Johnson</name>
<age>22</age>
<score>92.5</score>
</student>
<student id="002" grade="B">
<name>Bob Smith</name>
<age>21</age>
<score>78.0</score>
</student>
</students>
'''

# ── Parse XML from string ──────────────────────────────────

root = [Link](xml_data)
 print([Link]) # students

# ── Iterate over child elements ────────────────────────────

for student in [Link]('student'):
student_id = [Link]('id') # get attribute
grade = [Link]('grade') # get attribute
name = [Link]('name').text # get element text
age = int([Link]('age').text)
score = float([Link]('score').text)
print(f'[{student_id}] {name} — Grade: {grade}, Score: {score}')

# ── Parse from file ────────────────────────────────────────

tree = [Link]('[Link]') # reads file
root = [Link]()

# ── Create and write XML ───────────────────────────────────

new_root = [Link]('catalogue')
book1 = [Link](new_root, 'book', id='1')
[Link](book1, 'title').text = 'Python Mastery'
[Link](book1, 'author').text = 'Jane Doe'
[Link](book1, 'price').text = '29.99'

tree = [Link](new_root)
[Link](tree, space=' ') # pretty-print (Python 3.9+)
[Link]('[Link]', encoding='utf-8', xml_declaration=True)

9.8 The pathlib Module — Modern File Paths

The pathlib module (Python 3.4+) provides an object-oriented interface to the file system. It replaces
the older [Link] approach with a cleaner, more intuitive API. Using / to join path components is one of
its most beloved features.

# pathlib basics
from pathlib import Path

# ── Creating Path objects ──────────────────────────────────

p = Path('documents/reports/2024') # relative path
home = [Link]() # current user's home directory
cwd = [Link]() # current working directory

# ── Joining paths with / operator ─────────────────────────

report = Path('documents') / 'reports' / '2024' / '[Link]'
print(report) # documents/reports/2024/[Link]

# ── Path components ────────────────────────────────────────

p = Path('/home/alice/documents/[Link]')
print([Link]) # [Link] (filename with extension)
print([Link]) # report (filename without extension)
print([Link]) # .pdf (extension with dot)
print([Link]) # ['.pdf'] (list of all extensions)
print([Link]) # /home/alice/documents
print([Link][0]) # /home/alice/documents
print([Link][1]) # /home/alice
print([Link]) # ('/', 'home', 'alice', 'documents', '[Link]')

# ── Path existence checks ──────────────────────────────────

path = Path('[Link]')
print([Link]()) # True/False
print(path.is_file()) # True if file
print(path.is_dir()) # True if directory
print(path.is_symlink()) # True if symbolic link

# pathlib operations
from pathlib import Path

# ── Reading and writing with pathlib ──────────────────────

path = Path('[Link]')

path.write_text('Hello, pathlib!\n', encoding='utf-8') # write string

content = path.read_text(encoding='utf-8') # read string
path.write_bytes(b'binary data') # write bytes
raw = path.read_bytes() # read bytes

# ── Creating directories ───────────────────────────────────

new_dir = Path('output/reports/2024')
new_dir.mkdir(parents=True, exist_ok=True) # creates all intermediate
dirs

# ── Listing directory contents ─────────────────────────────

p = Path('.')
for item in [Link](): # all items in directory
print([Link], '(dir)' if item.is_dir() else '(file)')

# ── Glob patterns — find files by pattern ─────────────────

for py_file in Path('.').glob('*.py'): # all .py in current dir
print(py_file.name)

for py_file in Path('.').rglob('*.py'): # all .py recursively

print(py_file)

# ── File operations ────────────────────────────────────────

src = Path('[Link]')
dst = Path('backup/[Link]')
[Link](parents=True, exist_ok=True)
[Link](dst) # move/rename
[Link](dst) # move, overwriting if exists
[Link]() # delete a file
Path('empty_dir').rmdir() # delete empty directory
 # ── File metadata ──────────────────────────────────────────
p = Path('[Link]')
stat = [Link]()
print(f'Size: {stat.st_size:,} bytes')
print(f'Modified: {stat.st_mtime}')
import datetime
mtime = [Link](stat.st_mtime)
print(f'Modified: {mtime:%Y-%m-%d %H:%M:%S}')

9.9 Working with Large Files

A 10GB log file, a 500MB CSV dataset, or a multi-gigabyte database dump — reading these entirely
into memory would crash most machines. Python provides several strategies for processing large files
efficiently without exhausting RAM.

Strategy 1: Line-by-Line Streaming

# Line-by-line streaming
# WRONG: loads entire 10GB file into memory
with open('[Link]', 'r') as f:
content = [Link]() # uses 10GB RAM!
lines = [Link]('\n')

# CORRECT: processes one line at a time — constant O(1) memory

error_count = 0
with open('[Link]', 'r', encoding='utf-8') as f:
for line in f: # reads one line, processes, discards
if 'CRITICAL' in line:
error_count += 1
print(f'Critical errors: {error_count}')

Strategy 2: Fixed-Size Chunk Reading

# Chunk reading
def process_in_chunks(filename, chunk_size=1024*1024): # 1MB chunks
'''Read and process a file in fixed-size chunks.'''
total_bytes = 0
with open(filename, 'rb') as f: # binary mode for raw bytes
while True:
chunk = [Link](chunk_size)
if not chunk:
break # end of file
total_bytes += len(chunk)
process_chunk(chunk) # do something with this chunk
return total_bytes
 # Calculate checksum of large file without loading it all
import hashlib

def sha256_of_file(filename):
'''Calculate SHA-256 hash of a file using 64KB chunks.'''
hasher = hashlib.sha256()
with open(filename, 'rb') as f:
for chunk in iter(lambda: [Link](65536), b''): # read 64KB at a
time
[Link](chunk)
return [Link]()

print(sha256_of_file('large_file.iso'))

Strategy 3: Large CSV with Chunking

# Large CSV processing
import csv

def analyse_large_csv(filename, chunk_size=10000):

'''Process a large CSV file in chunks of rows.'''
total_rows = 0
total_revenue = 0.0
max_order = 0.0

with open(filename, 'r', encoding='utf-8', newline='') as f:

reader = [Link](f)
chunk = []

for row in reader:

[Link](row)
if len(chunk) >= chunk_size:
# Process this chunk
for r in chunk:
amount = float(r['Amount'])
total_revenue += amount
max_order = max(max_order, amount)
total_rows += len(chunk)
chunk = [] # clear chunk from memory
print(f' Processed {total_rows:,} rows...')

# Process remaining rows

for r in chunk:
amount = float(r['Amount'])
total_revenue += amount
max_order = max(max_order, amount)
total_rows += len(chunk)

print(f'Total rows: {total_rows:,}')

print(f'Total revenue: ${total_revenue:,.2f}')
 print(f'Largest order: ${max_order:,.2f}')

9.10 OS and shutil — File System Operations

The os module provides functions for interacting with the operating system — listing directories,
checking paths, getting environment variables. The shutil module provides high-level file operations —
copying, moving, and deleting files and entire directory trees.

# os and shutil
import os
import shutil
from pathlib import Path

# ── os module — environment and paths ─────────────────────

print([Link]()) # current working directory
[Link]('/tmp') # change directory
print([Link]('HOME'))# home directory from environment
print([Link]()) # current process ID

# ── [Link] — legacy path utilities (prefer pathlib) ──────

print([Link]('[Link]')) # True/False
print([Link]('docs', '[Link]')) # docs/[Link]
print([Link]('/home/alice/[Link]'))# [Link]
print([Link]('/home/alice/[Link]')) # /home/alice
print([Link]('[Link]')) # ('report', '.pdf')

# ── [Link] — traverse directory tree ─────────────────────

for root, dirs, files in [Link]('.'):
level = [Link]('.', '').count([Link])
indent = ' ' * 2 * level
print(f'{indent}{[Link](root)}/')
for file in files:
print(f'{indent} {file}')

# ── shutil — high-level file operations ───────────────────

[Link]('[Link]', '[Link]') # copy file (no metadata)
shutil.copy2('[Link]', '[Link]') # copy file WITH metadata
[Link]('src_dir', 'dst_dir') # copy entire directory
tree
[Link]('old_path.txt', 'new_path.txt') # move or rename
[Link]('old_directory') # delete dir and all
contents

# Create a zip archive

shutil.make_archive('backup_2024', 'zip', 'documents')

# Extract a zip archive

shutil.unpack_archive('backup_2024.zip', 'restored_documents')
 # Get disk usage
total, used, free = shutil.disk_usage('/')
print(f'Disk: {total//2**30}GB total, {used//2**30}GB used,
{free//2**30}GB free')

9.11 Real-World Programs

Program 1: Student Records Manager (CSV + JSON)

# Student records manager
import csv, json
from pathlib import Path

DATA_FILE = Path('[Link]')

def load_students():
if DATA_FILE.exists():
return [Link](DATA_FILE.read_text(encoding='utf-8'))
return []

def save_students(students):
DATA_FILE.write_text(
[Link](students, indent=2, ensure_ascii=False),
encoding='utf-8'
)

def import_from_csv(csv_path):
students = load_students()
imported = 0
with open(csv_path, 'r', encoding='utf-8', newline='') as f:
for row in [Link](f):
[Link]({
'id': row['ID'],
'name': row['Name'].strip().title(),
'age': int(row['Age']),
'grades': [float(g) for g in row['Grades'].split(';')]
})
imported += 1
save_students(students)
print(f'Imported {imported} students from {csv_path}')
return students

def export_report(output_path):
students = load_students()
with open(output_path, 'w', encoding='utf-8', newline='') as f:
fields = ['ID', 'Name', 'Age', 'Average', 'Grade', 'Status']
 writer = [Link](f, fieldnames=fields)
[Link]()
for s in sorted(students, key=lambda x: x['name']):
avg = sum(s['grades']) / len(s['grades'])
grade = 'A' if avg>=90 else 'B' if avg>=80 else 'C' if avg>=70
else 'F'
[Link]({
'ID': s['id'], 'Name': s['name'], 'Age': s['age'],
'Average': round(avg, 1), 'Grade': grade,
'Status': 'Pass' if avg >= 50 else 'Fail'
})
print(f'Report exported to {output_path}')

Program 2: Log File Analyser

# Log file analyser
import re
from pathlib import Path
from collections import Counter
from datetime import datetime

def analyse_log(log_file):
'''Analyse a web server log file and produce a summary report.'''
path = Path(log_file)
level_counts = Counter()
error_lines = []
ip_counts = Counter()
hourly = Counter()

# Regex for common log format:

# [Link] - - [15/Jan/2024:14:23:01 +0000] "GET /[Link]" 200
1234
pattern = [Link](
r'(\d+\.\d+\.\d+\.\d+).+\[(\d+/.+?/\d+:\d+):.+\] '
r'"(\w+) (.+?) HTTP.+" (\d{3})'
)

with open(path, 'r', encoding='utf-8', errors='replace') as f:

for line in f:
m = [Link](line)
if not m: continue
ip, dt_str, method, url, status = [Link]()
status = int(status)

ip_counts[ip] += 1
try:
hour = [Link](dt_str, '%d/%b/%Y:%H').hour
hourly[hour] += 1
except ValueError: pass
 if status < 400: level_counts['2xx/3xx'] += 1
elif status < 500:
level_counts['4xx'] += 1
error_lines.append(f'{status} {url}')
else:
level_counts['5xx'] += 1
error_lines.append(f'SERVER ERROR {status} {url}')

# Write summary report

report = path.with_suffix('.[Link]')
with open(report, 'w', encoding='utf-8') as f:
print(f'LOG ANALYSIS: {[Link]}', file=f)
print('='*50, file=f)
print(f'Response codes:', file=f)
for code, count in sorted(level_counts.items()):
print(f' {code}: {count:,}', file=f)
print(f'\nTop 5 IPs:', file=f)
for ip, count in ip_counts.most_common(5):
print(f' {ip}: {count:,} requests', file=f)
print(f'\nPeak hour: {hourly.most_common(1)[0][0]}:00', file=f)

print(f'Report saved to {report}')

9.12 Exercises and Projects

Exercise Set A: Text Files

1. Write a function word_count(filename) that reads a text file and returns a dictionary of each
unique word and how many times it appears (case-insensitive).
2. Write a function find_and_replace(filename, old_text, new_text) that reads a file, replaces all
occurrences, and writes the result back to a new file with '_edited' appended to the name.
3. Write a function merge_files(file_list, output_file) that combines multiple text files into one,
adding a separator line between each file's content.
4. Write a function line_stats(filename) that returns a dict containing: total_lines, blank_lines,
longest_line (chars), shortest_line (chars), and average_line_length.

Exercise Set B: CSV Files

5. Write a program that reads a CSV of employee records (name, department, salary, years) and
produces a summary CSV grouped by department, showing count, average salary, and total
payroll.
6. Write a function filter_csv(input_file, output_file, **filters) that copies only rows from input to
output where the specified column values match (e.g., filter_csv('[Link]', '[Link]',
department='Engineering')).
Exercise Set C: JSON Files
7. Build a simple task manager that stores tasks in a JSON file. Each task has an ID, title, priority
(1-5), and status (todo/done). Implement: add_task, complete_task, list_tasks, delete_task.
8. Write a function flatten_json(nested_dict, separator='.') that takes a nested dict and returns a flat
dict with dotted-path keys. (e.g., {'a': {'b': 1}} becomes {'a.b': 1}).

Project: File Organiser

Build an automatic file organiser that watches a directory and organises files by type:
9. Scan a source directory for all files.
10. Create subdirectories: Images/, Documents/, Videos/, Audio/, Code/, Archives/, Other/.
11. Move each file into the appropriate subdirectory based on its extension.
12. Keep a JSON log of every move made with original path, new path, and timestamp.
13. Generate a summary report (both printed and saved to [Link]) showing counts per category
and total size moved.
14. Use pathlib throughout. Handle FileExistsError for name conflicts (append _1, _2, etc.).

Project: Configuration Manager

Build a configuration manager class that:
15. Loads configuration from a JSON file, creating default config if missing.
16. Supports dot-notation access: [Link]('[Link]').
17. Supports setting and saving: [Link]('[Link]', 5433).
18. Validates config against a schema dict (type checking).
19. Merges with environment variables (ENV_DATABASE_HOST overrides [Link]).

9.13 Knowledge Check

# Question
1 What is the difference between open() modes 'w' and 'a'?
2 Why should you always specify encoding='utf-8' when opening text files?
3 What is the difference between [Link](), [Link](), and iterating over f?
4 Why is iterating over a file object more memory-efficient than [Link]()?
5 What does newline='' do when opening CSV files?
6 What is the difference between [Link] and [Link]?
7 What is the difference between [Link]() and [Link]()?
8 What types does [Link]() NOT serialise by default?
9 What does Path('/home/alice/[Link]').stem return?
10 How does rglob() differ from glob() in pathlib?
11 When should you process a large file in chunks rather than read() all at once?
12 What is the difference between [Link]() and shutil.copy2()?

9.14 Common Mistakes

Mistake Problem Fix

Not using 'with' File stays open if exception Always use: with open(...) as f:
statement occurs
Missing newline='' in Blank rows appear between Always pass newline='' to open() for CSV
CSV data rows
Wrong mode 'w' when Overwrites entire file on each Use mode='a' for append
appending run
Not specifying encoding Crashes on non-ASCII on Always use encoding='utf-8'
some OSes
[Link] on a string TypeError — load() needs a Use [Link]() for strings
file object
[Link]() on large file Loads entire file into memory Iterate over file object directly
Building paths with + 'dir' + '/' + 'file' is fragile Use pathlib: Path('dir') / 'file'
Forgetting to convert CSV data is always strings Convert: int(row['age']), float(row['price'])
types

9.15 Professional Tips

🏆 Industry Best Practices for File Handling

1. ALWAYS USE 'with': Never open a file without 'with'. It guarantees the file
is closed even when exceptions occur. No exceptions.

2. ALWAYS SPECIFY encoding='utf-8': The system default encoding varies by OS.

Explicit UTF-8 makes your code portable and predictable everywhere.

3. USE pathlib, NOT [Link]: pathlib is cleaner, more readable, and cross-platform.
Use Path('dir') / '[Link]' instead of [Link]('dir', '[Link]').

4. STREAM LARGE FILES: Never [Link]() a file larger than available RAM.
Iterate line by line or read in chunks for anything over a few hundred MB.

5. USE DictReader/DictWriter FOR CSV: Column names, not indexes, make your
CSV code self-documenting and resistant to column reordering.
 6. ATOMIC WRITES FOR IMPORTANT DATA: Write to a temporary file first, then
rename it. This prevents data corruption if the program crashes mid-write.
Path('[Link]').write_text(data); Path('[Link]').rename('[Link]')

7. VALIDATE JSON ON LOAD: Wrap [Link]() in try/except [Link]

to handle malformed files gracefully.

8. USE 'x' MODE FOR NEW EXCLUSIVE FILES: Use open('file', 'x') instead of 'w'
when you must not overwrite an existing file (config files, lock files).

9.16 Part 9 Summary

📚 What You Learned in Part 9

✓ open() with the 'with' statement is the safe, Pythonic way to work with files
✓ Always specify encoding='utf-8' to ensure portable text handling
✓ read() loads whole file; readline() reads one line; iterating is most memory-efficient
✓ File modes: 'r' read, 'w' overwrite, 'a' append, 'x' exclusive create, 'b' binary
✓ [Link]/DictWriter — best practice for named column access in CSV files
✓ [Link]()/dump() for files; [Link]()/dumps() for strings
✓ Custom JSON encoders handle non-serialisable types (datetime, Decimal, set)
✓ ElementTree parses and generates XML with findall(), get(), and SubElement()
✓ [Link] provides OOP file paths with / operator, glob(), rglob(), and stat()
✓ Large files must be streamed — line-by-line iteration or fixed-size chunk reading
✓ [Link]() traverses directory trees; shutil copies, moves, and archives files
✓ Atomic writes (write to temp, rename) prevent data corruption

➡️ Coming Up in Part 10: Modules and Packages

In Part 10 we learn how to organise and share Python code professionally:

• Creating and importing your own modules

• Packages — directories of modules with __init__.py
• Python Standard Library tour — os, sys, datetime, math, random, and more
• Virtual environments — isolating project dependencies
• pip — installing, upgrading, and managing third-party packages
• [Link] and [Link] — dependency management
• Publishing your own package to PyPI

— End of Part 9 —
Python Programming Mastery Guide | Part 9: File Handling