COMP(2041|9044) 26T1 — More on Python

[Link]

[Link] COMP(2041|9044) 26T1 — More on Python 1 / 46

Names and Types
Python associates types with values.
languages like C, Perl associate types with variables
A Python variables can refer to a value of any type.
optional type annotations can indicate a variable should refer only to a particular type
The type function allows introspection.

>>> a = 42
>>> type(a)
<type 'int'>
>>> a = "String"
>>> type(a)
<type 'str'>
>>> a = [1,2,3]
>>> type(a)
<type 'list'>
>>> a = {'ps':50,'cr':65,'dn':75}
>>> type(a)
<type 'dict'>

[Link] COMP(2041|9044) 26T1 — More on Python 2 / 46

More Types

>>> type("Hello") >>> type(float())

str float # same value as 0.0
>>> type('Hello') >>> type(5j)
str complex
>>> type("""Hello""") >>> type(3 + 1j)
str complex
>>> type('''Hello''') >>> type(complex())
str complex # same value as 0j (and 0+0j)
>>> type(str())
str # same value as "" (empty string)
>>> type(1)
int
>>> type(int())
int # same value as 0
>>> type(4.4)
float

[Link] COMP(2041|9044) 26T1 — More on Python 3 / 46

Python Sequences

Python does not have arrays

widely used Python package numpy does have arrays
Python has 3 basic sequence types: lists, tuples, and ranges
lists are mutable - they can be changed
tuples similar to lists but immutable - they can not be changed
some important operations require immutable types, e.g. hashing
ranges are immutable sequence of numbers
commonly used for loops

[Link] COMP(2041|9044) 26T1 — More on Python 4 / 46

Python Sequences - Examples

>>> l = [1,2,3,4,5]
>>> t = (1,2,3,4,5)
>>> r = range(1, 6)
>>> l[2]
3
>>> t[2]
3
>>> r[2]
3
>>> l[2] = 42
>>> l
[1, 2, 42, 4, 5]
>>> t[2] = 42
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment

[Link] COMP(2041|9044) 26T1 — More on Python 5 / 46

Some Useful Python Sequence Operations

These can be applied to lists, tuples and ranges

x in s True if an item of s is equal to x

x not in s False if an item of s is equal to x
s+t the concatenation of s and t, also s += t
s*n equivalent to adding s to itself n times, also s *= n
s[i] ith item of s
s[i:j] slice of s from i to j
s[i:j:k] slice of s from i to j with step k
len(s) length of s
min(s) smallest item of s
max(s) largest item of s
[Link](x[, i[, j]]) index of the first occurrence of x in s (at or after index i and before index j)
[Link](x) total number of occurrences of x in s

[Link] COMP(2041|9044) 26T1 — More on Python 6 / 46

Some Useful Python Mutable Sequence Operations

These can be applied to lists, not tuples or ranges

s[i] = x item i of s is replaced by x

s[i:j] = t slice of s from i to j is replaced by elements of t
del s[i:j] same as s[i:j] = []
s[i:j:k] = t the elements of s[i:j:k] are replaced by those of t
del s[i:j:k] removes the elements of s[i:j:k] from the list
[Link](x) appends x to the end of the sequence
[Link]() removes all items from s
[Link]() creates a shallow copy of s
[Link](i, x) inserts x into s at the index given by i
[Link]() or [Link](i) retrieves the item at i and also removes it from s
[Link](x) remove the first item from s where s[i] is equal to x
[Link]() reverses the items of s in place
[Link]() sort the items of s in place

[Link] COMP(2041|9044) 26T1 — More on Python 7 / 46

Ranges

>>> range(10)
range(0, 10)
>>> list(range(10))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> tuple(range(10))
(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
>>> list(range(5,10))
[5, 6, 7, 8, 9]
>>> list(range(5,10,3))
[5, 8]
>>> list(range(5, -10, -3))
[5, 2, -1, -4, -7]
>>> list(range(5, 3))
[]

[Link] COMP(2041|9044) 26T1 — More on Python 8 / 46

Even More Types
>>> type([]) >>> type({})
list dict # ??
>>> type([1]) >>> type({1})
list set
>>> type([1,]) >>> type({1,})
list set
>>> type(['a', 'b', 'c',]) >>> type({1, 2, 3})
list set
>>> type(list()) >>> type({'a', 'b', 'c',})
list # same value as [] set
>>> type(()) >>> type(set())
tuple set
>>> type((1)) >>> type({'a': 1, 'b': 2, 'c': 3,})
int # bracketed value, not tuple! dict
>>> type((1,)) >>> type(dict())
tuple dict # same value as {}
>>> type(('a', 'b', 'c',))
tuple
>>> type(tuple())
[Link] COMP(2041|9044) 26T1 — More on Python 9 / 46
Example - /bin/echo using while

# Python implementation of /bin/echo

# using indexing & while, not pythonesque
import sys
i = 1
while i < len([Link]):
if i > 1:
print(" ", end="")
print([Link][i], end="")
i += 1
print()
source code for [Link]

[Link] COMP(2041|9044) 26T1 — More on Python 10 / 46

Example - /bin/echo using for/range

# Python implementation of /bin/echo

# using indexing & range, not pythonesque
import sys
for i in range(1, len([Link])):
if i > 1:
print(' ', end='')
print([Link][i], end='')
print()
source code for [Link]

[Link] COMP(2041|9044) 26T1 — More on Python 11 / 46

Example - /bin/echo using just for

# Python implementation of /bin/echo

import sys
if [Link][1:]:
print([Link][1], end='')
for arg in [Link][2:]:
print('', arg, end='')
print()
source code for [Link]

[Link] COMP(2041|9044) 26T1 — More on Python 12 / 46

Example - /bin/echo - two other versions

# Python implementation of /bin/echo

import sys
print(' '.join([Link][1:]))
source code for [Link]

# Python implementation of /bin/echo

import sys
print(*argv[1:])
source code for [Link]

[Link] COMP(2041|9044) 26T1 — More on Python 13 / 46

Example - Summing Command-line Arguments

# sum integers supplied as command line arguments

# no check that arguments are integers
import sys
total = 0
for arg in [Link][1:]:
total += int(arg)
print("Sum of the numbers is", total)
source code for sum_arguments.[Link]

[Link] COMP(2041|9044) 26T1 — More on Python 14 / 46

Example - Summing Command-line Arguments with Checking

# sum integers supplied as command line arguments

import sys
total = 0
for arg in [Link][1:]:
try:
total += int(arg)
except ValueError:
print(f"error: '{arg}' is not an integer", file=[Link])
[Link](1)
print("Sum of the numbers is", total)
source code for sum_arguments.[Link]

[Link] COMP(2041|9044) 26T1 — More on Python 15 / 46

Example - Counting Lines on stdin

# Count the number of lines on standard input.

import sys
line_count = 0
for line in [Link]:
line_count += 1
print(line_count, "lines")
source code for line_count.[Link]

[Link] COMP(2041|9044) 26T1 — More on Python 16 / 46

Example - Counting Lines on stdin - two more versions

import sys
lines = [Link]()
line_count = len(lines)
print(line_count, "lines")
source code for line_count.[Link]

import sys
lines = list([Link])
line_count = len(lines)
print(line_count, "lines")
source code for line_count.[Link]

[Link] COMP(2041|9044) 26T1 — More on Python 17 / 46

Opening Files

Similar to C, file objects can be created via the open function:

# read from file 'data'

file = open('data')

# read from file 'data'

file = open('data', 'r')

# write to file 'results'

file = open("results", "w")

# append binary data to file 'stuff'

file = open('stuff', 'ab')

[Link] COMP(2041|9044) 26T1 — More on Python 18 / 46

Closing Files

File objects can be explicitly closed with [Link]()

All file objects closed on exit.

Original file objects are not closed if opened again, can cause issues in long running programs.
Data on output streams may be not written (buffered) until close - hence close ASAP.

[Link] COMP(2041|9044) 26T1 — More on Python 19 / 46

Reading and Writing a File: Example

file = open("[Link]", "r")

data = [Link]()
[Link]()

file = open("[Link]", "w")

[Link](data)
[Link]()

[Link] COMP(2041|9044) 26T1 — More on Python 20 / 46

Exceptions
Opening a file may fail - always check for exceptions:

try:
file = open('data')
except OSError as e:
print(e)

OSError is a group of errors that can be cased by syscalls, similar to errno in C

Specific errors can be caught

try:
file = open('data')
except PermissionError:

# handle first error type

...
except FileNotFoundError:

# handle second error type

[Link] COMP(2041|9044) 26T1 — More on Python 21 / 46
Context Managers

Closing files is annoying and error-prone. Python can do it for us with a context manager. The file will be closed
when execution leaves the code block.

sum = 0
with open("data", "r") as input_file:
for line in input_file:
try:
sum += int([Link]())
except ValueError:
pass
print(sum)

[Link] COMP(2041|9044) 26T1 — More on Python 22 / 46

Example - cp

# Simple cp implementation for text files using line-based I/O

# explicit close is used below, a with statement would be better
# no error handling
import sys
if len([Link]) != 3:
print("Usage:", [Link][0], "<infile> <outfile>", file=[Link])
[Link](1)
infile = open([Link][1], "r", encoding="utf-8")
outfile = open([Link][2], "w", encoding="utf-8")
for line in infile:
print(line, end='', file=outfile)
[Link]()
[Link]()
source code for [Link]

[Link] COMP(2041|9044) 26T1 — More on Python 23 / 46

Example - cp

# Simple cp implementation for text files using line-based I/O

# and with statement, but no error handling
import sys
if len([Link]) != 3:
print("Usage:", [Link][0], "<infile> <outfile>", file=[Link])
[Link](1)
with open([Link][1]) as infile:
with open([Link][2], "w") as outfile:
for line in infile:
[Link](line)
source code for [Link]

[Link] COMP(2041|9044) 26T1 — More on Python 24 / 46

Example - cp

# Simple cp implementation for text files using line-based I/O

# and with statement and error handling
import sys
if len([Link]) != 3:
print("Usage:", [Link][0], "<infile> <outfile>", file=[Link])
[Link](1)
try:
with open([Link][1]) as infile:
with open([Link][2], "w") as outfile:
for line in infile:
[Link](line)
except OSError as e:
print([Link][0], "error:", e, file=[Link])
[Link](1)
source code for [Link]

[Link] COMP(2041|9044) 26T1 — More on Python 25 / 46

Example - cp

# Simple cp implementation for text files using line-based I/O

# reading all lines into array (not advisable for large files)
import sys
if len([Link]) != 3:
print("Usage:", [Link][0], "<infile> <outfile>", file=[Link])
[Link](1)
try:
with open([Link][1]) as infile:
with open([Link][2], "w") as outfile:
lines = [Link]()
[Link](lines)
except OSError as e:
print([Link][0], "error:", e, file=[Link])
[Link](1)
source code for [Link]

[Link] COMP(2041|9044) 26T1 — More on Python 26 / 46

Example - cp

# Simple cp implementation using [Link]

import sys
from shutil import copyfile
if len([Link]) != 3:
print("Usage:", [Link][0], "<infile> <outfile>", file=[Link])
[Link](1)
try:
copyfile([Link][1], [Link][2])
except OSError as e:
print([Link][0], "error:", e, file=[Link])
[Link](1)
source code for [Link]

[Link] COMP(2041|9044) 26T1 — More on Python 27 / 46

Example - cp

# Simple cp implementation by running /bin/cp

import subprocess
import sys
if len([Link]) != 3:
print("Usage:", [Link][0], "<infile> <outfile>", file=[Link])
[Link](1)
p = [Link](['cp', [Link][1], [Link][2]])
[Link]([Link])
source code for [Link]

[Link] COMP(2041|9044) 26T1 — More on Python 28 / 46

UNIX-filter Behavior

fileinput can be used to get UNIX-filter behavior.

treats all command-line arguments as file names
opens and reads from each of them in turn
no command line arguments, then fileinput == stdin
accepts - as stdin
so this is cat in Python:

#! /usr/bin/env python3

import fileinput

for line in [Link]():

print(line)

[Link] COMP(2041|9044) 26T1 — More on Python 29 / 46

Python Dicts

many languages have arrays accessed with small integer indexes.

can be though of as a mapping integer -> value
Python has lists (see widely used package numpy for arrays)
easy to implement indexing
some languages have associative arrays - index doesn’t have to be integer
very useful, e.g. being able to use string as index
harder to implement indexing
Python has dicts - index can be almost any value
index value can not be mutable, e.g. can not be list or dict
can be though of as a mapping integer -> value

[Link] COMP(2041|9044) 26T1 — More on Python 30 / 46

Example - Remembering Snap - Dict

# Check if we've seen a line read from stdin,

# using a dict.
# Print snap! if a line has been seen previously
# Exit if an empty line is entered
line_count = {}
while True:
try:
line = input("Enter line: ")
except EOFError:
break
if line in line_count:
print("Snap!")
else:
line_count[line] = 1
source code for snap_memory.[Link]

[Link] COMP(2041|9044) 26T1 — More on Python 31 / 46

Example - Remembering Snap - Set

# Check if we've seen lines read from stdin,

# using a set.
# Print snap! if a line has been seen previously.
# Exit if an empty line is entered
lines_seen = set()
while True:
try:
line = input("Enter line: ")
except EOFError:
break
if line in lines_seen:
print("Snap!")
else:
lines_seen.add(line)
source code for snap_memory.[Link]

[Link] COMP(2041|9044) 26T1 — More on Python 32 / 46

Some Useful Python Dict Operations

These can be applied to dicts.

d[key] Return the item of d with key key

del d[key] Remove d[key] from d. Raises a KeyError if key is not in the map.
key in d Return True if d has a key key, else False.
key not in d Equivalent to not key in d.
keys() Return a new view of the dictionary’s keys
items() Return a new view of the dictionary’s items
get(key[, default]) Return the value for key if key is in the dictionary, else default
values() Return a new view of the dictionary’s values.
update([other]) Update the dictionary with the key/value pairs from other
setdefault(key[, default]) If key is in the dictionary, return its value. If not, insert and return default.
clear() Remove all items from the dictionary.
copy() Return a shallow copy of the dictionary.

[Link] COMP(2041|9044) 26T1 — More on Python 33 / 46

Running External Programs with subprocess

Python requires you to import the subprocess module to run external programs.

[Link]() is usually the function used to run external programs.

[Link]() can be used if lower level control is necessary.
>>> [Link](['date', '--utc'])
Tue 05 Aug 1997 01:11:01 UTC
CompletedProcess(args=['date', '--utc'], returncode=0)
>>>

By default stdout/stderr from the program gores directly to Python’s stdout/stderr.

By default stdin from the program comes directly From Python’s stdin.

[Link] COMP(2041|9044) 26T1 — More on Python 34 / 46

Capturing the output from an External Programs with subprocess

To capture the output from commands:

>>> p = [Link](["date"], capture_output=True, text=True)

>>> [Link]
'Mon 18 Jul 2022 10:27:28 AEST\n'
>>> [Link]
0
>>> q = [Link](["ls", "no-existent-file"], capture_output=True,
↪ text=True)
>>> [Link]
"ls: cannot access 'no-existent-file': No such file or directory\n"
>>> [Link]
2

captured output is a byte sequence (binary) by default.

the option text=True converts it to a string
we want this 90+% of time
assumes the binary is utf-8 (if that is the local encoding)

[Link] COMP(2041|9044) 26T1 — More on Python 35 / 46

Passing input to an External Programs with subprocess

To send input to a program:

>>> message = "I love COMP(2041|9044)\n"

>>> p = [Link](["tr", "a-z", "A-Z"], input=message,
↪ capture_output=True, text=True)
>>> [Link]
'I LOVE COMP(2041|9044)\n'
>>> # note, you don't need an external program for this
>>> [Link]()
'I LOVE COMP(2041|9044)\n'

[Link] COMP(2041|9044) 26T1 — More on Python 36 / 46

Example - Using Subprocess to Capture

import subprocess
p = [Link](["date"], capture_output=True, text=True)
if [Link] != 0:
print([Link])
exit(1)
weekday, day, month, year, time, timezone = [Link]()
print(f"{year} {month} {day}")
source code for parse_date.py

[Link] COMP(2041|9044) 26T1 — More on Python 37 / 46

Python and External Commands

Optionally subprocess can pass the command to a shell to evaluate, e.g.:

>>> [Link]("sort *.csv | cut -d, -f1,7 >[Link]", shell=True)

This conveniently allows use of shell features including pipes, I/O re-direction, globbing …

Beware, this can also prodsuce unexpected behaviour, e.g. if a Shell metacharacter appears in a filename.

Beware, this a common source of security vulnerabilties. It should be avoided when security is important.

[Link] COMP(2041|9044) 26T1 — More on Python 38 / 46

Serving Web Pages with Python

Python includes a http server - easy to use for development/testing.

>>> server_address = ('', 2041)

>>> handler = [Link]
>>> with [Link](server_address, handler) as h:
... h.serve_forever()

And there is a convenient command-line short cut:

$ echo hello from httpd >[Link]

$ python3 -m [Link] 2041
Serving HTTP on [Link] port 2041 ([Link] ...
[Link] - - [17/Jul/2023 10:19:00] "GET /[Link] HTTP/1.1" 200 -

in another terminal

$ curl -s [Link]
hello from httpd

[Link] COMP(2041|9044) 26T1 — More on Python 39 / 46

Example - Using Subprocess to Capture Curl Output
# Repeatedly download a specified web page
# until a specified regexp matches its source
# then notify the specified email address.
# implemented using subprocess
import re
import subprocess
import sys
import time
REPEAT_SECONDS = 300 # check every 5 minutes
if len([Link]) == 4:
url = [Link][1]
regexp = [Link][2]
email_address = [Link][3]
else:
print(f"Usage: {[Link][0]} <url> <regex> <email-address>",
↪ file=[Link])
[Link](1)
source code for watch_website.[Link]

[Link] COMP(2041|9044) 26T1 — More on Python 40 / 46

Example - Using Subprocess to Capture Curl Output
while True:
p = [Link](
["curl", "--silent", url], text=True, capture_output=True
)
webpage = [Link]
if not [Link](regexp, webpage):
[Link](REPEAT_SECONDS)
continue
mail_body = f"Generated by {[Link][0]}"
subject = f"website '{url}' now matches regex '{regexp}'"
# the echo is for testing, remove to really send email
[Link](["echo", "mail", "-s", subject], text=True,
↪ input=mail_body)
[Link](0)
source code for watch_website.[Link]

[Link] COMP(2041|9044) 26T1 — More on Python 41 / 46

Example - Using Urllib

while True:
response = [Link](url)
webpage = [Link]().decode()
if not [Link](regexp, webpage):
[Link](REPEAT_SECONDS)
continue
mail_body = f"Generated by {[Link][0]}"
subject = f"website '{url}' now matches regex '{regexp}'"
# the echo is for testing, remove to really send email
[Link](["echo", "mail", "-s", subject], text=True,
↪ input=mail_body)
[Link](0)
source code for watch_website.[Link]

[Link] COMP(2041|9044) 26T1 — More on Python 42 / 46

Example - Using Beautiful Soup
import bs4 as BeautifulSoup
IGNORE_WEBPAGE_ELEMENTS = set("[document] head meta style script
↪ title".split())
for url in [Link][1:]:
response = [Link](url)
webpage = [Link]().decode()
soup = [Link](webpage, "html5lib")
for element in [Link](text=True):
parent = [Link]()
if parent in IGNORE_WEBPAGE_ELEMENTS:
continue
text = [Link]()
# remove empty lines and leading whitespace
text = [Link](r"\n\s+", "\n", element)
text = [Link]()
if text:
print(text)
source code for fetch_website_text.py

[Link] COMP(2041|9044) 26T1 — More on Python 43 / 46

Example - File Operations

# Change the names of the specified files to lower case.

# (simple version of the Perl utility rename)
import os
import sys
for old_pathname in [Link][1:]:
new_pathname = old_pathname.lower()
if new_pathname == old_pathname:
continue
if [Link](new_pathname):
print(f"{[Link][0]}: '{new_pathname}' exists", file=[Link])
continue
try:
[Link](old_pathname, new_pathname)
except OSError as e:
print(f"{[Link][0]}: '{new_pathname}' {e}", file=[Link])
source code for rename_lower_case.py

[Link] COMP(2041|9044) 26T1 — More on Python 44 / 46

Type hints
Python doesn’t enforce types even when they are given, thus they are hints

Static type checkers are common that do enforce types as much as possible

For best results type enforcement should be including in your code

Type hints help you and others read your code and are highly recommended

from typing import Optional, Union

a = 5
b = "Hello World"
# a type hint
c: int = 6
# but not enforced
d: int = "this isn't an int"
# composition of types
e: list[int] = [1, 2, 3, 4, 5]
# more composition of types
f: dict[int, list[tuple[str, str]]] = {1: [('a', 'b'), ('a', 'c')], 3: [('c',
↪ 's'), ('c', 'g')]}
[Link] COMP(2041|9044) 26T1 — More on Python 45 / 46
Type hints
from typing import Optional, Union

# `Optional` allows for None values

g: Optional[float] = None
# `Union` allows for two or more types
h: Union[int, float] = 4
# type hints can also be used on function arguments and return values
def func(a: int, b: str = 'Hi\n') -> int:
return len(b * a)
# for variables used in loops, tuple unpacking, or assignment can be
↪ pre-hinted
# pre-hinting does not define the variable as it has not assigned a value and
↪ python variables must always be initialised
j: int
for j in range(0, 100):
pass

k: bool
if k := validate(data):
pass

l: bool
m: int
n: str
l, m, n = (True, 99, "Apple")

# a variables type can be changed by first deleting it then redefining it

o: int = 0
del o
o: str = ""

[Link] COMP(2041|9044) 26T1 — More on Python 46 / 46