Page 1 of 343

Getting started with PythonLanguage

Python
It was invented by Dutch programmer Guido van Rossum in the year 1991 Feb 20. During the
winter of 1989, Van Rossum began programming Python as a hobby to keep him occupied
during the holidays. According to Van Rossum, he wanted to write a programming language that
was accessible to everyone. The name of the Python programming language came from an old
BBC television comedy series called Monty Python's Flying Circus.
 Python is easy to learn
 Python is beginner friendly
 Python is a program language which looks similar to English
 Python is opensource (We don’t need to pay any penny for downloading and using it)
 It is having largest community support
 Easy syntax compared to other programming languages (Syntax is nothing but Grammer)
 Python is having some mechanism automatically happening.
 It is the futuristic language
What is Language?
The way of communicating with other humans, Living Beings, in an understandable way by
speech or writing or gesture.

What is Programming Language?

The way a human being can interact (gives instructions and makes some work done which is felt
burden) with computer is known as Programming Language.
*As per the various resource and latest report there are 250-2500 coding languages.

Why to interact with computer?

As a human, if we feel some work is difficult, we can make that work done by our
computer. If you want to guide your computer to do work for us, we have to interact with
computer.
Meaning, we will give instructions to the computer to make our work done.
What is Coding?
Coding is nothing but changing and writing code from one language to another, Coding is a
part of programming. Coding is the first step to becoming a developer. it is much easier and
simpler to analyze and understand than Programming.
Code: -Code is a system of words, letters, figures, or symbols used to represent for the
purpose.

Importance of Coding?
Now a days computer becomes part of everyone’s life and to serve the human needs,
multiple software’s has been creating.
Creating a software meaning giving set of instructions to the computer, Computer will
understand them and perform some actions accordingly.

What is Python?
Python is the most popular programming languages in today’s world. With its quickly
updated libraries and the ease of code.
 Page 2 of 343

Technically, we can define, Python is general purpose, dynamically typed, interpreted and
high-level programming language, which is used to create any kind of programs.

Dynamically typed programming language means, Type of the variable will be determined
during the run time, based on the value that particular variable is holding.
Interpreted in simple terms means running code line by line. It also means that the
instruction is executed without earlier compiling the whole program into machine language.
High-level programming language because its syntax so closely resembles the English
language. Higher-level means it’s more readable to humans and less readable to computers.
General-purpose language, which means it's designed to be used in a range of applications,
including data science, software and web development, automation, and generally getting
stuff done.

Future of python
As python Is the futuristic language it is having high demand in the following fields
 Data Science
 Data Analytics
 Data Visualization
 Task Automation
 Artificial Intelligence
 Machine learning
 Internet of things
 Web Development
 Game Development and many more.

Sample Program:
a = 10
b = 20
c = a+b
print (c)

Explanation: In my personal view program is nothing but the step-by-step process following
by our brain to find the solution for a problem.
How our brain remembers the values before performing some calculations, Computer
program also Remembers values using variables and later performs an operation and prints
the output.

Program Readability:
It can be defined by the ease with which software is read and understood.
Readable code is the simply code that clearly communicates its intent to the reader. Most
likely the code we write will be read by other developers, who will either want to
understand or modify the way our code works.
Rules to write readable code:
1. Think about other people in your team, everyone in team should understand the
code clearly.
2. Make reusable code and avoid copy-pasting.
3. Leave your code a bit better than you found it.
 Page 3 of 343

4. Keep your modules, classes and functions small.

5. Maintain consistent code styles.
6. Trust your feelings about your code.
7. Use second fair of eyes.

Verify if Python is installed

To confirm that Python was installed correctly, you can verify that by running the
following command in your favorite terminal (If you are using Windows OS, you
need to add path of python to the environment variable before using it in command
prompt):

Python 3.x Version ≥ 3.0

If you have Python 3 installed, and it is your default version (see Troubleshooting for
more details) you should see something like this:

Hello, World in Python using IDLE

Hello, World in Python using IDLE

IDLE is a simple editor for Python, that comes bundled with Python.

How to create Hello, World program in IDLE

Open IDLE on your system of choice.
 In older version of windows, it can be found at all programs under the windows
menu.
 In windows 8+, search for IDLE or find it in the apps that are present in your
system.

 On Unix-based (including Mac) systems you can open it from the shell by typing $
idle python_file.py

 It will open a shell with options along the top.

 In the shell, Ther is a prompt of three right angle brackets:

 Page 4 of 343

Now write the following code in the prompt:

Hit Enter

Hello World Python file

Create a new file [Link] that contains

the following line: Python 3.x Version ≥

3.0

You can use the Python 3 print

function:

In your terminal, navigate to the directory containing the file [Link].

Type python [Link] then hit the Enter

You should see Hello, World printed to the console.

You can also substitute [Link] with the path to your file. For example, if you
have the file in your home directory and your user is "user" on Linux, you can
type python /home/user/[Link].
 Page 5 of 343

Comments and Documentation

Single line, inline and multiline comments
Comments are used to explain code when the basic code itself isn't clear.

Python ignores comments, and so will not execute code in there, or raise syntax

errors for plain English sentences. Single-line comments begin with the hash

character (#) and are terminated by the end of line.

Single line comment:

Inline comment:

Comments spanning multiple lines have """ or ''' on either end. This is the
same as a multiline string, but they can be used as comments:

PEP8 defines guidelines for formatting Python code. Formatting code well is
important so you can quickly read what the code does.

The Print Function

Print basics In Python 3 and higher, print is a function rather than a keyword.
 Page 6 of 343

You can also pass a number of parameters to print:

Another way to print multiple parameters is by using a +

What you should be careful about when using + to print multiple parameters,
though, is that the type of the parameters should be the same. Trying to print the
above example without the cast to string first would result in an error, because it
would try to add the number 1 to the string "bar" and add that to the number 3.14.

This is because the content of print will be evaluated first:

Otherwise, using a + can be very helpful for a user to read output of variables In the
example below the output is very easy to read!

The script below demonstrates this

You can prevent the print function from automatically printing a newline by using the
end parameter:
 Page 7 of 343

If you want to write to a file, you can pass it as the parameter file:

this goes to the file!

Print parameters

You can do more than just print text. print also has several

parameters to help you. Argument ‘sep’ place a string between

arguments.

Do you need to print a list of words separated by a comma or some other string?

Argument end: use something other than a newline at the end

Without the end argument, all print () functions write a line and then go to the
beginning of the next line. You can change it to do nothing (use an empty string of ''),
or double spacing between paragraphs by using two newlines.

Argument file: send output to someplace other than [Link].

Now you can send your text to either stdout, a file, or StringIO and not care which you
are given. If it quacks like a file, it works like a file.
 Page 8 of 343

There is a fourth parameter flush which will forcibly flush the stream.

Creating variables and assigning values

To create a variable in Python, all you need to do is specify the variable name, and then
assign a value to it.

Python uses = to assign values to variables. There's no need to declare a variable in

advance (or to assign a data type to it), assigning a value to a variable itself declares and
initializes the variable with that value. There's no way to declare a variable without
assigning it an initial value.
 Page 9 of 343

Variable assignment works from left to right. So, the following will give you a syntax error.

You cannot use python's keywords as a valid variable name.

Now you know the basics of assignment, let's get this subtlety about assignment in python
out of the way.

When you use = to do an assignment operation, what's on the left of = is a name for the
object on the right. Finally, what = does is assign the reference of the object on the
right to the name on the left.
That is:
 Page 10 of 343

So, from many assignment examples above, if we pick pi = 3.14, then pi is a name
(not the name, since an object can have multiple names) for the object 3.14.

You can assign multiple values to multiple variables in one line. Note that there must
be the same number of arguments on the right and left sides of the = operator:

The error in last example can be obviated by assigning remaining values to equal
number of arbitrary variables. This dummy variable can have any name, but it is
conventional to use the underscore (_) for assigning unwanted values:

Note that the number of _ and number of remaining values must be equal.
Otherwise 'too many values to unpack error' is thrown as above:

You can also assign a single value to several variables simultaneously.

# Output: 1 1 1

When using such cascading assignment, it is important to note that all three variables a,
b and c refer to the same object in memory, an int object with the value of 1. In other
words, a, b and c are three different names given to the same int object. Assigning a
different object to one of them afterwards doesn't change the others, just as expected:
 Page 11 of 343

Even though there's no need to specify a data type when declaring a variable in
Python, while allocating the necessary area in memory for the variable, the Python
interpreter automatically picks the most suitable built-in type for it:
a= 2
print(type(a))
# Output: <type 'int'>

b = 9223372036854775807
print(type(b))
# Output: <type 'int'>

pi = 3.14
print(type(pi))
# Output: <type 'float'>

c = 'A'
print(type(c))
# Output: <type 'str'>

name = ‘Ankal’
print(type(name))
# Output: <type 'str'>

q = True
print(type(q))
# Output: <type 'bool'>

Datatypes
Built-in Types
Booleans
bool: A boolean value of either True or False. Logical operations like and, or, not can be
performed on booleans.

If boolean values are used in arithmetic operations, their integer values (1 and 0 for
True and False) will be used to return an integer result:
 Page 12 of 343

Numbers

int: Integer number

Integers in Python are of arbitrary sizes.

Note: in older versions of Python, a long type was available and this was distinct
from int. The two have been unified.

float: Floating point number precision depends on the implementation and

system architecture, for CPython the float datatype corresponds to a C double.

complex: Complex numbers

These are referred as True values and Imaginary values.

The <, <=, > and >= operators will raise a TypeError exception when any operand is a
complex number.

Strings

str: a unicode string.

The type of 'hello'

Conversion between datatypes

You can perform explicit datatype conversion.

For example, '123' is of str type and it can be converted to integer using int function.
 Page 13 of 343

Converting from a float string such as '123.456' can be done using float function.

You can also convert sequence or collection types

a = 'hello'
list(a) # ['h', 'e', 'l', 'l', 'o']
set(a) # {'o', 'e', 'l', 'h'}
tuple(a) # ('h', 'e', 'l', 'l', 'o')
1111111111111111
Operators:
Operators are used to evaluate an expression
Operators are symbols or keywords in programming that perform various operations on data or
variables. They are a fundamental part of any programming language and allow you to manipulate
and process data. Here are some common types of operators:

Arithmetic operators:
Addition (+): Adds two values together.
a=5
b=3
result = a + b #>> 8
Subtraction (-): Subtracts the right operand from the left operand.
x = 10;
y = 7;
result = x – y #>> 3
Multiplication (*): Multiplies two values.
num1 = 6;
num2 = 4;
result= num1 * num2 #>> 24
Division (/): Divides the left operand by the right operand.
dividend = 15.0;
divisor = 4.0;
quotient = dividend / divisor #>> 3.75
Modulus (%): Returns the remainder after division.
x = 10
y=3
reminder = x % y #>> 1
 Page 14 of 343

Exponentiation (or) ^ (or) ** >> Raises a number to a power (not available in all
programming languages).
base = 2
exponent = 3
result = base ** exponent #>> 8

Comparison operators:
These are used to compare multiple conditions
Comparison operators are -->>> ('<', '<=', '==', '!=', '>', '>=')
< Less than
<= less than or equals to
== Equal equals to
! = Not equals to
>Greater than
>= Greater than or equals to
# Example 1: Using if statement with comparison operators
x = 10
y=5

if x > y:
print ("x is greater than y")

# Example 2: Using if-else statement with comparison operators

temperature = 25

if temperature >= 30:

print ("It's a hot day.")
else:
print ("It's not a hot day.")

# Example 3: Using if-elif-else statement with multiple conditions

grade = 85

if grade >= 90:

print("A")
elif grade >= 80:
print("B")
elif grade >= 70:
print("C")
else:
print("D")
 Page 15 of 343

# Example 4: Using logical operators (and, or) with comparison operators

age = 25
income = 50000

if age >= 18 and income >= 30000:

print ("Eligible for a loan")

# Example 5: Using comparison operators with strings

name = "Alice"

if name == "Alice":
print ("Hello, Alice!")
else:
print ("You are not Alice.")

Logical operators:
`and`, `or`, `not`
Logical operators are used to combine multiple conditions

For and, it will check if all the conditions are satisfying or not. If all the conditions satisfies
then it will return True or executes the next statements.

A simple example
In Python you can compare a single element using two binary operators--one on either side:

In many (most?) programming languages, this would be evaluated in a way contrary to

regular math: (3.14 < x) < 3.142, but in Python it is treated like 3.14 < x and x <
3.142, just like most non-programmers would expect.
 Page 16 of 343

The 1's in the above example can be changed to any truth value, and the 0's can be changed
to any false value.
Or
Evaluates to the first truth argument if either one of the arguments is true. If both arguments
are false, evaluates to the second argument.
 Page 17 of 343

The 1's in the above example can be changed to any truth value, and the 0's can be changed
to any false value.
not
It returns the opposite of the following statement:

Boolean Logic Expressions

Boolean logic expressions, in addition to evaluating to True or False, return the value that
was interpreted as True
or False. It is Pythonic way to represent logic that might otherwise require an if-else test.

and operator
The and operator evaluates all expressions are True or not, If all the expressions are
True it will returns True or executes next statement. Otherwise it will returns false or
does not executes next statements.

>>> 2 1 and 2

>>> 0 1 and 0
>>> 1 and "Hello World"
 Page 18 of 343

or operator
The or operator Checks if any of the condition is True or not. If any of the condition is True
it will return True otherwise it will return False.

Lazy evaluation

When you use this approach, remember that the evaluation is lazy. Expressions that are
not required to be evaluated to determine the result are not evaluated. For example:

In the above example, print_me is never executed because Python can determine the entire
expression is False
when it encounters the 0 (False). Keep this in mind if print_me needs to execute to serve
your program logic.

Testing for multiple conditions

A common mistake when checking for multiple conditions is to apply the logic incorrectly.

This example is trying to check if two variables are each greater than 2. The statement
is evaluated as - if (a) and (b > 2). This produces an unexpected result because
bool(a) evaluates as True when a is not zero.
 Page 19 of 343

Each variable needs to be compared separately.

Another, similar, mistake is made when checking if a variable is one of multiple values.
The statement in this example is evaluated as - if (a == 3) or (4) or (6). This
produces an unexpected result because bool(4) and bool(6) each evaluate to True

Again each comparison must be made separately

Using the in operator is the canonical way to write this.

Assignment operators:
Assignment (=): Assigns the value on the right to the variable on the left.
Compound Assignment (+=, -=, *=, /=, %=): Performs an operation and assigns the result to
the left operand.
Assignment (=)
x = 10
Addition Assignment (+=)
count = 5
count += 3 #>>> count = count+3
Subtraction Assignment (-=)
total = 15
total -= 7 #>>> total = total-7
 Page 20 of 343

Multiplication Assignment (*=)

price = 2.5
price *= 3 #>>> price = price *3
Division Assignment (/=)
result = 20.0
result = 4 #>>> result = result/4
Bitwise Operators
Bitwise operations alter binary strings at the bit level. These operations are incredibly basic
and are directly supported by the processor. These few operations are necessary in working
with device drivers, low-level graphics, cryptography, and network communications. This
section provides useful knowledge and examples of Python's bitwise operators.

Bitwise NOT
The ~ operator will flip all of the bits in the number. Since computers use signed
number representations — most notably, the two's complement notation to encode
negative binary numbers where negative numbers are written with a leading one (1)
instead of a leading zero (0).

This means that if you were using 8 bits to represent your two's-complement
numbers, you would treat patterns from 0000 0000 to 0111 1111 to represent
numbers from 0 to 127 and reserve 1xxx xxxx to represent negative numbers.

Eight-bit two's-complement numbers

Bits Unsigned Value Two's-complement Value

0000 0000 0 0
0000 0001 1 1
0000 0010 2 2
0111 1110 126 126
0111 1111 127 127
1000 0000 128 -
128
1000 0001 129 -
127
1000 0010 130 -
126
1111 1110 254 -2
1111 1111 255 -1

In essence, this means that whereas 1010 0110 has an unsigned value of 166 (arrived at by
adding (128 * 1) +
(64 * 0 90 (
of -
 Page 21 of 343

0)), mplement value

it 0) - (4 * 1) - (2 *
has 1) -
a
two
's-
co
(1 * 0), and complementing the value).

In this way, negative numbers range down to -128 (1000 0000). Zero (0) is
represented as 0000 0000, and minus one (-1) as 1111 1111.

In general, though, this means ~n = -n - 1.

# 0 = 0b0000 0000
~0

# 1 = 0b0000 0001
~1

# -2 = 1111 1110

# 2 = 0b0000 0010
~2

# 123 = 0b0111 1011

~123

# -124 = 0b1000 0100

 Page 22 of 343

Note, the overall effect of this operation when applied to positive numbers can be
summarized:

~n -> -|n+1|

And then, when applied to negative numbers, the corresponding effect is:

~-n -> |n-1|

The following examples illustrate this last rule...

# 0 = 0b0000 0000

# 1 = 0b0000 0001

# -123 = 0b1111 1011

# 122 = 0b0111 1010

Bitwise XOR (Exclusive OR)

The ^ operator will perform a binary XOR in which a binary 1 is copied if and only if it is
the value of exactly one
operand. Another way of stating this is that the result is 1 only if the operands are different.
Examples include:

# 0 ^ 0 = 0
# 0 ^ 1 = 1
# 34 = 0b100010 # 1 ^ 0 = 1
# 1 ^ 1 = 0
 Page 23 of 343

Bitwise AND
The & operator will perform a binary AND, where a bit is copied if it exists in both
operands. That means:
# 60 = 0b111100 #
30 = 0b011110 60 &
30 # 0 & 0 = 0
# 28 = 0b11100 # 0 & 1 = 0
# 1 & 0 = 0
# 1 & 1 = 1

Bitwise OR
The | operator will perform a binary "or," where a bit is copied if it exists in either operand.
That means:

# 0 | 0 = 0
# 0 | 1 = 1
# 62 = 0b111110 # 1 | 0 = 1
# 1 | 1 = 1

Bitwise Left Shift

The << operator will perform a bitwise "left shift," where the left operand's value is
moved left by the number of bits given by the right operand.

# 2 = 0b10

# 8 = 0b1000

Performing a left bit shift of 1 is equivalent to multiplication by 2:

Performing a left bit shift of n is equivalent to multiplication by 2**n:

 Page 24 of 343

Bitwise Right Shift

The >> operator will perform a bitwise "right shift," where the left operand's value is
moved right by the number of bits given by the right operand.

# 8 = 0b1000

# 2 = 0b10

Performing a right bit shift of 1 is equivalent to integer division by 2:

Performing a right bit shift of n is equivalent to integer division by 2**n:

Identity operators:
Identity operators in Python are used to compare the memory addresses (identities) of two objects to
check whether they are the same object or not. Python provides two identity operators: is and is not.
Is
is used to compare the addresses of two different variables

a = [1, 2, 3]
b=a # Both 'a' and 'b' reference the same list object
c = [1, 2, 3] # A new list object with the same values

print(a is b) # True, as 'a' and 'b' reference the same list object
print(a is c) # False, as 'a' and 'c' reference different list object

is not
This returns True if both variables are not the same object
a = [1, 2, 3]
b = a # Both 'a' and 'b' reference the same list object
c = [1, 2, 3] # A new list object with the same values

print(a is not b) # False, as 'a' and 'b' reference the same list object
print(a is not c) # True, as 'a' and 'c' reference different list objects
 Page 25 of 343

Membership operators:
Membership operators are used to check whether specific value available or not

my_list = [1, 2, 3, 4, 5]

# Check if values are present in the list

print(3 in my_list) # True, because 3 is in the list
print(6 in my_list) # False, because 6 is not in the list

my_string = "Hello, World!"

# Check if substrings are absent in the string

print("Python" not in my_string) # True, because "Python" is not in the string
print("Hello" not in my_string) # False, because "Hello" is in the string

Comparison between `is` and `==`

A common pitfall is confusing the equality comparison

is used to compare the addresses of the variables

== used to compare the values of an assigned variables

a is b will compare the entities of a and b. To illustrate:

Basically, is can be thought of as shorthand for id(a) == id(b).

Beyond this, there are quirks of the run-time environment that further complicate
things. Short strings and small integers will return True when compared with is, due
to the Python machine attempting to use less memory for identical objects.
 Page 26 of 343

But longer strings and larger integers will be stored separately.

You should use is to test for None:

# None

Block Indentation
Python uses indentation to define control and loop constructs. This contributes to
Python's readability; however, it requires the programmer to pay close attention to the
use of whitespace. Thus, editor miscalibration could result in code that behaves in
unexpected ways.

Python uses the colon symbol (:) and indentation for showing where blocks of code
begin and end. That is, blocks in Python, such as functions, loops, if clauses and other
constructs, have no ending identifiers. All blocks start with a colon and then contain
the indented lines below it.

For example:

or
 Page 27 of 343

Blocks that contain exactly one single-line statement may be put on the same line, though
this form is generally not considered good style:

Attempting to do this with more than a single statement will not work:

An empty block causes an IndentationError. Use pass (a command that does

nothing) when you have a block with no content:

Spaces vs. Tabs

In short: always use 4 spaces for indentation.

User Input
Interactive input

To get input from the user, use the input function

The reminder of this example will be using Python 3 syntax.

The function takes a string argument, which displays it as a prompt and returns a string. The
above code provides a prompt, waiting for the user to input.

If the user types "Bob" and hits enter, the variable name will be assigned to the string
"Bob":
 Page 28 of 343

Note that the input is always of type str, which is important if you want the user to
enter numbers. Therefore, you need to convert the str before trying to use it as a
number:

Decision making statements

Conditional expressions, involving keywords such as if, elif, and else, provide Python
programs with the ability to perform different actions depending on a boolean
condition: True or False. This section covers the use of Python conditionals, boolean
logic, and ternary statements.
Conditional Expression (or "The TernaryOperator")
The ternary operator is used for inline conditional expressions. It is best used in simple,
concise operations that are easily read.

The order of the arguments is different from many other languages (such as C,
Ruby, Java, etc.), which may lead to bugs when people unfamiliar with Python's
"surprising" behaviour use it (they may reverse the order). Some find it

"unwieldy", since it goes contrary to the normal flow of thought (thinking of

the condition first and then the effects).

The result of this expression will be as it is read in English - if the conditional

expression is True, then it will evaluate to the expression on the left side, otherwise,
the right side.

Ternary operations can also be nested, as here:

 Page 29 of 343

They also provide a method of including conditionals in lambda functions.

if, if else, elif(if else if ladder), and else

In Python you can define a series of conditionals using if for the first one, elif for the
rest, up until the final (optional) else for anything not caught by the other conditionals.

Outputs Number is bigger than 2

Using else if instead of elif will trigger a syntax error and is not allowed.

T r u t h Values
The following values are considered false, in that they evaluate to False when applied to a
boolean operator.

NoneFalse

NoneFalse

0, or any numerical value equivalent to zero, for example 0L, 0.0, 0j

Empty sequences: '', "", (), []
Empty mappings: {}
User-defined types where the bool or len methods return 0 or False

All other values in Python evaluate to True.

Note: A common mistake is to simply check for the Falseness of an operation which
returns different Falsey values where the difference matters. For example, using if
foo() rather than the more explicit if foo() is None
 Page 30 of 343

If statement

The if statements check the condition. If it evaluates to True, it executes the body of the
if statement. If it evaluates to False, it skips the body.

The condition can be any valid expression:

Elif statement
Certainly! Here are two examples of if-elif-else ladders in Python:

In this example, we'll create a simple grading system that takes a student's score as input and assigns a grade
based on the score.

# Input the student's score

score = int(input("Enter the student's score: "))

# Determine the grade based on the score

if score >= 90:
grade = "A"
elif score >= 80:
grade = "B"
elif score >= 70:
grade = "C"
elif score >= 60:
grade = "D"
else:
grade = "F"

# Display the grade

 Page 31 of 343

print(grade)

Else statement

The else statement will execute its body only if conditional statements all evaluate to False.

Loops
Loops are used to execute some set of statements for multiple times.

Parameter Details
Boolean expression that can be evaluated in a Boolean context, e.g. x < 10variable name
for the current element from the iterable anything that implements iterations.
As one of the most basic functions in programming, loops are an important piece to nearly
every programming language. Loops enable developers to set certain portions of their code
to repeat through a number of loops which are referred to as iterations. This topic covers
using multiple types of loops and applications of loops in Python.

While Loop
A while loop will cause the loop statements to be executed until the loop condition is false.
The following code will execute the loop statements a total of 4 times.

While the above loop can easily be translated into a more elegant for loop, while loops
 Page 32 of 343

are useful for checking if some condition has been met. The following loop will
continue to execute until my Object is ready.

while loops can also run without a condition by using numbers (complex or real) or True:

If the condition is always true the while loop will run forever (infinite loop) if it is not
terminated by a break or return statement or an exception.

Break and Continue in Loops

break statement
break is used to come out of the loop.

(In other words)

When a break statement executes inside a loop, control flow "breaks" out of the loop
immediately:

The loop conditional will not be evaluated after the break statement is executed. Note
that break statements are only allowed inside loops, syntactically. A break statement
inside a function cannot be used to terminate loops that called that function.
 Page 33 of 343

Executing the following prints every digit until number 4 when the break statement is met
and the loop stops:

0
1
2
3
4
Breaking from loop

continue statement
continue is used to skip the current iteration and moves to the next iteration.

(In other words)

A continue statement will skip to the next iteration of the loop bypassing the rest of the
current block but continuing the loop. As with break, continue can only appear inside
loops:
# Initialize a variable
count = 0
# Start a while loop
while count < 5:
count += 1
# Check if count is even
if count % 2 == 0:
# If count is even, skip this iteration and continue with the next
continue
print(count)
For loops
Fetching the data from some container (Data structure) and it is using the data in the loop block .
In for loop, we are not having any initial value, Condition checking not increment or decrement.
(In other words)

for loops iterate over a collection of items, such as list or dict, and run a block of code
with each element from the collection.
 Page 34 of 343

Iterating over lists

To iterate through a list, you can use for:

This will print out the elements of the list:

The above for loop iterates over a list of numbers.

Each iteration sets the value of i to the next element of the list. So first it will be 0, then
1, then 2, etc. The output will be as follow:

0
1
2
3
4

Range is the function given by python which is used to generate the sequence of numbers.
(In other words)
range is a function that returns a series of numbers under an iterable form, thus it can be
used in for loops:

gives the exact same result as the first for loop. Note that 5 is not printed as the range
here is the first five numbers counting from 0.
The range function generates numbers which are also often used in a for loop.

The result will be a special range sequence type in python >=3 and a list in python <=2.
Both can be looped through using the for loop.
 Page 35 of 343

1
2
3
4
5

break statements can also be used inside for loops, the other looping construct provided by
Python:

Executing this loop now prints:

0
1
2

Note that 3 and 4 are not printed since the loop has ended.
If a loop has an else clause, it does not execute when the loop is terminated through a break
statement.

continue statement

A continue statement will skip to the next iteration of the loop bypassing the rest of the
current block but continuing the loop. As with break, continue can only appear inside
loops:

Note that 2 and 4 aren't printed, this is because continue goes to the next iteration instead of
continuing on to
print(i) when i == 2 or i == 4.

Nested Loops

break and continue only operate on a single level of loop. The following example will only
 Page 36 of 343

break out of the inner

for loop, not the outer while loop:

Python doesn't have the ability to break out of multiple levels of loop at once -- if this
behavior is desired, refactoring one or more loops into a function and replacing break
with return may be the way to go.

Use return from within a function as a break

The return statement exits from a function, without executing the code that comes after it.

If you have a loop inside a function, using return from inside that loop is equivalent to
having a break as the rest of the code of the loop is not executed (note that any code after
the loop is not executed either):

If you have nested loops, the return statement will break all loops:

will output:

Loops with an "else" clause

The for and while compound statements (loops) can optionally have an else clause (in
practice, this usage is fairly rare).
 Page 37 of 343

The else clause only executes after a for loop terminates by iterating to completion, or
after a while loop terminates by its conditional expression becoming false.

output:

0
1
2
done

The else clause does not execute if the loop terminates some other way (through a break
statement or by raising an exception):

output:

0
1

Most other programming languages lack this optional else clause of loops. The use of the
keyword else in particular is often considered confusing.

The original concept for such a clause date back to Donald Knuth and the meaning of the
else keyword becomes clear if we rewrite a loop in terms of if statements and go to
statements from earlier days before structured programming or from a lower-level
assembly language.
 Page 38 of 343

For example:

These remain equivalent if we attach an else clause to each of them.

For example:

A for loop with an else clause can be understood the same way. Conceptually, there is a
loop condition that remains True as long as the iterable object or sequence still has some
remaining elements.

Why would one use this strange construct?

The main use case for the for...else construct is a concise implementation of search as for
instance:

To make the else in this construct less confusing one can think of it as "if not break" or "if not
found".

The Pass Statement

pass is a null statement for when a statement is required by Python syntax (such as
within the body of a for or while loop), but no action is required or desired by the
programmer. This can be useful as a placeholder for code that is yet to be written.
 Page 39 of 343

In this example, nothing will happen. The for loop will complete without error, but no
commands or code will be actioned. pass allows us to run our code successfully without
having all commands and action fully implemented.

Similarly, pass can be used in while loops, as well as in selections and function definitions etc.

Collection Types
There are a number of collection types in Python. While types such as int and str hold a
single value, collection types hold multiple values.

Lists
The list type is probably the most commonly used collection type in Python. Despite its
name, a list is more like an array in other languages, mostly JavaScript. In Python, a list is
merely an ordered collection of valid Python values. A list can be created by enclosing
values, separated by commas, in square brackets

The Python List is a general data structure widely used in Python programs. They are
found in other languages, often referred to as dynamic arrays. They are both mutable and a
sequence data type that allows them to be indexed and sliced. The list can contain different
types of objects, including other list objects.

A list can be empty:

The elements of a list are not restricted to a single data type, which makes sense given that
Python is a dynamic language:

A list can contain another list as its element:

The elements of a list can be accessed via an index, or numeric representation of their position.
Lists in Python are
 Page 40 of 343

zero-indexed meaning that the first element in the list is at index 0, the second element is at
index 1 and so on:

Indexes can also be negative which means counting from the end of the list (-1 being the
index of the last element). So, using the list from the above example:

Lists are mutable, so you can change the values in a list:

Besides, it is possible to add and/or remove elements from a list:

Append object to end of list with [Link](object), returns None.

Add a new element to list at a specific index. [Link](index, object)

Remove the first occurrence of a value with [Link](value), returns None

Get the index in the list of the first item whose value is x. It will show an error if there is no
such item.

Count length of list

count occurrence of any item in list

 Page 41 of 343

Reverse the list

Remove and return item at index (defaults to the last item) with [Link]([index]), returns
the item

You can iterate over the list elements like below:

List methods and supported operators

Starting with a given list a:

1. append(value) – appends a new element to the end of the list.

Note that the append() method only appends one new element to the end of the list. If
you append a list to another list, the list that you append becomes a single element at
the end of the first list.
 Page 42 of 343

2. extend(enumerable) – extends the list by appending elements from another enumerable.

Lists can also be concatenated with the + operator. Note that this does not modify any of
the original lists:

3. index(value, [startIndex]) – gets the index of the first occurrence of the input
value. If the input value is not in the list a ValueError exception is raised. If a
second argument is provided, the search is started at that specified index.

4. insert (index, value) – inserts value just before the specified index. Thus,
after the insertion the new element occupies position index.

5. pop([index]) – removes and returns the item at index. With no argument it

removes and returns the last element of the list.
 Page 43 of 343

5, 6, 7, 7, 8, 9, 10]

5, 6, 7, 8, 9, 10]
remove(value) – removes the first occurrence of
the specified value. If the provided value cannot be found, aValueError is raised.

[Link](0)
[Link](9)
# a: [1, 2, 3, 4, 5, 6, 7, 8]
[Link](10)
# ValueError, because 10 is not in a

[Link]() – reverses the list in-place and returns None.

There are also other ways of reversing a list.

[Link](value) – counts the number of occurrences of some value in the list.

[Link]() – sorts the list in numerical and lexicographical order and returns None.

[Link]() – removes all items from the list

[Link] – multiplying an existing list by an integer will produce a larger list

consisting of that many copies of the original. This can be useful for example for list
initialization:
 Page 44 of 343

Take care doing this if your list contains references to objects (eg a list of lists), see
Common Pitfalls - List multiplication and common references.

[Link] deletion – it is possible to delete multiple elements in the list using the
del keyword and slice notation:

[Link]
The default assignment "=" assigns a reference of the original list to the new name.
That is, the original name and new name are both pointing to the same list object.
Changes made through any of them will be reflected in another. This is often not what
you intended.

If you want to create a copy of the list you have below options.

You can slice it:

You can use the built in list() function:

You can use generic [Link]():

 Page 45 of 343

This is a little slower than list() because it has to find out the datatype of old_list first.

If the list contains objects and you want to copy them as well, use generic
[Link]():

Obviously the slowest and most memory-needing method, but sometimes unavoidable.
copy() – Returns a shallow copy of the list

Accessing list values

Python lists are zero-indexed, and act like arrays in other languages.

Attempting to access an index outside the bounds of the list will raise an IndexError.

Negative indexes are interpreted as counting from the end of the list.

This is functionally equivalent to

Lists allow to use slice notation as lst[start:end:step]. The output of the slice
notation is a new list containing elements from index start to end-1. If options are
omitted start defaults to beginning of list, end to end of list and step to 1:

lst[1:] # [2, 3, 4]
lst[:3] # [1, 2, 3]
lst[::2] # [1, 3]
lst[::-1] # [4, 3, 2, 1]
lst[-1:0:-1] # [4, 3, 2]

With this in mind, you can print a reversed version of the list by calling
 Page 46 of 343

When using step lengths of negative amounts, the starting index has to be greater than
the ending index otherwise the result will be an empty list.

Using negative step indexes are equivalent to the following code:

The indexes used are 1 less than those used in negative indexing and are reversed.

Checking if list is empty

The emptiness of a list is associated to the boolean False, so you len(lst) == 0, but just lst
don't have to check or not lst

Iterating over a list

Python supports using a for loop directly on a list:

You can also get the position of each item at the same time:
 Page 47 of 343

The other way of iterating a list based on the index value:

Note that changing items in a list while iterating on it may have unexpected results:

for item in my_list:

if item == 'foo':
del my_list[0]
print(item)

# Output: foo
# Output: baz
 Page 48 of 343

Checking whether an item is in a list

Python makes it very simple to check whether an item is in a list. Simply use the in operator.

Note: the in operator on sets is asymptotically faster than on lists. If you need to
use it many times on potentially large lists, you may want to convert your list to
a set, and test the presence of elements on the set.

Any and All

You can use all() to determine if all the values in an iterable evaluate to True

Likewise, any() determines if one or more values in an iterable evaluate to True

While this example uses a list, it is important to note these built-ins work with any iterable,
including generators.
 Page 49 of 343

Reversing list elements

You can use the reversed function which returns an iterator to the reversed list:

Note that the list "numbers" remains unchanged by this operation, and remains in the same
order it was originally. To reverse in place, you can also use the reverse method.
You can also reverse a list (actually obtaining a copy, the original list is unaffected) by using
the slicing syntax, setting the third argument (the step) as -1:

Concatenate and Merge lists

1. The simplest way to concatenate list1 and list2:

2. zip returns a list of tuples, where the i-th tuple contains the i-th element from
each of the argument sequences or iterables:

If the lists have different lengths then the result will include only as many elements as
the shortest one:
 Page 50 of 343

For padding lists of unequal length to the longest one with Nones use
itertools.zip_longest
(itertools.izip_longest in Python 2)

# None None c4

3. Insert to a specific index values:

Output:

Length of a list
Use len() to get the one-dimensional length of a list.

len() also works on strings, dictionaries, and other data structures

similar to lists. Note that len() is a built-in function, not a method

of a list object.

Also note that the cost of len() is O(1), meaning it will take the same amount of time to
get the length of a list regardless of its length.
 Page 51 of 343

Remove duplicate values in list

Removing duplicate values in a list can be done by converting the list to a set (that is an
unordered collection of distinct objects). If a list data structure is needed, then the set can
be converted back to a list using the function list():

Note that by converting a list to a set the original ordering is lost.

Comparison of lists
It's possible to compare lists and other sequences lexicographically using comparison
operators. Both operands must be of the same type.

If one of the lists is contained at the start of the other, the shortest list wins.

Accessing values in nested list

Starting with a three-dimensional list:

Accessing items in the list:

#2

#10
 Page 52 of 343

Performing support operations:

#11

Using nested for loops to print the list:

Note that this operation can be used in a list comprehension or even as a generator to produce
efficiencies, e.g.:

Not all items in the outer lists have to be lists themselves:

Another way to use nested for loops. The other way is better but I've needed to use this on
occasion:

#15

Using slices in nested list:

The final list:

 Page 53 of 343

Initializing a List to a Fixed Number of Elements

For immutable elements (e.g. None, string literals etc.):

For mutable elements, the same construct will result in all elements of the list referring to
the same object, for example, for a set:

Instead, to initialize the list with a fixed number of different mutable objects, use:

List comprehensions
List comprehensions in Python are concise, syntactic constructs. They can be utilized to
generate lists from other lists by applying functions to each element in the list. The
following section explains and demonstrates the use of these expressions.

List Comprehensions
A list comprehension creates a new list by applying an expression to each element of an
iterable. The most basic form is:

There's also an optional 'if' condition:

Each <element> in the <iterable> is plugged in to the <expression> if the (optional)

<condition> evaluates to true
. All results are returned at once in the new list. Generator expressions are evaluated lazily, but
list comprehensions evaluate the entire iterator immediately - consuming memory proportional
to the iterator's length.
 Page 54 of 343

To create a list of squared integers:

The for expression sets x to each value in turn from (1, 2, 3, 4). The result of the
expression x * x is appended to an internal list. The internal list is assigned to the
variable squares when completed.

Besides a speed increase (as explained here), a list comprehension is roughly equivalent to the
following for-loop:

The expression applied to each element can be as complex as needed:

else
else can be used in List comprehension constructs, but be careful regarding the syntax. The
if/else clauses should
be used before for loop, not after:

Note this uses a different language construct, a conditional expression, which itself is not
part of the comprehension syntax. Whereas the if after the for…in is a part of list
comprehensions and used to filter elements from the source iterable.
 Page 55 of 343

Double Iteration
Order of double iteration [... for x in ... for y in ...] is either natural or
counter-intuitive. The rule of thumb is to follow an equivalent for loop:

This becomes:

This can be compressed into one line as [str(x) for i in range(3) for x in
foo(i)]

Conditional List Comprehensions

Given a list comprehension you can append one or more if conditions to filter values.

For each <element> in <iterable>; if <condition> evaluates to True, add

<expression> (usually a function of
<element>) to the returned list.

For example, this can be used to extract only even numbers from a sequence of integers:

The above code is equivalent to:Also, a conditional list comprehension of the form [e for x
in y if c] (where e and c are expressions in terms o
 Page 56 of 343

x) is equivalent to list(filter(lambda x: c, map(lambda x: e, y))).

Despite providing the same result, pay attention to the fact that the former example is
almost 2x faster than the letter one. For those who are curious, this is a nice explanation of
the reason why.

Note that this is quite different from the ... if ... else ... conditional expression
(sometimes known as a ternary expression) that you can use for the <expression> part
of the list comprehension. Consider the following example:

Here the conditional expression isn't a filter, but rather an operator determining the value
to be used for the list items:

This becomes more obvious if you combine it with other operators:

If you are using Python 2.7, xrange may be better than range for several reasons as described
in the xrange
documentation.

The above code is equivalent to:

One can combine ternary expressions and if conditions. The ternary operator works on the
filtered result:
 Page 57 of 343

The same couldn't have been achieved just by ternary operator only:

See also: Filters, which often provide a sufficient alternative to conditional list comprehensions.

List Comprehensions with Nested Loops

List Comprehensions can use nested for loops. You can code any number of nested for loops
within a list comprehension, and each for loop may have an optional associated if test.
When doing so, the order of the for
constructs are the same order as when writing a series of nested for statements. The general
structure of list comprehensions looks like this:

For example, the following code flattening a list of lists using multiple for statements:

can be equivalently written as a list comprehension with multiple for constructs:

Changing Types in a List

Quantitative data is often read in as strings that must be converted to numeric types before
processing. The types of all list items can be converted with either a List Comprehension
or the map() function.
 Page 58 of 343

Nested List Comprehensions

Nested list comprehensions, unlike list comprehensions with nested loops, are List
comprehensions within a list comprehension. The initial expression can be any arbitrary
expression, including another list comprehension.

Iterate two or more list simultaneously withinlist comprehension

For iterating more than two lists simultaneously within list comprehension, one may use zip()
as:

List slicing (selecting parts oflists)

Using the third "step" argument
 Page 59 of 343

Selecting a sublist from a list

Reversing a list with slicing

Shifting a list using slicing

 Page 60 of 343

unique elements of a list

Let's say you've got a list of restaurants -- maybe you read it from a file. You care about
the unique restaurants in the list. The best way to get the unique elements from a list is to
turn it into a set:

Note that the set is not in the same order as the original list; that is because sets are unordered,
just like dicts.

This can easily be transformed back into a List with Python's built in list function,
giving another list that is the same list as the original but without duplicates:

It's also common to see this as one line:

Now any operations that could be performed on the original list can be done again.

Tuple
A tuple is similar to a list except that it is fixed-length and immutable. So, the values in
the tuple cannot be changed nor the values be added to or removed from the tuple. Tuples
are commonly used for small collections of values that will not need to change, such as an
IP address and port. Tuples are represented with parentheses instead of square brackets:
 Page 61 of 343

The same indexing rules for lists also apply to tuples. Tuples can also be nested and the values
can be any valid Python valid.
A tuple with only one member must be defined (note the comma) this way:

or

or just using tuple syntax

A tuple is an immutable list of values. Tuples are one of Python's simplest and most
common collection types, and can be created with the comma operator (value = 1, 2, 3).

Tuple
Syntactically, a tuple is a comma-separated list of values:

Although not necessary, it is common to enclose tuples in parentheses:

Create an empty tuple with parentheses:

To create a tuple with a single element, you have to include a final comma:

Note that a single value in parentheses is not a tuple:

 Page 62 of 343

To create a singleton tuple, it is necessary to have a trailing comma.

Note that for singleton tuples it's recommended (see PEP8 on trailing commas) to use
parentheses. Also, no space after the trailing comma (see PEP8 on whitespaces)

Another way to create a tuple is the built-in function tuple.

These examples are based on material from the book Think Python by Allen B. Downey.
Tuples are immutable

One of the main differences between lists and tuples in Python is that tuples are
immutable, that is, one cannot add or modify items once the tuple is initialized. For
example:

Similarly, tuples don't have .append and .extend methods as list does. Using += is
possible, but it changes the binding of the variable, and not the tuple itself:

Be careful when placing mutable objects, such as lists, inside tuples. This may lead to
very confusing outcomes when changing them. For example:
 Page 63 of 343

Will both raise an error and change the contents of the list within the tuple:

You can use the += operator to "append" to a tuple - this works by creating a new tuple with
the new element you "appended" and assign it to its current variable; the old tuple is not
changed, but replaced!

This avoids converting to and from a list, but this is slow and is a bad practice, especially
if you're going to append multiple times.

Packing and Unpacking Tuples

Tuples in Python are values separated by commas. Enclosing parentheses for inputting
tuples are optional, so the two assignments

and

are equivalent. The assignment a = 1, 2, 3 is also called packing because it packs values
together in a tuple.

Note that a one-value tuple is also a tuple. To tell Python that a variable is a tuple and not a
single value you can use a trailing comma

A comma is needed also if you use parentheses

To unpack values from a tuple and do multiple assignments use

# x == 1
# y == 2
# z == 3
 Page 64 of 343

# x == 2
# y == 3

The symbol _ can be used as a disposable variable name if one only needs some elements of a
tuple, acting as a placeholder:
Single element tuples:

In Python 3 a target variable with a * prefix can be used as a catch-all variable (see

Unpacking Iterables ): Python 3.x Version ≥ 3.0

Built-in Tuple Functions

Tuples support the following buila-in functions

Comparison

If elements are of the same type, python performs the comparison and returns the result. If
elements are different types, it checks whether they are numbers.

If numbers, perform comparison.

If either element is a number, then the other element is
returned. Otherwise, types are sorted alphabetically.

If we reached the end of one of the lists, the longer list is "larger." If both lists are same, it
returns 0.
 Page 65 of 343

Tuple Length

The function len returns the total length of the tuple

Max of a tuple

The function max returns item from the tuple with the max value

Min of a tuple

The function min returns the item from the tuple with the min value

Convert a list into tuple

The built-in function tuple converts a list into a tuple.

Tuple concatenation

Use + to concatenate two tuples

 Page 66 of 343

Indexing Tuples

Indexing with negative numbers will start from the last element as -1:

x[-1] # 3
Indexing a range of elements
x[-2] # 2
x[-3] # 1
x[-4] # IndexError: tuple index out of
range

Reversing Elements
Reverse elements within a tuple

Or using reversed (reversed gives an iterable which is converted to a tuple):

rev = tuple(reversed(colors))
# rev: ("blue", "green", "red")
colors = rev
# colors: ("blue", "green", "red")

Comprehensions involving tuples

The for clause of a list comprehension can specify more than one variable:
 Page 67 of 343

This is just like regular for loops:

# 3
# 7
# 11

Note however, if the expression that begins the comprehension is a tuple then it must be
parenthesized:

set
A set is a collection of elements with no repeats and without insertion order but sorted
order. They are used in situations where it is only important that some things are
grouped together, and not what order they were included. For large groups of data, it is
much faster to check whether or not an element is in a set than it is to do the same for a
list.

Defining a set is very similar to defining a dictionary:

Or you can build a set using an existing list:

Check membership of the set using in:

You can iterate over a set exactly like a list, but remember: the values will be in an
arbitrary, implementation- defined order.
 Page 68 of 343

Set
Operations on sets

with single elements

# Add and Remove

 Page 69 of 343

Set operations return new sets, but have the corresponding in-place versions:

method in-place operation in-place method

union s |= t update
intersection s &= t intersection_update
difference s -= t difference_update
symmetric_difference s ^= t symmetric_difference_update

For example:

==

Set of Sets

leads to:

Instead, use frozenset:

Set Operations using Methods and Builtins

We define two sets a and b

NOTE: {1} creates a set of one element, but {} creates an empty dict. The
correct way to create an empty set is set().
Intersection

[Link](b) returns a new set with elements present in both a and b

Union
[Link](b) returns a new set with elements present in either a and b
 Page 70 of 343

Difference

[Link](b) returns a new set with elements present in a but not in b

Symmetric Difference

a.symmetric_difference(b) returns a new set with elements present in either a or b but not
in both

NOTE: a.symmetric_difference(b) == b.symmetric_difference(a)

Subset and superset

[Link](a) tests whether each element of c
is in a. [Link](c) tests whether each
element of c is in a.

The latter operations have equivalent operators as shown below:

Method Operator
[Link](b) a & b
[Link](b) a|b
[Link](b)
a-b a - b
a.symmetric_difference(
b) a ^ b [Link](b)
a <= b
[Link](b) a >= b
 Page 71 of 343

Disjoint sets
Sets a and d are disjoint if no element in a is also in d and vice versa.

Testing membership

The builtin in keyword searches for occurances

Length

The builtin len() function returns the number of elements in the set

Set Comprehensions
Set comprehension is similar to list and dictionary comprehension, but it produces a set,
which is an unordered collection of unique elements.

Python 2.x Version ≥ 2.7

 Page 72 of 343

Immutable datatypes (int,float, str, tuple and frozenset)

Individual characters of strings are notassignable

Immutable variable value cannot be changed once they are created.

Tuple's individual members aren't assignable

Second line would return an error since tuple members once created aren't assignable.
Because of tuple's immutability.

Frozenset's are immutable and not assignable

Second line would return an error since frozenset members once created aren't assignable.
Third line would return error as frozensets do not support functions that can manipulate
members.

Dictionaries
A dictionary in Python is a collection of key-value pairs. The dictionary is surrounded by
curly braces. Each pair is separated by a comma and the key and value are separated by a colon.
(In other words)

A dictionary is an example of a key value store also known as Mapping in Python. It

allows you to store and retrieve elements by referencing a key. As dictionaries are
referenced by key, they have very fast lookups. As they are primarily used for referencing
items by key, they are not sorted.
 Page 73 of 343

Here is an example:

To get a value, refer to it by its key:

You can also get all of the keys in a dictionary and then iterate over them:

Dictionaries strongly resemble JSON syntax. The native json module in the Python
standard library can be used to convert between JSON and dictionaries.

Dictionary
Parameter Details
key the desired key to
lookup value The value to
set or return

Introduction to Dictionary
creating a dict
Dictionaries can be initiated in many ways:

literal syntax
Python 3.x Version ≥ 3.5
 Page 74 of 343

dict comprehension
see also: Comprehensions

built-in class: dict()

modifying a dict

To add items to a dictionary, simply create a new key with a value:

It also possible to add list and dictionary as value:

To delete an item, delete the key from the dictionary:

Iterating over dictionaries

Considering the following dictionary:

To iterate through its keys, you can use:

Output:

This is equivalent to:

 Page 75 of 343

or in Python 2:

To iterate through its values, use:

Output:

To iterate through its keys and values, use:

Output:

Note that in Python 2, .keys(), .values() and .items() return a list object. If you
simply need to iterate through the result, you can use the equivalent .iterkeys(),
.itervalues() and .iteritems().

The difference between .keys() and .iterkeys(), .values() and .itervalues(),

.items() and .iteritems() is that the iter* methods are generators. Thus, the elements
within the dictionary are yielded one by one as they are evaluated. When a list object is
returned, all of the elements are packed into a list and then returned for further
evaluation.

Note also that in Python 3, Order of items printed in the above manner does not follow
any order.
Avoiding KeyError Exceptions
One common pitfall when using dictionaries is to access a non-existent key. This typically
results in a KeyError
exception
 Page 76 of 343

One way to avoid key errors is to use the [Link] method, which allows you to specify a
default value to return in the case of an absent key.

Which returns mydict[key] if it exists, but otherwise returns default_value. Note that
this doesn't add key to mydict. So if you want to retain that key value pair, you should
use [Link](key, default_value), which does store the key value pair.

An alternative way to deal with the problem is catching the exception

You could also check if the key is in the dictionary.

Do note, however, that in multi-threaded environments it is possible for the key to be

removed from the dictionary after you check, creating a race condition where the
exception can still be thrown.

Another option is to use a subclass of dict, collections. defaultdict, that has a

default_factory to create new entries in the dict when given a new_key.
 Page 77 of 343

Iterating Over a Dictionary

If you use a dictionary as an iterator (e.g. in a for statement), it traverses the keys of the
dictionary. For example:

# b 2

The same is true when used in a comprehension

Python 3.x Version ≥ 3.0

The items() method can be used to loop over both the key and value simultaneously:

# b 2

While the values() method can be used to iterate over only the values, as would be expected:

# 3
# 2
# 1

Python 2.x Version ≥ 2.2

Here, the methods keys(), values() and items() return lists, and there are the three extra
methods iterkeys() itervalues() and iteritems() to return iterators.

Merging dictionaries
Consider the following dictionaries:

Python 3.5+
 Page 78 of 343

As this example demonstrates, duplicate keys map to their lattermost value (for example
"Clifford" overrides "Nemo").

Accessing keys and values

When working with dictionaries, it's often necessary to access all the keys and values in the
dictionary, either in a for loop, a list comprehension, or just as a plain list.

Given a dictionary like:

You can get a list of keys using the keys() method:

If instead you want a list of values, use the values() method:

If you want to work with both the key and its corresponding value, you can use the items()
method:

NOTE: Because a dict is unsorted, keys(), values(), and items() have no sort order. Use
sort(), sorted(), or an
OrderedDict if you care about the order that these methods return.

Python 2/3 Difference: In Python 3, these methods return special iterable objects, not
lists, and are the equivalent of the Python 2 iterkeys(), itervalues(), and
iteritems() methods. These objects can be used like lists for the most part, though there
 Page 79 of 343

are some differences. See PEP 3106 for more details.

Accessing values of a dictionary

The above code will print 1234.

The string "Hello" in this example is called a key. It is used to lookup a value in the dict
by placing the key in square brackets.

The number 1234 is seen after the respective colon in the dict definition. This is called
the value that "Hello" maps to in this dict.

Looking up a value like this with a key that does not exist will raise a KeyError
exception, halting execution if uncaught. If we want to access a value without risking a
KeyError, we can use the [Link] method. By default if the key does not exist,
the method will return None. We can pass it a second value to return instead of None in
the event of a failed lookup.

In this example w will get the value None and x will get the value "nuh-uh".

Creating a dictionary
Rules for creating a dictionary:
Every key must be unique (otherwise it will be overridden)
Every key must be hashable (can use the hash function to hash it; otherwise
TypeError will be thrown) There is no particular order for the keys.

# Creating and populating it with values

stock = {'eggs': 5, 'milk': 2}

# Or creating an empty dictionary

dictionary = {}

# And populating it after

dictionary['eggs'] = 5
dictionary['milk'] = 2
 Page 80 of 343

# Values can also be lists

mydict = {'a': [1, 2, 3], 'b': ['one', 'two', 'three']}

# Use [Link]() method to add new elements to the values list

mydict['a'].append(4) # => {'a': [1, 2, 3, 4], 'b': ['one', 'two',
'three']}
mydict['b'].append('four') # => {'a': [1, 2, 3, 4], 'b': ['one',
'two', 'three', 'four']}

# We can also create a dictionary using a list of two-items tuples

iterable = [('eggs', 5),
('milk', 2)] dictionary =
dict(iterables)

# Or using keyword argument:

dictionary = dict(eggs=5, milk=2)

# Another way will be to use the [Link]:

dictionary = [Link]((milk, eggs)) # => {'milk': None, 'eggs':
None}
dictionary = [Link]((milk, eggs), (2, 5)) # => {'milk': 2,
'eggs': 5}

Unpacking dictionaries using the ** operator

You can use the ** keyword argument unpacking operator to deliver the key-value pairs
in a dictionary into a function's arguments. A simplified example from the official
documentation:

As of Python 3.5 you can also use this syntax to merge an arbitrary number of dict objects.
 Page 81 of 343

As this example demonstrates, duplicate keys map to their lattermost value (for example
"Clifford" overrides "Nemo").

The trailing comma

Like lists and tuples, you can include a trailing comma in your dictionary.

PEP 8 dictates that you should leave a space between the trailing comma and the closing brace.

The dict() constructor

The dict() constructor can be used to create dictionaries from keyword arguments, or
from a single iterable of key-value pairs, or from a single dictionary and keyword
arguments.

Dictionaries Example
Dictionaries map keys to values.

Dictionary values can be accessed by their keys.

Dictionaries can also be created in a JSON style:

 Page 82 of 343

Dictionary values can be iterated over:

All combinations of dictionary values

Dictionary Comprehensions
A dictionary comprehension is similar to a list comprehension except that it produces a
dictionary object instead of a list.

A basic example:

Python 2.x Version ≥ 2.7

which is just another way of writing:

As with a list comprehension, we can use a conditional statement inside the dict
comprehension to produce only the dict elements meeting some criterion.

Python 2.x Version ≥ 2.7

Or, rewritten using a generator expression.

 Page 83 of 343

Starting with a dictionary and using dictionary comprehension as a key-value pair filter

Python 2.x Version ≥ 2.7

Switching key and value of dictionary (invert dictionary)

If you have a dict containing simple hashable values (duplicate values may have unexpected
results):

and you wanted to swap the keys and values you can take several approaches depending on
your coding style:

swapped = {v: k for k, v in my_dict.items()}

swapped = dict((v, k) for k, v in
my_dict.iteritems()) swapped =
dict(zip(my_dict.values(), my_dict))
swapped = dict(zip(my_dict.values(),

my_dict.keys())) swapped =
dict(map(reversed, my_dict.items()))
Python 2.x Version ≥ 2.3

If your dictionary is large, consider importing itertools and utilize izip or imap.

Merging Dictionaries
Combine dictionaries and optionally override old values with a nested dictionary
comprehension.
 Page 84 of 343

However, dictionary unpacking (PEP 448) may

be a preferred. Python 3.x Version ≥ 3.5

Note: dictionary comprehensions were added in Python 3.0 and backported to 2.7+, unlike list
comprehensions, which were added in 2.0. Versions < 2.7 can use generator expressions and
the dict() builtin to simulate the behavior of dictionary comprehensions.
Copying data
Copy a dictionary
A dictionary object has the method copy. It performs a shallow copy of the dictionary.

Performing a shallow copy

A shallow copy is a copy of a collection without performing a copy of its elements.

Performing a deep copy

If you have nested lists, it is desirable to clone the nested lists as well. This action is called deep
copy.
 Page 85 of 343

Performing a shallow copy of a list

You can create shallow copies of lists using slices.

Copy a set
Sets also have a copymethod. You can use this method to perform a shallow copy.

Strings
A string is a fundamental data type in most programming languages, including Python. In simple terms, a
string is a sequence of characters. In Python, strings are used to represent text and are enclosed in either
single quotes (' '), double quotes (" "), or triple quotes (''' ''' or """ """).
Here are some key characteristics of strings in Python:
Sequence of Characters: A string is a sequence of individual characters, which can include letters, digits,
symbols, spaces, and even special characters.

Immutable: Strings in Python are immutable, which means you cannot change the characters of a string
once it's created. Instead, you create a new string with the desired modifications.
 Page 86 of 343

String Methods
Changing the capitalization of a string
Python's string type provides many functions that act on the capitalization of a string. These
include:

[Link]
old
[Link]
[Link]
[Link]
lize
[Link]
[Link]
se

With unicode strings (the default in Python 3), these operations are not 1:1 mapping or
reversible. Most of these operations are intended for display purposes, rather than
normalization.

Python 3.x Version ≥ 3.3

[Link]()

[Link] creates a lowercase string that is suitable for case insensitive comparisons.
This is more aggressive than [Link] and may modify strings that are already in
lowercase or cause strings to grow in length, and is not intended for display purposes.

The transformations that take place under casefolding are defined by the Unicode
Consortium in the [Link] file on their website.

[Link]()

[Link] takes every character in a string and converts it to its uppercase equivalent, for
example:
 Page 87 of 343

[Link]()

[Link] does the opposite; it takes every character in a string and converts it to its
lowercase equivalent:

[Link]()

[Link] returns a capitalized version of the string, that is, it makes the first
character have upper case and the rest lower:

[Link]()

[Link] returns the title cased version of the string, that is, every letter in the
beginning of a word is made upper case and all others are made lower case:

[Link]()

[Link] returns a new string object in which all lower case characters are swapped
to upper case and all upper case characters to lower:

Usage as str class methods

It is worth noting that these methods may be called either on string objects (as shown
above) or as a class method of the str class (with an explicit call to [Link], etc.)
 Page 88 of 343

This is most useful when applying one of these methods to many strings at once in say, a map
function.

[Link] and f-strings: Format values into astring

Python provides string interpolation and formatting functionality through the
[Link] function, introduced in version 2.6 and f-strings introduced in version 3.6.

Given the following variables:

The following statements are all equivalent

For reference, Python also supports C-style qualifiers for string formatting. The
examples below are equivalent to those above, but the [Link] versions are preferred
due to benefits in flexibility, consistency of notation, and extensibility:

The braces uses for interpolation in [Link] can also be numbered to reduce
duplication when formatting strings. For example, the following are equivalent:
 Page 89 of 343

While the official python documentation is, as usual, thorough enough, [Link] has
a great set of examples with detailed explanations.

Additionally, the { and } characters can be escaped by using double brackets:

See String Formatting for additional information. [Link]() was proposed in PEP 3101 and
f-strings in PEP 498.
Stripping unwanted leading/trailing charactersfrom a string

Three methods are provided that offer the ability to strip leading and trailing characters
from a string: [Link], [Link] and [Link]. All three methods have the same
signature and all three return a new string object with unwanted characters removed.
[Link]([chars])

[Link] acts on a given string and removes (strips) any leading or trailing characters
contained in the argument
chars; if chars are not supplied or is None, all white space characters are removed by default.
For example:

If chars are supplied, all characters contained in it are removed from the string, which is
returned. For example:

[Link]([chars]) and [Link]([chars])

These methods have similar semantics and arguments with [Link](), their difference
lies in the direction from which they start. [Link]() starts from the end of the
string while [Link]() splits from the start of the string.
For example, using [Link]:
 Page 90 of 343

While, using [Link]:

Reversing a string
A string can reversed using the built-in reversed() function, which takes a string and
returns an iterator in reverse order.

reversed() can be wrapped in a call to ''.join() to make a string from the iterator.

While using reversed() might be more readable to uninitiated Python users, using extended
slicing with a step of
-1 is faster and more concise. Here, try to implement it as function:

Split a string based on a delimiter into a list ofstrings

[Link](sep=None, maxsplit=-1)

[Link] takes a string and returns a list of substrings of the original string. The
behavior differs depending on whether the sep argument is provided or omitted.

If sep isn't provided, or is None, then the splitting takes place wherever there is
whitespace. However, leading and trailing whitespace is ignored, and multiple consecutive
whitespace characters are treated the same as a single whitespace character:
 Page 91 of 343

The sep parameter can be used to define a delimiter string. The original string is split
where the delimiter string occurs, and the delimiter itself is discarded. Multiple
consecutive delimiters are not treated the same as a single occurrence, but rather cause
empty strings to be created.

The default is to split on every occurrence of the delimiter, however the maxsplit parameter
limits the number of splittings that occur. The default value of -1 means no limit:

[Link](sep=None, maxsplit=-1)

[Link] ("right split") differs from [Link] ("left split") when maxsplit is
specified. The splitting starts at the end of the string rather than at the beginning:
 Page 92 of 343

Note: Python specifies the maximum number of splits performed, while most other
programming languages specify the maximum number of substrings created. This may
create confusion when porting or comparing code.

Replace all occurrences of one substring withanother substring

Python's str type also has a method for replacing occurrences of one sub-string with
another sub-string in a given string. For more demanding cases, one can use [Link].

[Link](old, new[, count]):

[Link] takes two arguments old and new containing the old sub-string which is to be
replaced by the new sub- string. The optional argument count specifies the number of
replacements to be made:

For example, in order to replace 'foo' with 'spam' in the following string, we can
call [Link] with old = 'foo' and new = 'spam':

If the given string contains multiple examples that match the old argument, all
occurrences are replaced with the value supplied in new:

unless, of course, we supply a value for count. In this case count occurrences are going to get
replaced:
 Page 93 of 343

Testing what a string is composed of

Python's str type also features a number of methods that can be used to evaluate the
contents of a string. These are [Link], [Link], [Link], [Link].
Capitalization can be tested with [Link], [Link] and [Link].

[Link]

[Link] takes no arguments and returns True if the all characters in a given string are
alphabetic, for example:

As an edge case, the empty string evaluates to False when used with "".isalpha().

[Link], [Link], [Link]

These methods test the capitalization in a given string.

[Link] is a method that returns True if all characters in a given string are uppercase and
False otherwise.

Conversely, [Link] is a method that returns True if all characters in a given string are
lowercase and False
otherwise.
 Page 94 of 343

[Link] returns True if the given string is title cased; that is, every word begins with
an uppercase character followed by lowercase characters.

[Link], [Link], [Link]

[Link] returns whether the string is a sequence of decimal digits, suitable for
representing a decimal number.

[Link] includes digits not in a form suitable for representing a decimal number, such as
superscript digits.

[Link] includes any number values, even if not digits, such as values outside the range
0-9.

12345 True True True

?2??5 True True True
?²³????? False True True
?? False False True
Five False False False

Bytestrings (bytes in Python 3, str in Python 2), only support isdigit, which only

checks for basic ASCII digits. As with [Link], the empty string evaluates to

False.

[Link]

This is a combination of [Link] and [Link], specifically it evaluates to

True if all characters in the given string are alphanumeric, that is, they consist of
alphabetic or numeric characters:
 Page 95 of 343

[Link]

Evaluates True
to if the string contains only whitespace characters.

Sometimes a string looks “empty” but we don't know whether it's because it contains just
whitespace or no character at all

To cover this case we need an additional test

But the shortest way to test if a string is empty or just contains whitespace characters is
to use strip(with no arguments it removes all leading and trailing whitespace characters)

String Contains
Python makes it extremely intuitive to check if a string contains a given substring. Just use the
in operator:
 Page 96 of 343

Note: testing an empty string will always result in True:

Join a list of strings into one string

A string can be used as a separator to join a list of strings together into a single string using the
join() method. For example you can create a string where each element in a list is separated
by a space.

The following example separates the string elements with three hyphens.

Counting number of times a substring appearsin a string

One method is available for counting the number of occurrences of a sub-string in another
string, [Link].

[Link](sub[, start[, end]])

start =
[Link] returns an int indicating the number of non-overlapping occurrences of the
sub-string sub in another string. The optional arguments start and end indicate the
beginning and the end in which the search will takeplace. By default 0 and end =
len(str) meaning the whole string will be searched:

By specifying a different value for start, end we can get a more localized search and
count, for example, if start is equal to 13 the call to:
 Page 97 of 343

is equivalent to:

Case insensitive string comparisons

Comparing string in a case insensitive way seems like something that's trivial, but it's
not. This section only considers unicode strings (the default in Python 3). Note that
Python 2 may have subtle weaknesses relative to Python 3 - the later's unicode handling
is much more complete.

The first thing to note it that case-removing conversions in unicode aren't trivial. There is text
for which
[Link]() != [Link]().lower(), such as "ß":

But let's say you wanted to caselessly compare "BUSSE" and "Buße". You probably also want
to compare "BUSSE"
□ E" equal - that's the newer capital form. The recommended
and "BU

way is to use casefold:

Do not just use lower. If casefold is not available, doing .upper().lower() helps (but only
somewhat).

Then you should consider accents. If your font renderer is good, you probably think "ê" ==
"ê" - but it doesn't:
 Page 98 of 343

This is because they are actually

The simplest way to deal with this is [Link]. You probably want to use
NFKD normalization, but feel free to check the documentation. Then one does

To finish up, here this is expressed in functions:

Justify strings
Python provides functions for justifying strings, enabling text padding to make aligning
various strings much easier. Below is an example of [Link] and [Link]:

4 - 255 m (41 km
0 > 5 i. 12 .)
1 - 63 m (10 km
9 > i. 2 .)
5 - 138 m (22 km
 Page 99 of 343

> 1 i. 22 .)
9 - 189 m (30 km
3 > i. 5 .)

ljust and rjust are very similar. Both have a width parameter and an optional
fillchar parameter. Any string created by these functions is at least as long as the width
parameter that was passed into the function. If the string is longer than width already, it
is not truncated. The fillchar argument, which defaults to the space character ' ' must
be a single character, not a multicharacter string.

The ljust function pads the end of the string it is called on with the fillchar until it is
width characters long. The rjust function pads the beginning of the string in a similar
fashion. Therefore, the l and r in the names of these functions refer to the side that the
original string, not the fillchar, is positioned in the output string.

Test the starting and ending characters of a string

In order to test the beginning and ending of a given string in Python, one can use the methods
[Link]()
and [Link]().

[Link](prefix[, start[, end]])

As its name implies, [Link] is used to test whether a given string starts with the given
characters in
prefix.

The optional arguments start and end specify the start and end points from which the
testing will start and finish. In the following example, by specifying a start value of 2 our
string will be searched from position 2 and afterwards:

This yields True since s[2] == 'i' and s[3] == 's'.

You can also use a tuple to check if it starts with any of a set of strings
 Page 100 of 343

[Link](prefix[, start[, end]])

[Link] is exactly similar to [Link] with the only difference being that it
searches for ending characters and not starting characters. For example, to test if a string
ends in a full stop, one could write:

as with startswith more than one characters can used as the ending sequence:

You can also use a tuple to check if it ends with any of a set of strings

String Formatting
When storing and transforming data for humans to see, string formatting can become very
important. Python offers a wide variety of string formatting methods which are outlined in
this topic.

Basics of String Formatting

You can use [Link] to format output. Bracket pairs are replaced with arguments in
the order in which the arguments are passed:
 Page 101 of 343

Indexes can also be specified inside the brackets. The numbers correspond to indexes of
the arguments passed to the [Link] function (0-based).

Named arguments can be also used:

Object attributes can be referenced when passed into [Link]:

Dictionary keys can be used as well:

Same applies to list and tuple indices:

Note: In addition to [Link], Python also provides the modulo operator %--also
known as the string formatting or interpolation operator (see PEP 3101)--for
formatting strings. [Link] is a successor of %
and it offers greater flexibility, for instance by making it easier to carry out multiple
substitutions.

In addition to argument indexes, you can also include a format specification inside the curly
brackets. This is an expression that follows special rules and must be preceded by a colon
(:). See the docs for a full description of format specification. An example of format
 Page 102 of 343

specification is the alignment directive :~^20 (^ stands for center alignment, total width
20, fill with ~ character):

format allows behavior not possible with %, for example repetition of arguments:

As format is a function, it can be used as an argument in other functions:

Format literals (f-string)

Literal format strings were introduced in PEP 498 (Python3.6 and upwards), allowing you
to prepend f to the beginning of a string literal to effectively apply .format to it with all
variables in the current scope.

This works with more advanced format strings too, including alignment and dot notation.

Note: The f'' does not denote a particular type like b'' for bytes or u'' for unicode in
python2. The formatting is immediately applied, resulting in a normal string.
 Page 103 of 343

The format strings can also be nested:

The expressions in an f-string are evaluated in left-to-right order. This is detectable only if the
expressions have side effects:

Float formatting

Same hold for other way of referencing:

Floating point numbers can also be formatted in scientific notation or as percentages:

 Page 104 of 343

You can also combine the {0} and {name} notations. This is especially useful when you
want to round all variables to a pre-specified number of decimals with 1 declaration:

Functions
Parameter Details
arg1, ..., argN Regular arguments
*args Unnamed positional arguments
kw1, ..., kwN Keyword-only arguments
**kwargs The rest of keyword arguments

Functions in Python provide organized, reusable and modular code to perform a set of
specific actions. Functions simplify the coding process, prevent redundant logic, and
make the code easier to follow. This topic describes the declaration and utilization of
functions in Python.

Python has many built-in functions like print(), input(), len(). Besides built-ins you
can also create your own functions to do more specific jobs—these are called user-
defined functions.

Defining and calling simple functions

Using the def statement is the most common way to define a function in python. This
statement is a so called single clause compound statement with the following syntax:

function_name is known as the identifier of the function. Since a function definition is an

executable statement, its execution binds the function name to the function object which can be
called later on using the identifier.
 Page 105 of 343

parameters are an optional list of identifiers that get bound to the values supplied as
arguments when the function is called. A function may have an arbitrary number of arguments
which are separated by commas.

statement(s) – also known as the function body – are a nonempty sequence of statements
executed each time the function is called. This means a function body cannot be empty, just like
any indented block.
Here’s an example of a simple function definition which purpose is to print Hello each time
it’s called:

Now let’s call the defined greet() function:

That’s another example of a function definition which takes one single argument and
displays the passed in value each time the function is called:

After that the greet_two() function must be called with an argument:

Also, you can give a default value to that function argument:

Now you can call the function without giving a value:

You'll notice that unlike many other languages, you do not need to explicitly declare a
return type of the function. Python functions can return values of any type via the return
keyword. One function can return any number of different types!
 Page 106 of 343

As long as this is handled correctly by the caller, this is perfectly valid Python code.

A function that reaches the end of execution without a return statement will always return
None:

As mentioned previously a function definition must have a function body, a nonempty

sequence of statements. Therefore, the pass statement is used as function body, which is a
null operation – when it is executed, nothing happens. It does what it means, it skips. It is
useful as a placeholder when a statement is required syntactically, but no code needs to be
executed.

Using loops within functions

In Python function will be returned as soon as execution hits "return" statement.
Return statement inside loop in a function
In this example, function will return as soon as value var has 1
 Page 107 of 343

output

Got value 5
Still looping
Got value 3
Still looping
Got value 1
>>>> Got 1

Argument passing and mutability

First, some terminology:

argument (actual parameter): the actual variable being passed to a function;

parameter (formal parameter): the receiving variable that is used in a function.

In Python, arguments are passed by assignment (as opposed to other languages, where
arguments can be passed by value/reference/pointer).

Mutating a parameter will mutate the argument (if the argument's type is mutable).

Reassigning the parameter won’t reassign the argument.

In Python, we don’t really assign values to variables, instead we bind (i.e. assign,
attach) variables (considered as names) to objects.

Immutable: Integers, strings, tuples, and so on. All operations make copies.
 Page 108 of 343

Mutable: Lists, dictionaries, sets, and so on. Operations may or may not mutate.

Defining a function with optional arguments

Optional arguments can be defined by assigning (using =) a default value to the argument-
name:

Calling this function is possible in 3 different ways:

Warning
Mutable types (list, dict, set, etc.) should be treated with care when given as
default attribute. Any mutation of the default argument will change it
permanently. See Defining a function with optional mutable arguments.

Defining a function with optional mutablearguments

There is a problem when using optional arguments with a mutable default type
(described in Defining a function with optional arguments), which can potentially lead to
unexpected behavior.
 Page 109 of 343

Explanation

This problem arises because a function's default arguments are initialized once, at the
point when the function is defined, and not (like many other languages) when the
function is called. The default values are stored inside the function object's defaults
member variable.

For immutable types (see Argument passing and mutability) this is not a problem because
there is no way to mutate the variable; it can only ever be reassigned, leaving the original
value unchanged. Hence, subsequent are guaranteed to have the same default value.
However, for a mutable type, the original value can mutate, by making calls to its various
member functions. Therefore, successive calls to the function are not guaranteed to have
the initial default value.

Note: Some IDLEs like PyCharm will issue a warning when a mutable type is
specified as a default attribute.

Solution

If you want to ensure that the default argument is always the one you specify in the
function definition, then the solution is to always use an immutable type as your default
argument.

A common idiom to achieve this when a mutable type is needed as the default, is to use
 Page 110 of 343

None (immutable) as the default argument and then assign the actual default value to the
argument variable if it is equal to None.

*args and **kwargs

Using **kwargs when writing functions
You can define a function that takes an arbitrary number of keyword (named) arguments by
using the double star
** before a parameter name:

When calling the method, Python will construct a dictionary of all keyword arguments and
make it available in the function body:

Note that the **kwargs parameter in the function definition must always be the last
parameter, and it will only match the arguments that were passed in after the previous
ones.

Inside the function body, kwargs is manipulated in the same way as a dictionary; in order
to access individual elements in kwargs you just loop through them as you would with a
normal dictionary:
 Page 111 of 343

Now, calling print_kwargs(a="two", b=1) shows the following output:

Using *args when writing functions

You can use the star * when writing a function to collect all positional (ie. unnamed)
arguments in a tuple:

Calling method:

In that call, for arg will be assigned as always, and the two others will be fed into the args
tuple, in the order they were received.

Populating kwarg values with a dictionary

Keyword-only and Keyword-requiredarguments

Python 3 allows you to define function arguments which can only be assigned by keyword,
even without default values. This is done by using star * to consume additional positional
parameters without setting the keyword parameters. All arguments after the * are
keyword-only (i.e., non-positional) arguments. Note that if keyword-only arguments
aren't given a default, they are still required when calling the function.
 Page 112 of 343

Using **kwargs when calling functions

You can use a dictionary to assign values to the function's parameters; using parameters
name as keys in the dictionary and the value of these arguments bound to each key:

**kwargs and default values

To use default values with **kwargs

Using *args when calling functions

The effect of using the * operator on an argument when calling a function is that of
unpacking the list or a tuple argument
 Page 113 of 343

# 12

# 34

Note that the length of the starred argument need to be equal to the number of the function's
arguments.

A common python idiom is to use the unpacking operator * with the zip function to reverse
its effects:

Defining a function with an arbitrary number of arguments

Arbitrary number of positional arguments:

Defining a function capable of taking an arbitrary number of arguments can be done by

prefixing one of the arguments with a *

# 2
# 3

# 2
# 3
 Page 114 of 343

You can't provide a default for args, for example func(*args=[1, 2, 3]) will raise a
syntax error (won't even compile).

You can't provide these by name when calling the function, for example func(*args=[1,
2, 3]) will raise a TypeError.

But if you already have your arguments in an array (or any other Iterable), you can invoke
your function like this:
func(*my_stuff).

These arguments (*args) can be accessed by index, for example args[0] will return the first
argument
Arbitrary number of keyword arguments
You can take an arbitrary number of arguments with a name by defining an argument in the
definition with two *
in front of it:

You can't provide these without names, for example func(1, 2, 3) will raise a
TypeError.

kwargs is a plain native python dictionary. For example, args['value1'] will give the
value for argument value1. Be sure to check beforehand that there is such an argument
or a KeyError will be raised.

Warning
You can mix these with other optional and required arguments but the order inside
the definition matters. The positional/keyword arguments come first. (Required
arguments).
 Page 115 of 343

Then comes the arbitrary *arg arguments. (Optional).

Then keyword-only arguments come next.
(Required). Finally the arbitrary keyword
**kwargs come. (Optional).

arg1 must be given, otherwise a TypeError is raised. It can be given as positional

(func(10)) or keyword argument (func(arg1=10)).
kwarg1 must also be given, but it can only be provided as keyword-argument:
func(kwarg1=10).
arg2 and kwarg2 are optional. If the value is to be changed the same rules as for arg1
(either positional or keyword) and kwarg1 (only keyword) apply.
*args catches additional positional parameters. But note, that arg1 and arg2 must be
provided as positional arguments to pass arguments to *args: func(1, 1, 1, 1).
**kwargs catches all additional keyword parameters. In this case any parameter
that is not arg1, arg2, kwarg1 or kwarg2. For example: func(kwarg3=10).
In Python 3, you can use * alone to indicate that all subsequent arguments must be
specified as keywords. For instance the [Link] function in Python 3.5 and
higher is defined using def [Link] (a, b,*, rel_tol=1e-09,
abs_tol=0.0), which means the first two arguments can be supplied positionally
but the optional third and fourth parameters can only be supplied as keyword
arguments.

Note on Naming

The convention of naming optional positional arguments args and optional keyword
arguments kwargs is just a convention you can use any names you like but it is useful to
follow the convention so that others know what you are doing, or even yourself later so
please do.

Note on Uniqueness

Any function can be defined with none or one *args and none or one **kwargs but
not with more than one of each. Also *args must be the last positional argument and
**kwargs must be the last parameter. Attempting to use more than one of either will
result in a Syntax Error exception.
 Page 116 of 343

Note on Nesting Functions with Optional Arguments

It is possible to nest such functions and the usual convention is to remove the items that
the code has already handled but if you are passing down the parameters you need to pass
optional positional args with a * prefix and optional keyword args with a ** prefix,
otherwise args with be passed as a list or tuple and kwargs as a single dictionary. e.g.:

def f1(**kwargs):
print(len(kwargs))

fn(a=1, b=2)
# Out:
# {'a': 1, 'b': 2}
#2

Iterable and dictionary unpacking

Functions allow you to specify these types of parameters: positional, named, variable
positional, Keyword args (kwargs). Here is a clear and concise use of each type.

print(a, b, c, d, args, kwargs)

>>> 1 2 unpacking(1, 45 60 () 2)
>>> {} 2, 3, 4)
unpacking(1,
1 2 3 4 () {}
>>> unpacking(1, 2, c=3, d=4)
1 2 3 4 () {}
>>> unpacking(1, 2, d=4, c=3)
12 3 4 () {}

>>> pair = (3,) unpacking(1,

>>> 2, *pair, d=4)
1 2 3 4 () {}
>>> unpacking(1, 2, d=4, *pair)
1 2 3 4 () {}
>>> unpacking(1, 2, *pair, c=3)
 Page 117 of 343

Traceback (most recent

call last): File
"<stdin>", line 1, in
<module>
TypeError: unpacking() got multiple values for argument 'c'
>>> unpacking(1, 2, c=3,
*pair) Traceback (most
recent call last):
File "<stdin>", line 1, in <module>
TypeError: unpacking() got multiple values for argument 'c'

>>> args_list = [3]

>>> unpacking(1, 2,
*args_list, d=4) 1 2 3 4
() {}
>>> unpacking(1, 2, d=4,
*args_list) 1 2 3 4 () {}
>>> unpacking(1, 2, c=3,
*args_list) Traceback (most
recent call last):
File "<stdin>", line 1, in <module>
TypeError: unpacking() got multiple values for argument 'c'
>>> unpacking(1, 2,
*args_list, c=3) Traceback
(most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unpacking() got multiple values for argument 'c'

>>> pair = (3, 4)

>>> unpacking(1,
2, *pair) 1 2 3 4
() {}
>>> unpacking(1, 2, 3, 4, *pair)
1 2 3 4 (3, 4) {}
>>> unpacking(1, 2, d=4,
*pair) Traceback (most
recent call last):
File "<stdin>", line 1, in <module>
TypeError: unpacking() got multiple values for argument 'd'
>>> unpacking(1, 2,
*pair, d=4) Traceback
(most recent call last):
 Page 118 of 343

File "<stdin>", line 1, in <module>

TypeError: unpacking() got multiple values for argument 'd'

>>> args_list = [3, 4]

>>> unpacking(1, 2,
*args_list) 1 2 3 4
() {}
>>> unpacking(1, 2, 3, 4, *args_list)
1 2 3 4 (3, 4) {}
>>> unpacking(1, 2, d=4,
*args_list) Traceback (most
recent call last):
File "<stdin>", line 1, in <module>
TypeError: unpacking() got multiple values for argument 'd'
>>> unpacking(1, 2,
*args_list, d=4) Traceback
(most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unpacking() got multiple values for argument 'd'

>>> arg_dict = {'c':3, 'd':4}

>>> unpacking(1, 2,
**arg_dict) 1 2 3 4
() {}
>>> arg_dict = {'d':4, 'c':3}
>>> unpacking(1, 2,
**arg_dict) 1 2 3 4
() {}
>>> arg_dict = {'c':3, 'd':4, 'not_a_parameter': 75}
>>> unpacking(1, 2, **arg_dict)
1 2 3 4 () {'not_a_parameter': 75}

>>> unpacking(1, 2, *pair,

**arg_dict) Traceback (most
recent call last):
File "<stdin>", line 1, in <module>
TypeError: unpacking() got multiple values for argument 'd'
>>> unpacking(1, 2, 3, 4,
**arg_dict) Traceback (most
recent call last):
 Page 119 of 343

File "<stdin>", line 1, in <module>

TypeError: unpacking() got multiple values for argument 'd'

# Positional arguments take priority over any other form of argument

passing
>>> unpacking(1, 2, **arg_dict, c=3)
1 2 3 4 () {'not_a_parameter': 75}
>>> unpacking(1, 2, 3,
**arg_dict, c=3) Traceback
(most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unpacking() got multiple values for argument 'c'

Returning values from functions

Functions can return a value that you can use directly:

or save the value for later use:

or use the value for any operations:

If return is encountered in the function the function will be exited immediately and
subsequent operations will not be evaluated:

You can also return multiple values (in the form of a tuple):
 Page 120 of 343

A function with no return statement implicitly returns None. Similarly, a function with
a return statement, but no return value or variable returns None.

Forcing the use of named parameters

All parameters specified after the first asterisk in the function signature are keyword-only.

In Python 3 it's possible to put a single asterisk in the function signature to ensure that the
remaining arguments may only be passed using keyword arguments.

Generators

Introduction

Generator functions are similar to regular functions, except that they have one or more
yield statements in their body. Such functions cannot return any values (however
empty returns are allowed if you want to stop the generator early).
 Page 121 of 343

This generator function is equivalent to the previous generator expression, it outputs the same.

Note: all generator expressions have their own equivalent functions, but not vice versa.

A generator expression can be used without parentheses if both parentheses would be repeated
otherwise:

Instead of:

But not:

Calling a generator function produces a generator object, which can later be iterated over.
Unlike other types of iterators, generator objects may only be traversed once.

Notice that a generator's body is not immediately executed: when you call function() in
the example above, it immediately returns a generator object, without executing even the first
print statement. This allows generators to consume less memory than functions that return a
list, and it allows creating generators that produce infinitely long sequences.
For this reason, generators are often used in data science, and other contexts involving
large amounts of data. Another advantage is that other code can immediately use the
values yielded by a generator, without waiting for the complete sequence to be produced.

However, if you need to use the values produced by a generator more than once, and if
generating them costs more than storing, it may be better to store the yielded values as
a list than to re-generate the sequence. See 'Resetting a generator' below for more
details.
 Page 122 of 343

What can be iterable

Iterable can be anything for which items are received one by one, forward only. Built-in
Python collections are iterable:

Generators return iterables:

data
Iterator isn't reentrant!

Typically a generator object is used in a loop, or in any function that requires an iterable:

# Outp
ut:
# Receiv 0
ed
# Receiv 1
ed
# Receiv 4
ed
# Receiv 9
ed
# Receiv 1
ed 6
# Receiv 2
ed 5
 Page 123 of 343

# Receiv 6
ed 4
# Receiv 8
ed 1

Since generator objects are iterators, one can iterate over them manually using the next()
function. Doing so will return the yielded values one by one on each subsequent
invocation.

Under the hood, each time you call next() on a generator, Python executes statements
in the body of the generator function until it hits the next yield statement. At this point
it returns the argument of the yield command, and remembers the point where that
happened. Calling next() once again will resume execution from that point and continue
until the next yield statement.

If Python reaches the end of the generator function without encountering any more yields,
a StopIteration exception is raised (this is normal, all iterators behave in the same way).

# a becomes 0
# b becomes
1
# c becomes

Note that in Python 2 generator objects had .next() methods that could be used to
iterate through the yielded values manually. In Python 3 this method was replaced
with the . next () standard for all
iterators.

Resetting a generator

Remember that you can only iterate through the objects generated by a generator once. If you
have already iterated through the objects in a script, any further attempt does so will yield
None.
If you need to use the objects generated by a generator more than once, you can either define
the generator
function again and use it a second time, or, alternatively, you can store the output of the
generator function in a list on first use. Re-defining the generator function will be a good
option if you are dealing with large volumes of data, and storing a list of all data items
would take up a lot of disc space. Conversely, if it is costly to generate the items initially,
you may prefer to store the generated items in a list so that you can re-use them.
 Page 124 of 343

Variable Scope and Binding

Global Variables
In Python, variables inside functions are considered local if and only if they appear in the
left side of an assignment statement, or some other binding occurrence; otherwise such a
binding is looked up in enclosing functions, up to the global scope. This is true even if the
assignment statement is never executed.

Normally, an assignment inside a scope will shadow any outer variables of the same name:
 Page 125 of 343

Declaring a name global means that, for the rest of the scope, any assignments to the
name will happen at the module's top level:

The global keyword means that assignments will happen at the module's top level, not at
the program's top level. Other modules will still need the usual dotted access to variables
within the module.

To summarize: in order to know whether a variable x is local to a function, you should read
the entire function:
1. if you've found global x, then x is a global variable
2. If you've found nonlocal x, then x belongs to an enclosing function, and is neither
local nor global
3. If you've found x = 5 or for x in range(3) or some other binding, then x is a local
variable
4. Otherwise, x belongs to some enclosing scope (function scope, global scope, or builtins)

Local Variables
If a name is bound inside a function, it is by default accessible only within the function:

Control flow constructs have no impact on the scope (with the exception of except), but
accessing variable that was not assigned yet is an error:
 Page 126 of 343

Common binding operations are assignments, for loops, and augmented assignments such as a
+= 5

Nonlocal Variables
Python 3 added a new keyword called nonlocal. The nonlocal keyword adds a scope
override to the inner scope. One of the most common examples is to create function that
can increment:

If you try running this code, you will receive an UnboundLocalError because the num
variable is referenced before it is assigned in the innermost function. Let's add nonlocal to
the mix:

Basically nonlocal will allow you to assign to variables in an outer scope, but not a global
scope. So, you can't use nonlocal in our counter function because then it would try to
assign to a global scope. Give it a try and you will quickly get a SyntaxError. Instead,
 Page 127 of 343

you must use nonlocal in a nested function.

(Note that the functionality presented here is better implemented using generators.)

The del command

This command has several related yet distinct forms.
del v

If v is a variable, the command del v removes the variable from its scope. For example:

Note that del is a binding occurrence, which means that unless explicitly stated
otherwise (using nonlocal or global), del v will make v local to the current
scope. If you intend to delete v in an outer scope, use nonlocal v or global v in
the same scope of the del v statement.

In all the following, the intention of a command is a default behavior but is not enforced
by the language. A class might be written in a way that invalidates this intention.

del [Link]

This command triggers a call to v. delattr (name).

(del – delete attr- attribute)

The intention is to make the attribute name unavailable. For example:

del v[item]

This command triggers a call to v. delitem (item).

The intention is that item will not belong in the mapping implemented by the object v. For
example:
 Page 128 of 343

del v[a:b]

This actually calls v. delslice (a, b).

The intention is similar to the one described above, but with slices - ranges of items
instead of a single item. For example:

See also Garbage Collection #The del command.

Nested functions
Functions in python are first-class objects. They can be defined in any scope

Functions capture their enclosing scope can be passed around like any other sort of object
 Page 129 of 343

Decorators
Parameter Details
The function to be decorated (wrapped)

Decorator functions are software design patterns. They dynamically alter the functionality
of a function, method, or class without having to directly use subclasses or change the
source code of the decorated function. When used correctly, decorators can become
powerful tools in the development process. This topic covers implementation and
applications of decorator functions in Python.

Decorator function

Decorators augment the behavior of other functions or methods. Any function that takes a
function as a parameter and returns an augmented function can be used as a decorator.

The @-notation is syntactic sugar that is equivalent to the following:

It is important to bear this in mind in order to understand how the decorator’s work. This
"unsugared" syntax makes it clear why the decorator function takes a function as an
argument, and why it should return another function. It also demonstrates what would
happen if you don't return a function:
 Page 130 of 343

Thus, we usually define a new function inside the decorator and return it. This new
function would first do something that it needs to do, then call the original function,
and finally process the return value. Consider this simple decorator function that prints
the arguments that the original function receives, then calls it.

num_a

Decorator with arguments (decorator factory)

A decorator takes just one argument: the function to be decorated. There is no way to

pass other arguments. But additional arguments are often desired. The trick is then to

make a function which takes arbitrary arguments and returns a decorator.

 Page 131 of 343

Decorator functions

The decorator wants to tell you: Hello World

Important Note:

With such decorator factories you must call the decorator with a pair of parentheses:

TypeError: decorator() missing 1 required positional argument: 'func'

Decorator classes

Inside the decorator with arguments (10,)

 Page 132 of 343

Closure
Closures in Python are created by function calls. Here, the call to makeInc creates a binding for
x that is referenced inside the function inc. Each call to makeInc creates a new instance of this
function, but each instance has a link to a different binding of x.

Notice that while in a regular closure the enclosed function fully inherits all variables
from its enclosing environment, in this construct the enclosed function has only read
access to the inherited variables but cannot make assignments to them

Python 3 offers the nonlocal statement (Nonlocal Variables) for realizing a full closure

with nested functions. Python 3.x Version ≥ 3.0

nonlocal x
# now assigning a value to x is allowed
x += y
return x
 Page 133 of 343

return inc

incOne = makeInc(1)
incOne(5) # returns 6

Functional Programming inPython

Functional programming decomposes a problem into a set of functions. Ideally, functions
only take inputs and produce outputs, and don’t have any internal state that affects the
output produced for a given input. below are functional techniques common to many
languages: such as lambda, map, reduce.

Lambda Function
An anonymous, in lined function defined with lambda. The parameters of the lambda are
defined to the left of the colon. The function body is defined to the right of the colon. The
result of running the function body is (implicitly) returned.

Map Function
Map takes a function and a collection of items. It makes a new, empty collection, runs the
function on each item in the original collection and inserts each return value into the new
collection. It returns the new [Link] is a simple map that takes a list of names and
returns a list of the lengths of those names:

Reduce Function
Reduce takes a function and a collection of items. It returns a value that is created by
combining the items. This is a simple reduce. It returns the sum of all the items in the
collection.
 Page 134 of 343

Filter Function
Filter takes a function and a collection. It returns a collection of every item for which the
function returned True.

Lambda (Inline/Anonymous) Functions

The lambda keyword creates an inline function that contains a single expression. The
value of this expression is what the function returns when invoked.

Consider the function:

which, when called as:

prints:

This can be written as a lambda function as follows:

See note at the bottom of this section regarding the assignment of lambdas to
variables. Generally, don't do it.

This creates an inline function with the name greet_me that returns Hello. Note that
you don't write return when creating a function with lambda. The value after : is
automatically returned.

Once assigned to a variable, it can be used just like a regular function:

 Page 135 of 343

prints:

lambdas can take arguments, too:

returns the string:

HELLO
 Page 136 of 343

They can also take arbitrary number of arguments / keyword arguments, like normal functions.

prints:

lambdas are commonly used for short functions that are convenient to define at the point
where they are called (typically with sorted, filter and map).

For example, this line sorts a list of strings ignoring their case and ignoring whitespace at
the beginning and at the end:

Sort list just ignoring whitespaces:

Examples with map:

Examples with numerical lists:

 Page 137 of 343

One can call other functions (with/without arguments) from inside a lambda function.

prints:

This is useful because lambda may contain only one expression and by using a subsidiary
function one can run multiple statements.

NOTE

Bear in mind that PEP-8 (the official Python style guide) does not recommend assigning
lambdas to variables (as we did in the first two examples):

Always use a def statement instead of an assignment statement that binds a

lambda expression directly to an identifier.

Yes:

No:

The first form means that the name of the resulting function object is specifically f instead
of the generic <lambda>. This is more useful for tracebacks and string representations in
general. The use of the assignment statement eliminates the sole benefit a lambda
expression can offer over an explicit def statement (i.e., that it can be embedded inside a
larger expression).
 Page 138 of 343

Reduce
Parameter Details
function function that is used for reducing the iterable (must take two
arguments). (positional-only) iterable iterable that's going to be reduced.
(positional-only)
initializer start-value of the reduction. (optional, positional-only)

Overview

reduce reduces an iterable by applying a function repeatedly on the next element of an

iterable and the cumulative result so far.

In this example, we defined our own add function. However, Python comes with a
standard equivalent function in the operator module:

reduce can also be passed a starting value:

 Page 139 of 343

Using reduce

Given an initializer the function is started by applying it to the initializer and the first
iterable element:

Without initializer parameter the reduce starts by applying the function to the first two
list elements:

Map Function
Parameter Details
Function function for mapping (must take as many parameters as there are iterables)
(positional-only) iterable the function is applied to each element of the iterable
(positional-only)
*additional_iterables see iterable, but as many as you like (optional, positional-only)

Basic use of map

The map function is the simplest one among Python built-ins used for functional
programming. map() applies a specified function to each element in an iterable:
 Page 140 of 343

Mapping each value in an iterable

For example, you can take the absolute value of each element:

Anonymous function also supports for mapping a list:

or converting decimal values to percentages:

or converting dollars to euros (given an exchange rate):

[Link] is a convenient way to fix parameters of functions so that they can be

used with map instead of using lambda or creating customized functions.

Refactoring filter and map to listcomprehensions

The filter or map functions should often be replaced by list comprehensions. Guido Van
Rossum describes this well in an open letter in 2005:

filter(P, S) is almost always written clearer as [x for x in S if P(x)],

 Page 141 of 343

and this has the huge advantage that the most common usages involve predicates
that are comparisons, e.g. x==42, and defining a lambda for that just requires
much more effort for the reader (plus the lambda is slower than the list
comprehension). Even more so for map (F, S) which becomes [F(x) for x in
S]. Of course, in many cases you'd be able to use generator expressions instead.

The following lines of code are considered "not pythonic" and will raise errors in many python
linters.

Taking what we have learned from the previous quote, we can break down these filter
and map expressions into their equivalent list comprehensions; also removing the lambda
functions from each - making the code more readable in the process.

# Map

Readability becomes even more apparent when dealing with chaining functions. Where
due to readability, the results of one map or filter function should be passed as a result to
the next; with simple cases, these can be replaced with a single list comprehension.
Further, we can easily tell from the list comprehension what the outcome of our process is,
where there is more cognitive load when reasoning about the chained Map & Filter
process.

Refactoring - Quick
 Page 142 of 343

Reference Map

Filter

where F and P are functions which respectively transform input values and return a bool

Iterators
An iterator in Python is an object that is used to iterate over iterable objects like lists, tuples, dicts,
and sets. The Python iterators object is initialized using the iter() method. It uses the next() method
for iteration.
_iter(): The iter() method is called for the initialization of an iterator. This returns an iterator object
_next(): The next method returns the next value for the iterable. When we use a for loop to traverse
any iterable object, internally it uses the iter () method to get an iterator object, which further uses
the next () method to iterate over. This method raises a StopIteration to signal the end of the
iteration.
Python iter() Example

string = "GFG"
ch_iterator = iter(string)

print(next(ch_iterator))
print(next(ch_iterator))
print(next(ch_iterator))
Output:

G
F
G

Programmatically accessing docstrings

Docstrings are - unlike regular comments - stored as an attribute of the function they
document, meaning that you can access them programmatically.
 Page 143 of 343

An example function
The docstring can be accessed using the doc attribute:

This is a function that does nothing at all

Help on function func in module main :

func()

This is a function that does nothing at all

Another example function

function. doc is just the actual docstring as a string, while the help function
provides general information about a function, including the docstring. Here's a more
helpful example:

Help on function greet in module main :

greet(name, greeting='Hello')

Print a greeting to the user name

Optional parameter greeting can change what they're greeted with.
 Page 144 of 343

Advantages of docstrings over regular comments

Just putting no docstring or a regular comment in a function makes it a lot less helpful.

None

Help on function greet in module main:

greet(name, greeting='Hello')

Write documentation using docstrings

A docstring is a multi-line comment used to document modules, classes, functions and
methods. It has to be the first statement of the component it describes.

The value of the docstring can be accessed within the program and is - for example - used by
the help command.

Syntax
conventions
 Page 145 of 343

PEP 257 defines a syntax standard for docstring comments. It

basically allows two types: One-line Docstrings:

According to PEP 257, they should be used with short and simple functions. Everything is
placed in one line, e.g:

The docstring shall end with a period, the verb should be in the imperative form.

Multi-line Docstrings:

Multi-line docstring should be used for longer, more complex functions, modules or classes.

They start with a short summary (equivalent to the content of a one-line docstring) which
can be on the same line as the quotation marks or on the next line, give additional detail
and list parameters and return values.

Call stack
1. Introduction to Python's Call Stack

In the world of programming, understanding how the call stack works is crucial for any Python
developer. This article aims to demystify Python's call stack through clear explanations,
diagrams, and practical examples. By grasping the concept of the call stack, programmers can
gain deeper insights into debugging and optimizing their code. Join us as we uncover the inner
workings of Python's call stack.

2. What is the Call Stack in Python?

The call stack in Python is a data structure that keeps track of function calls during program
execution. When a function is called, it is added to the top of the call stack, and when the
function returns, it is removed from the stack. Understanding the call stack is essential for
debugging and analyzing the flow of execution in Python programs.
 Page 146 of 343

3. Importance of Understanding the Call Stack

Understanding the call stack is crucial for efficient debugging and analyzing program
execution in Python. By visualizing the call stack as a stack of function calls, programmers can
track the order of function invocations, identify errors or infinite loops, and optimize code
efficiency. Real-world examples and diagrams will be used to illustrate the significance of
comprehending the call stack in Python programming.

4. How Does the Call Stack Work?

In Python, the call stack works based on the Last-In-First-Out (LIFO) principle. Each time a
function is called, a new frame is added to the top of the call stack. When a function
completes, its frame is removed from the stack. This process continues until the main program
finishes executing. Understanding how the call stack works is essential for tracking function
calls, diagnosing errors, and optimizing code performance.

5. Visualizing the Call Stack with Diagrams

To better understand the call stack in Python, let's visualize it with diagrams. Visual
representations help us grasp the concept more easily. We'll walk through real-world examples
to see how the call stack changes as functions are called and completed. This visualization will
enhance our ability to track function calls, identify potential errors, and optimize the
performance of our Python code.

6. Real-world Examples of the Call Stack

To further illustrate the concept of the call stack, let's dive into some real-world examples. We
will explore different scenarios where functions are called and completed, tracking the changes
in the call stack along the way. By delving into these examples, we will enhance our
understanding of how the call stack works and gain practical insights into troubleshooting and
optimizing our Python code.

7. Common Issues and Errors related to the Call Stack

While understanding the call stack is crucial, it's also important to be aware of common issues
and errors that can occur. Stack overflow errors, infinite recursion, and forgetting to return a
value are some of the common pitfalls programmers may encounter. By recognizing these
 Page 147 of 343

issues and learning how to troubleshoot them, we can write more efficient and bug-free Python
code.

8. Best Practices for Working with the Call Stack

To effectively work with the call stack in Python, consider the following best practices:

1. Limit the depth: Avoid a deep call stack by breaking down complex tasks into smaller,
manageable functions.
2. Use recursion carefully: Recursive calls can quickly consume the call stack's memory.
Ensure there are proper base cases and termination conditions to avoid infinite recursion.

3. Understand memory allocation: Be mindful of memory usage when passing large data
structures as function arguments or returning them.

4. Debugging with tracebacks: Use tracebacks to trace the execution path and identify any
errors or unexpected behavior in your code.

5. Keep code modular: Breaking your code into logical functions makes it easier to understand,
test, and maintain.
6. Document and comment: Clearly document the purpose, inputs, and outputs of each
function to enhance understanding and collaboration.

7Test extensively: Thoroughly test your functions, including edge cases, to ensure they behave
as expected and avoid unexpected errors.

8. Optimize for performance: Assess your code's efficiency, optimizing for speed and memory
usage, if necessary.

Remember, by following these best practices, you can effectively leverage the call stack in
Python while minimizing errors and maximizing code efficiency.

9. Advanced Concepts and Techniques with the Call Stack

Once you have a solid understanding of the basics of the call stack in Python, you can explore
more advanced concepts and techniques. This includes topics such as recursive algorithms, tail
recursion optimization, call stack visualization tools, and strategies for handling stack
overflow errors. Understanding these advanced concepts will enable you to take your Python
programming skills to the next level and write more efficient and scalable code.

[Link] and Key Takeaways

 Page 148 of 343

Understanding the call stack in Python is crucial for writing efficient and scalable code. By
visualizing the call stack using diagrams and real-world examples, you can gain a deeper
understanding of how function calls are executed and managed in Python. Remember to
consider advanced concepts like recursive algorithms, tail recursion optimization, and
strategies for handling stack overflow errors to enhance your Python programming skills.

Memory Allocation in Python

A Comprehensive Guide to Memory Allocation in Python: From Scratch to Advanced

1. Introduction to Memory Allocation in Python

Welcome to the comprehensive guide on Memory Allocation in Python. Whether you are a
beginner or an advanced Python programmer, this guide will take you through every aspect of
memory allocation in Python. From understanding the basics to diving deep into advanced
techniques, this guide will equip you with the knowledge and skills necessary to optimize
memory usage and write efficient Python code. Let's embark on this journey together!

2. Basic concepts of Memory Allocation

In order to understand memory allocation in Python, it is important to grasp the basic concepts.
This includes understanding how memory is allocated and deallocated, the role of variables
and data types in memory allocation, and the difference between stack and heap memory.
Having a strong foundation in these concepts will pave the way for exploring more advanced
techniques in memory management. Let's delve into the basics of memory allocation in
Python.

3. Understanding Heap and Stack

To fully understand memory allocation in Python, it is crucial to differentiate between heap

and stack memory. The heap is used for dynamically allocated objects, while the stack stores
local variables and function calls. Understanding how these two memory spaces work together
is essential for optimizing memory usage and avoiding common pitfalls in Python
programming.

4. Memory Management in Python

In Python, memory management is handled automatically through a process called garbage

collection. The garbage collector identifies unused objects and frees up memory for reuse.
However, it is still important for programmers to be aware of memory management principles,
such as avoiding circular references and efficiently managing large data structures, to optimize
memory usage and enhance program performance.

5. Memory Allocation Algorithms and Techniques

 Page 149 of 343

These are vast no of Techniques please go and research on it will help to understand the way
of allocation and its pattern based on structures of data to search and insert, delete the data in
the memory in the real world

Recursion
The What, How, and When of Recursion
Recursion occurs when a function call causes that same function to be called again before
the original function call terminates. For example, consider the well-known mathematical
expression x! (i.e. the factorial operation). The factorial operation is defined for all non
negative integers as follows:

If the number is 0, then the answer is 1.

Otherwise, the answer is that number times the factorial of one less than that number.

In Python, a naïve implementation of the factorial operation can be defined as a function as

follows:

Recursion functions can be difficult to grasp sometimes, so let's walk through this step-by-
step. Consider the expression factorial(3). This and all function calls create a new
environment. An environment is basically just a table that maps identifiers (e.g. n,
factorial, print, etc.) to their corresponding values. At any point in time, you can
access the current environment using locals(). In the first function call, the only local
variable that gets defined is n = 3. Therefore, printing locals() would show {'n': 3}.
Since n == 3, the return value becomes n * factorial(n - 1).

At this next step is where things might get a little confusing. Looking at our new
expression, we already know what n is. However, we don't yet know what factorial(n -
1) is. First, n - 1 evaluates to 2. Then, 2 is passed to factorial as the value for n. Since
this is a new function call, a second environment is created to store this new n. Let A be
the first environment and B be the second environment. A still exists and equals {'n':
3}, however, B (which equals {'n': 2}) is the current environment. Looking at the
function body, the return value is, again, n * factorial(n - 1). Without evaluating
 Page 150 of 343

this expression, let's substitute it into the original return expression. By doing this, we're
mentally discarding B, so remember to substitute n accordingly (i.e. references to B's n are
replaced with n - 1 which uses A's n). Now, the original return expression becomes n *
((n - 1) * factorial((n
- 1) - 1)). Take a second to ensure that you understand why this is so.

Now, let's evaluate the factorial((n - 1) - 1)) portion of that. Since A's n == 3, we're
passing 1 into factorial. Therefore, we are creating a new environment C which equals
{'n': 1}. Again, the return value is n * factorial(n1). So let's replace factorial((n -
1) - 1)) of the “original” return expression similarly to how we adjusted the original return
expression earlier. The “original” expression is now n * ((n - 1) * ((n - 2) *
factorial((n - 2) - 1))).

Almost done. Now, we need to evaluate factorial((n - 2) - 1). This time, we're
passing in 0. Therefore, this evaluates to 1. Now, let's perform our last substitution. The
“original” return expression is now n * ((n - 1) * ((n2) * 1)). Recalling that the
original return expression is evaluated under A, the expression becomes 3 * ((3 - 1) *
((3 - 2) * 1)). This, of course, evaluates to 6. To confirm that this is the correct
answer, recall that 3! == 3*2 * 1 == 6. Before reading any further, be sure that you fully
understand the concept of environments and how they apply to [Link] statement

if n == 0: return 1 is called a base case. This is because, it exhibits no recursion.

A base case is absolutely required. Without one, you'll run into infinite recursion. With that
said, as long as you have at least one base case, you can have as many cases as you want. For
example, we could have equivalently written factorial as
follows:

You may also have multiple recursion cases, but we won't get into that since it's relatively
uncommon and is often difficult to mentally process.
You can also have “parallel” recursive function calls. For example, consider the Fibonacci
sequence which is defined as follows:

If the number is 0, then the

answer is 0. If the number is 1,
then the answer is 1.
 Page 151 of 343

Otherwise, the answer is the sum of the previous two Fibonacci numbers.

We can define this is as follows:

I won't walk through this function as thoroughly as I did with factorial(3), but the final
return value of fib(5) is equivalent to the following (syntactically invalid) expression:

This becomes (1 + (0 + 1)) + ((0 + 1) + (1 + (0 + 1))) which of

course evaluates to 5. Now, let's cover a few more vocabulary

terms:

A tail call is simply a recursive function call which is the last operation to be
performed before returning a value. To be clear, return foo(n - 1) is a tail call,
but return foo(n - 1) + 1 is not (since the addition is the last operation).
Tail call optimization (TCO) is a way to automatically reduce recursion in recursive
 Page 152 of 343

functions.
Tail call elimination (TCE) is the reduction of a tail call to an expression that
can be evaluated without recursion. TCE is a type of TCO.

Tail call optimization is helpful for a number of reasons:

The interpreter can minimize the amount of memory occupied by environments.

Since no computer has unlimited memory, excessive recursive function calls
would lead to a stack overflow.
The interpreter can reduce the number of stack frame switches.

Python has no form of TCO implemented for a number of a reasons. Therefore, other
techniques are required to skirt this limitation. The method of choice depends on the use
case. With some intuition, the definitions of factorial and fib can relatively easily be
converted to iterative code as follows:

This is usually the most efficient way to manually eliminate recursion, but it can become
rather difficult for more complex functions.

Another useful tool is Python's lru_cache decorator which can be used to reduce the
number of redundant calculations.

You now have an idea as to how to avoid recursion in Python, but when should you use
recursion? The answer is “not often”. All recursive functions can be implemented
iteratively. It's simply a matter of figuring out how to do so. However, there are rare cases
in which recursion is okay. Recursion is common in Python when the expected inputs
wouldn't cause a significant number of a recursive function calls.

If recursion is a topic that interests you, I implore you to study functional languages such
 Page 153 of 343

as Scheme or Haskell. In such languages, recursion is much more useful.

Please note that the above example for the Fibonacci sequence, although good at showing
how to apply the definition in python and later use of the lru cache, has an inefficient

running time since it makes 2 recursive calls for each non base case. The number of calls
to the function grows exponentially to n.
Rather non-intuitively a more efficient implementation would use linear recursion:

But that one has the issue of returning a pair of numbers. This emphasizes that some
functions really do not gain much from recursion.

Tree exploration with recursion

Say we have the following tree:

AA
AB

BA
BB

Now, if we wish to list all the names of the elements, we could do this with a simple for-
loop. We assume there is a function get_name() to return a string of the name of a
node, a function get_children() to return a list of all the sub-nodes of a given node in
the tree, and a function get_root() to get the root node.
 Page 154 of 343

This works well and fast, but what if the sub-nodes, got sub-nodes of its own? And those
sub-nodes might have more sub-nodes... What if you don't know beforehand how many
there will be? A method to solve this is the use of recursion.

Perhaps you wish to not print, but return a flat list of all node names. This can be done by
passing a rolling list as a parameter.

Sum of numbers from 1 to n

If I wanted to find out the sum of numbers from 1 to n where n is a natural number, I can do
1 + 2 + 3 + 4 + ...
+ (several hours later) + n. Alternatively, I could write a for loop:

Or I could use a technique known as recursion:

Recursion has advantages over the above two methods. Recursion takes less time than
writing out 1 + 2 + 3 for a sum from 1 to 3. For recursion(4), recursion can be used to
work backwards:

Function calls: ( 4 -> 4 + 3 -> 4 + 3 + 2 -> 4 + 3 + 2 + 1 -> 10 )

 Page 155 of 343

Whereas the for loop is working strictly forwards: ( 1 -> 1 + 2 -> 1 + 2 + 3 -> 1 + 2 + 3 +
4 -> 10 ). Sometimes the recursive solution is simpler than the iterative solution. This is
evident when implementing a reversal of a linked list.

Increasing the Maximum Recursion Depth

There is a limit to the depth of possible recursion, which depends on the Python
implementation. When the limit is reached, a RuntimeError exception is raised:

Here's a sample of a program that would cause this error:

It is possible to change the recursion depth limit by using

You can check what the current parameters of the limit are by running:

Running the same method above with our new limit we get

From Python 3.5, the exception is a RecursionError, which is derived from RuntimeError.
Recursion limit
There is a limit to the depth of possible recursion, which depends on the Python
implementation. When the limit is reached, a RuntimeError exception is raised:
 Page 156 of 343

It is possible to change the recursion depth limit by using [Link](limit)

and check this limit by
[Link]().

From Python 3.5, the exception is a RecursionError, which is derived from RuntimeError.

Recursive Lambda using assigned variable

One method for creating recursive lambda functions involves assigning the function to a
variable and then referencing that variable within the function itself. A common example
of this is the recursive calculation of the factorial of a number - such as shown in the
following code:

Description of code

The lambda function, through its variable assignment, is passed a value (4) which it
evaluates and returns 1 if it is 0 or else it returns the current value (i) * another
calculation by the lambda function of the value - 1 (i-1). This continues until the passed
value is decremented to 0 (return 1). A process which can be visualized as:
 Page 157 of 343

Recursive functions
A recursive function is a function that calls itself in its definition. For example the
mathematical function, factorial, defined by factorial(n) = n*(n-1)*(n-2)*...*3*2*1.
can be programmed as

the outputs here are:

 Page 158 of 343

as expected. Notice that this function is recursive because the second return
factorial(n-1), where the function calls itself in its definition.

Some recursive functions can be implemented using lambda, the factorial function using
lambda would be something like this:

The function outputs the same as above.

Mutable vs Immutable (and Hashable) in Python

Mutable vs Immutable
There are two kind of types in Python. Immutable types and mutable types.

Immutables

An object of an immutable type cannot be changed. Any attempt to modify the object will
result in a copy being created.

This category includes: integers, floats, complex, strings, bytes, tuples, ranges and frozensets.

To highlight this property, let's play with the id builtin. This function returns the unique
identifier of the object passed as parameter. If the id is the same, this is the same object. If
it changes, then this is another object. (Some say that this is actually the memory address of
the object, but beware of them, they are from the dark side of the force...)

Okay, 1 is not 3... Breaking news... Maybe not. However, this behavior is often forgotten
when it comes to more complex types, especially strings.
 Page 159 of 343

Aha! See? We can modify it!

No. While it seems we can change the string named by the variable stack, what we actually
do, is creating a new object to contain the result of the concatenation. We are fooled
because in the process, the old object goes nowhere, so it is destroyed. In another situation,
that would have been more obvious:

In this case it is clear that if we want to retain the first string, we need a copy. But is that so
obvious for other types?

Exercise

Now, knowing how immutable types work, what would you say with the below piece of code?
Is it wise?

Mutables

An object of a mutable type can be changed, and it is changed in-situ. No

implicit copies are done. This category includes: lists, dictionaries, bytearrays

and sets.

Let's continue to play with our little id function.

 Page 160 of 343

(As a side note, I use bytes containing ascii data to make my point clear, but remember that bytes are
not designed to hold textual data. May the force pardon me.)
What do we have? We create a bytearray, modify it and using the id, we can ensure that
this is the same object, modified. Not a copy of it.

Of course, if an object is going to be modified often, a mutable type does a much better
job than an immutable type. Unfortunately, the reality of this property is often forgotten
when it hurts the most.

Okay...

Waiiit a second...

Indeed. c is not a copy of b. c is b.

Exercise Now you better understand what side effect is implied by a mutable type, can
you explain what is going wrong in this example?
 Page 161 of 343

Mutable and Immutable as Arguments

One of the major use cases when a developer needs to take mutability into account is when
passing arguments to a function. This is very important, because this will determine the
ability for the function to modify objects that doesn't belong to its scope, or in other
words if the function has side effects. This is also important to understand where the
result of a function has to be made available.

Here, the mistake is to think that lin, as a parameter to the function, can be modified
locally. Instead, lin and a reference the same object. As this object is mutable, the
modification is done in-place, which means that the object referenced by both lin and a is
modified. lin doesn't really need to be returned, because we already have a reference to
this object in the form of a. a and b end referencing the same object.

This doesn't go the same for tuples.

At the beginning of the function, tin and a reference the same object. But this is an
immutable object. So, when the function tries to modify it, tin receive a new object with
the modification, while a keeps a reference to the original object. In this case, returning
tin is mandatory, or the new object would be lost.
 Page 162 of 343

Exercise

Note: reverse operates in-place.

What do you think of this function? Does it have side effects? Is the return necessary?
After the call, what is the value of saying? Of focused? What happens if the function is
called again with the same parameters?

Difference between Procedural programming

and
Object-Oriented Programming (OOP)
Procedural programming and Object-Oriented Programming (OOP) are two different
programming paradigms, each with its own approach to structuring code and solving
problems. Here's a comparison of the two paradigms in the context of Python:
Procedural Programming (Procedural-Oriented):
Structure:
In procedural programming, code is organized around procedures, functions, or methods that
perform specific tasks.
It typically involves writing a sequence of functions or procedures that are executed one after
another, often with shared data passed as arguments.
Data:
Data in procedural programming is often organized in data structures like lists, arrays, and
dictionaries.
Data and functions are loosely coupled, meaning that data can be accessed and modified by
any part of the program.
Modularity:
Modularity is achieved through functions or procedures.
Code is divided into functions that perform specific tasks, and these functions can be reused
across the program.
 Page 163 of 343

Examples:
Procedural Python code might include a series of functions that manipulate data, like sorting a
list or calculating statistics.

def calculate_average(numbers):
total = sum(numbers)
return total / len(numbers)
Object-Oriented Programming (OOP):
Structure:
In OOP, code is organized around objects, which represent real-world entities and encapsulate
both data (attributes) and behavior (methods).
Objects are instances of classes, which define the blueprint for creating objects.
Data:
Data and methods that operate on that data are bundled together within objects, promoting data
encapsulation.
Access to an object's data is controlled through methods, making it easier to maintain and
protect the data.
Modularity:
Modularity is achieved through classes and objects.
Code is divided into classes that define the structure and behavior of objects, and these classes
can be instantiated into multiple objects.
Examples:
Object-oriented Python code might include classes like "Person" with attributes (e.g., name,
age) and methods (e.g., "speak" or "eat") to model real-world entities.

class Person:
def _init_(self, name, age):
[Link] = name
[Link] = age

def speak(self):
return f"{[Link]} says hello!"

alice = Person("Alice", 30)

print([Link]()) # Outputs "Alice says hello!"
 Page 164 of 343

Key Differences:
Procedural programming is function-centered and focuses on procedures and functions.
Object-oriented programming is object-centered and focuses on creating and manipulating
objects.
OOP promotes encapsulation, making it easier to maintain and protect data.
OOP supports concepts like inheritance, polymorphism, and encapsulation, which allow for
more complex and structured code organization.
Procedural programming is often simpler and more suitable for smaller programs, while OOP
is favored for larger and more complex applications due to its ability to model real-world
entities effectively.
Both paradigms have their strengths and use cases. In Python, you have the flexibility to use
both procedural and object-oriented approaches, depending on the nature and requirements
of your project.

Classes
Python offers itself not only as a popular scripting language, but also supports the object-
oriented programming paradigm. Classes describe data and provide methods to manipulate that
data, all encompassed under a single object. Furthermore, classes allow for abstraction by
separating concrete implementation details from abstract representations of data.
Code utilizing classes is generally easier to read, understand, and maintain.
Introduction to classes
A class, functions as a template that defines the basic characteristics of a particular object.

"Now
 Page 165 of 343

Here's an example:

There are a few things to note when looking at the above example.

1. The class is made up of attributes (data) and methods (functions).

2. Attributes and methods are simply defined as normal variables and functions.
3. As noted in the corresponding docstring, the init () method is called the
initializer. It's equivalent to the constructor in other object oriented languages, and is
the method that is first run when you create a new object, or new instance of the
class.
4. Attributes that apply to the whole class are defined first, and are called class attributes.
5. Attributes that apply to a specific instance of a class (an object) are called instance
attributes. They are generally defined inside init (); this is not necessary, but it
is recommended (since attributes defined outside of init () run the risk of being
accessed before they are defined).
6. Every method, included in the class definition passes the object in question as its first
parameter. The word self is used for this parameter (usage of self is actually by
convention, as the word self has no inherent meaning in Python, but this is one of
Python's most respected conventions, and you should always follow it).
7. Those used to object-oriented programming in other languages may be surprised by a
few things. One is that Python has no real concept of private elements, so
everything, by default, imitates the behavior of the C++/Java public keyword. For
more information, see the "Private Class Members" example on this page.
8. Some of the class's methods have the following form: functionname
(self, other_stuff). All such methods are called "magic methods" and are an
important part of classes in Python. For instance, operator overloading in Python is
implemented with magic methods. For more information, see the relevant
documentation.

Now let's make a few instances of our Person class!

 Page 166 of 343

We currently have three Person objects, kelly, joseph, and john_doe.

We can access the attributes of the class from each instance using the dot operator . Note
again the difference between class and instance attributes:

We can execute the methods of the class using the same dot operator .:

Certainly! In Python, classes and objects are fundamental concepts in object-oriented programming
(OOP). Let's break down these concepts and clarify how they relate to each other:
Classes:

Definition: A class is a blueprint or template for creating objects. It defines a set of attributes (data)
and methods (functions) that the objects created from the class will have.
Attributes: Attributes are variables defined within a class and represent the characteristics or
properties of objects created from that class. They store data that is associated with objects.
Methods: Methods are functions defined within a class. They define the behaviors or actions that
objects created from the class can perform. Methods can manipulate the attributes of objects and
perform various tasks.

Example:

class Person:
def _init_(self, name, age):
[Link] = name
[Link] = age

def greet(self):
print(f"Hello, my name is {[Link]} and I am {[Link]} years old.")
 Page 167 of 343

# In this example, "Person" is a class with attributes (name and age) and a method (greet).
Objects:

Definition: An object is an instance of a class. It is a concrete realization of the blueprint defined

by the class. Objects have their own unique data (attribute values) and can perform actions
(method calls) as defined by the class.

Creation: Objects are created from classes using the class constructor. In Python, the class
constructor is usually the _init_ method, which initializes the object's attributes.

Example:

# Creating objects (instances) of the Person class

person1 = Person("Alice", 30)
person2 = Person("Bob", 25)

# Calling the greet method on objects

[Link]() # Outputs: Hello, my name is Alice and I am 30 years old.
[Link]() # Outputs: Hello, my name is Bob and I am 25 years old.
Relationship between Classes and Objects:

Classes define the structure and behavior of objects.

Objects are instances of classes, and they inherit the attributes and methods defined in their
class.
Each object created from the same class can have different attribute values while sharing the
same methods.
In the example above, the Person class defines the blueprint for creating person objects. When
we create person1 and person2, we are creating two distinct objects with their own name and age
attributes. These objects also share the same greet method, which can be called on each object to
display personalized information.

In summary, classes are used to create templates for objects, defining their structure and
behavior. Objects, on the other hand, are instances of these classes, each with its own data and
the ability to perform actions specified by the class methods. This encapsulation and abstraction
provided by classes and objects are fundamental principles of object-oriented programming.

Default values for instance variables

If the variable contains a value of an immutable type (e.g. a string) then it is okay to
assign a default value like this
 Page 168 of 343

One needs to be careful when initializing mutable objects such as lists in the
constructor. Consider the following example:

This behavior is caused by the fact that in Python default parameters are bound at
function execution and not at function declaration. To get a default instance variable
that's not shared among instances, one should use a construct like this:
 Page 169 of 343

See also Mutable Default Arguments and “Least Astonishment” and the Mutable Default
Argument.

Class and instance variables

Instance variables are unique for each instance, while class variables are shared by all
instances.

# 2

#2

#3

#2

#4

Class variables can be accessed on instances of this class, but assigning to the class attribute
will create an instance variable which shadows the class variable

# 4

# 2

Note that mutating class variables from instances can lead to some unexpected
consequences.
 Page 170 of 343

Class composition
Class composition allows explicit relations between objects. In this example, people live in
cities that belong to countries. Composition allows people to access the number of all
people living in their country:
 Page 171 of 343

# 15

Variables and Attributes

Variables are annotated using s:

Python 3.x Version ≥ 3.6

Starting from Python 3.6, there is also new syntax for variable annotations. The code above
might use the form

Unlike with comments, it is also possible to just add a type hint to a variable that was not
previously declared, without setting a value to it:
 Page 172 of 343

Additionally if these are used in the module or the class level, the type hints can be retrieved
using
typing.get_type_hints(class_or_module):

Alternatively, they can be accessed by using the annotations special variable or attribute:

Attribute Access
Basic Attribute Access using the Dot Notation
Let's take a sample class.

In Python you can access the attribute title of the class using the dot notation.

If an attribute doesn't exist, Python throws an error:

 Page 173 of 343

Monkey Patching
In this case, "monkey patching" means adding a new variable or method to a class after it's
been defined. For instance, say we defined class A as

But now we want to add another function later in the code. Suppose this function is as follows.

But how do we add this as a method in A? That's simple we just essentially place that
function into A with an assignment statement.

Why does this work? Because functions are objects just like any other object, and methods
are functions that belong to the class.

The function get_num shall be available to all existing (already created) as well to the new
instances of A
These additions are available on all instances of that class (or its subclasses) automatically. For
example:

Note that, unlike some other languages, this technique does not work for certain built-in
types, and it is not considered good style.
 Page 174 of 343

Listing All Class Members

The dir() function can be used to get a list of the members of a class:

For example:

It is common to look only for "non-magic" members. This can be done using a simple
comprehension that lists members with names not starting with :

Class methods: alternate initializers

Class methods present alternate ways to build instances of classes. To illustrate, let's
look at an example. Let's suppose we have a relatively simple Person class:

It might be handy to have a way to build instances of this class specifying a full name
instead of first and last name separately. One way to do this would be to have last_name
be an optional parameter, and assuming that if it isn't given, we passed the full name in:
 Page 175 of 343

However, there are two main problems with this bit of code:

1. The parameters first_name and last_name is now misleading, since you can
enter a full name for first_name. Also, if there are more cases and/or more
parameters that have this kind of flexibility, the if/elif/else branching can get
annoying fast.
2. Not quite as important, but still worth pointing out: what if last_name is None,
but first_name doesn't split into two or more things via spaces? We have yet
another layer of input validation and/or exception handling...
Enter class methods. Rather than having a single initializer, we will create a separate
initializer, called
from_full_name, and decorate it with the (built-in) classmethod decorator.
 Page 176 of 343

Notice cls instead of self as the first argument to from_full_name. Class methods are
applied to the overall class, not an instance of a given class (which is what self usually
denotes). So, if cls is our Person class, then the returned value from the from_full_name
class method is Person(first_name, last_name, age), which uses Person's
init to create an instance of the Person class. In particular, if we were to make a
subclass Employee of Person, then from_full_name would work in the Employee class
as well.

To show that this works as expected, let's create instances of Person in more than one
way without the branching in init :

Class Members and Methods

Forward reference of the current class is needed since annotations are evaluated when the
function is defined. Forward references can also be used when referring to a class that
would cause a circular import if imported.
 Page 177 of 343

User-Defined Methods
Creating user-defined method objects
User-defined method objects may be created when getting an attribute of a class (perhaps
via an instance of that class), if that attribute is a user-defined function object, an unbound
user-defined method object, or a class method object.

When the attribute is a user-defined method object, a new method object is only created if
the class from which it is being retrieved is the same as, or a derived class of, the class
stored in the original method object; otherwise, the original method object is used as it is.
 Page 178 of 343

Static methods
In Python, a static method is a method that belongs to a class rather than an instance of the class.
Static methods are defined using the @staticmethod decorator and do not have access to
instance-specific attributes or methods. They are primarily used for utility functions or
operations that don't require access to instance-specific data. Here's a clear example and
description of a static method in Python:

class MathUtils:
@staticmethod
def add(x, y):
return x + y

@staticmethod
def subtract(x, y):
return x - y

# Using the static methods without creating an instance of the class

result1 = [Link](5, 3)
result2 = [Link](10, 4)

print("Addition result:", result1) # Output: Addition result: 8

print("Subtraction result:", result2) # Output: Subtraction result: 6
In the example above, we have created a MathUtils class with two static methods: add and
subtract. Here's how static methods work:

No Instance Required: You can call static methods directly on the class itself, without needing to
create an instance of the class. In the example, we call [Link](5, 3) and
[Link](10, 4) without creating any MathUtils objects.
 Page 179 of 343

No Access to Instance-specific Data: Static methods do not have access to instance-specific data
or attributes. They only have access to their parameters and other static members of the class.
This makes them suitable for utility functions that don't rely on the state of individual objects.

Decorating with @staticmethod: To define a static method, you use the @staticmethod decorator
before the method definition.

Common Use Cases:

Utility Functions: Static methods are often used for utility functions that are related to the class
but don't depend on the state of instances. For example, math operations, data validation, or
formatting functions.
Factory Methods: They can be used as factory methods to create instances of the class, but this is
less common than other use cases.
Accessing Static Methods: Static methods can be accessed through the class itself, as shown in
the example. You can also access them through instances of the class, but this is not a
recommended practice.

In summary, static methods in Python are defined within a class but are not tied to instances of
the class. They are useful for encapsulating utility functions or operations that don't need access
to instance-specific data. Static methods provide a clean way to organize and use functions
related to a class without the need for object instantiation.
In a class, you can have both static methods and regular (instance) methods. The key difference
between them lies in how they are called and their access to class attributes. Let's illustrate the
difference with an example:

class MyClass:
class_variable = "I'm a class variable"

def _init_(self, instance_variable):

self.instance_variable = instance_variable

def instance_method(self):
print("This is an instance method")
print("Instance variable:", self.instance_variable)
print("Class variable:", MyClass.class_variable)

@staticmethod
def static_method():
print("This is a static method")
# Static methods don't have access to instance variables or methods
# They can access class variables directly
print("Class variable:", MyClass.class_variable)

# Creating an instance of MyClass

 Page 180 of 343

obj = MyClass("I'm an instance variable")

# Calling an instance method

obj.instance_method()

# Calling a static method

MyClass.static_method()
In this example, we have a class MyClass with both an instance method (instance_method) and a
static method (static_method).

Instance Method (instance_method):

An instance method is a method that operates on an instance of the class.

It has access to both instance variables (like self.instance_variable) and class variables (like
MyClass.class_variable) because it operates within the context of an instance.
To call an instance method, you need to create an instance of the class (in this case, obj) and then
call the method on that instance.
Static Method (static_method):

A static method is a method that belongs to the class itself, not to instances of the class.
It doesn't have access to instance-specific data (e.g., self.instance_variable), but it can access
class variables (e.g., MyClass.class_variable) directly because it operates at the class level.
You can call a static method on the class itself, without creating an instance of the class.
When you run the code:

Calling obj.instance_method() invokes the instance method, and it can access both instance and
class variables.

Calling MyClass.static_method() invokes the static method, and it can only access class
variables directly. It doesn't need an instance to work.

In summary, instance methods are associated with instances of a class and can access both
instance-specific and class-level data, while static methods are associated with the class itself
and can access class-level data but not instance-specific data.

Factory method
A factory method is a design pattern in object-oriented programming that provides a way to
create objects without specifying the exact class of object that will be created. In Python, you
can implement the factory method pattern using class methods or static methods within a class.
Here's an example of a factory method in a Python class:

class Animal:
def _init_(self, name):
[Link] = name

def speak(self):
pass
 Page 181 of 343

@classmethod
def create_animal(cls, name, animal_type):
if animal_type == "dog":
return Dog(name)
elif animal_type == "cat":
return Cat(name)
else:
raise ValueError("Invalid animal type")

class Dog(Animal):
def speak(self):
return f"{[Link]} says Woof!"

class Cat(Animal):
def speak(self):
return f"{[Link]} says Meow!"

# Using the factory method to create different animals

dog = Animal.create_animal("Buddy", "dog")
cat = Animal.create_animal("Whiskers", "cat")

print([Link]()) # Output: Buddy says Woof!

print([Link]()) # Output: Whiskers says Meow!

In this example:

We have a base class Animal with an _init_ method and an abstract speak method.

The Animal class contains a class method called create_animal. This method acts as a factory
method. It takes two arguments: name (the name of the animal) and animal_type (the type of
animal to create).

Inside the create_animal method, we check the animal_type parameter and create an instance of
the appropriate subclass (Dog or Cat) based on the type. This allows us to create different types
of animals without knowing the specific class implementation.

We have two subclasses, Dog and Cat, which inherit from the Animal class. Each subclass
implements the speak method.

Finally, we use the factory method to create instances of Dog and Cat without needing to know
the implementation details of these classes. We call the speak method on the created objects to
demonstrate their specific behaviors.

The factory method pattern is useful when you want to create objects with complex initialization
logic or when you want to decouple the client code from the concrete class implementations. It
provides a way to create objects based on certain conditions or parameters, making your code
more flexible and maintainable.
 Page 182 of 343

Here's a simpler example of a factory method in Python:

class Shape:
def area(self):
pass

@classmethod
def create_shape(cls, shape_type, *args):
if shape_type == "circle":
return Circle(*args)
elif shape_type == "rectangle":
return Rectangle(*args)
else:
raise ValueError("Invalid shape type")

class Circle(Shape):
def _init_(self, radius):
[Link] = radius

def area(self):
return 3.14 * [Link] ** 2

class Rectangle(Shape):
def _init_(self, length, width):
[Link] = length
[Link] = width

def area(self):
return [Link] * [Link]

# Using the factory method to create different shapes

circle = Shape.create_shape("circle", 5)
rectangle = Shape.create_shape("rectangle", 4, 6)

print("Circle Area:", [Link]()) # Output: Circle Area: 78.5

print("Rectangle Area:", [Link]()) # Output: Rectangle Area: 24
In this example:

We have a base class Shape with an area method and a class method create_shape acting as the
factory method.

The create_shape factory method takes a shape_type argument and additional arguments (such
as radius or length and width) depending on the shape type.

Inside the factory method, we determine the shape type and create an instance of the appropriate
subclass (Circle or Rectangle) based on that type.
 Page 183 of 343

We have two subclasses, Circle and Rectangle, each implementing their own area method.

We use the factory method to create instances of Circle and Rectangle without needing to know
the specific class details. Then, we call the area method on these objects to calculate and print
their areas.

This simple factory method example demonstrates how to create objects of different shapes
using a common interface (Shape) and a factory method, which makes it easy to extend the code
to support additional shape types in the future

Access Modifiers
Access modifiers in Python are keywords used to specify the visibility and accessibility of class
members (variables and methods) from outside the class. In Python, there are three primary
access modifiers:

Public (No Modifier):

If a class member has no access modifier, it is considered public.

Public members can be accessed from anywhere, both inside and outside the class.
Example:

class MyClass:
def _init_(self):
self.public_variable = 10

obj = MyClass()
print(obj.public_variable) # Accessing a public variable
Protected (Single Underscore Prefix):

A member with a single underscore prefix (e.g., _protected_variable) is considered protected.

Protected members should not be accessed directly from outside the class, although Python
doesn't enforce this restriction. It's more of a convention.
Example:

class MyClass:
def _init_(self):
self._protected_variable = 20

obj = MyClass()
print(obj._protected_variable) # Accessing a protected variable (not recommended)
Private (Double Underscore Prefix):

A member with a double underscore prefix (e.g., __private_variable) is considered private.

Private members are intended to be private to the class and should not be accessed directly from
outside the class. Python enforces name mangling to make it harder to access these variables
directly.
 Page 184 of 343

Example:

class MyClass:
def _init_(self):
self.__private_variable = 30

obj = MyClass()

# Accessing a private variable (Name mangling applied)

print(obj.MyClass_private_variable)
It's important to note that Python follows the principle of "we are all consenting adults here,"
which means that it doesn't enforce strict access control like some other programming languages
do. In Python, even private members can be accessed if needed, but it's considered a best
practice to respect the intended access levels (public, protected, private) and not access private
members from outside the class.

The use of underscores for protected and private members is more of a convention to indicate the
intended level of visibility and should be followed by developers to write clean and maintainable
code.

Basic inheritance
Inheritance in Python is based on similar ideas used in other object oriented languages
like Java, C++ etc. A new class can be derived from an existing class as follows.

The BaseClass is the already existing (parent) class, and the DerivedClass is the new
(child) class that inherits (or subclasses) attributes from BaseClass. Note: As of Python
2.2, all classes implicitly inherit from the object class, which is the base class for all
built-in types.

We define a parent Rectangle class in the example below, which implicitly inherits from
object:
 Page 185 of 343

The Rectangle class can be used as a base class for defining a Square class, as a square
is a special case of rectangle.

The Square class will automatically inherit all attributes of the Rectangle class as
well as the object class. super() is used to call the init () method of Rectangle class,
essentially calling any overridden method of the base class. Note: in Python 3, super()
does not require [Link] class objects can access and modify the attributes
of its base classes:

Built-in functions that work with inheritance

issubclass(DerivedClass, BaseClass): returns True if DerivedClass is a subclass
of the BaseClass isinstance(s, Class): returns True if s is an instance of Class or
any of the derived classe
 Page 186 of 343

Single Inheritance
In single inheritance, a subclass inherits from a single superclass.

class Animal:
def speak(self):
pass

class Dog(Animal):
def speak(self):
return "Woof!"

dog = Dog()
print([Link]()) # Output: Woof!

Multilevel Inheritance:
Multilevel inheritance occurs when a subclass inherits from a superclass, and then another class inherits
from this subclass.

class Grandparent:
def greet(self):
return "Hello from Grandparent!"

class Parent(Grandparent):
def greet(self):
return "Hello from Parent!"

class Child(Parent):
pass

child = Child()
print([Link]()) # Output: Hello from Parent

Multiple Inheritance
Python uses the C3 linearization algorithm to determine the order in which to resolve
class attributes, including methods. This is known as the Method Resolution Order
(MRO).
 Page 187 of 343

Here's a simple example:

Now if we instantiate FooBar, if we look up the foo attribute, we see that Foo's attribute is
found first

and

Here's the MRO of FooBar:

It can be simply stated that Python's MRO algorithm is

1. Depth first (e.g. FooBar then Foo) unless
2. a shared parent (object) is blocked by a child (Bar) and
3. no circular relationships allowed.

That is, for example, Bar cannot inherit from FooBar while FooBar

inherits from Bar. For a comprehensive example in Python, see the

Wikipedia entry.

Another powerful feature in inheritance is super. super can fetch parent classes features.
 Page 188 of 343

Multiple inheritance with init method of class, when every class has own init method
then we try for multiple inheritance then only init method get called of class which is
inherit first.

for below example only Foo class init method getting called Bar class init not getting called

Output:

But it doesn't mean that Bar class is not inherit. Instance of final FooBar class is also
instance of Bar class and Foo
class.
 Page 189 of 343

Output:

Method Overriding
Basic method overriding
Here is an example of basic overriding in Python (for the sake of clarity and compatibility
with both Python 2 and 3, using new style class and print with ()):

When the Child class is created, it inherits the methods of the Parent class. This
means that any methods that the parent class has, the child class will also have. In the
example, the introduce is defined for the Child class because it is defined for Parent,
despite not being defined explicitly in the class definition of Child.

In this example, the overriding occurs when Child defines its own print_name
method. If this method was not declared, then c.print_name() would have printed
"Parent". However, Child has overridden the Parent's definition of print_name,
and so now upon calling c.print_name(), the word "Child" is printed.
 Page 190 of 343

Method Overriding:
Method overriding is a concept in object-oriented programming (OOP) where a subclass provides a specific
implementation of a method that is already defined in its superclass. This allows the subclass to customize
or extend the behavior of the inherited method. Method overriding is a fundamental principle of
polymorphism, where objects of different classes can be treated as objects of a common superclass.

Here's an example of method overriding in Python:

class Animal:
def speak(self):
return "This is a generic animal sound."

class Dog(Animal):
def speak(self):
return "Woof!"

class Cat(Animal):
def speak(self):
return "Meow!"

# Creating instances of Dog and Cat

dog = Dog()
cat = Cat()

# Calling the speak method on Dog and Cat instances

print([Link]()) # Output: Woof!
print([Link]()) # Output: Meow!

Method overriding is a concept in object-oriented programming (OOP) where a subclass provides a specific
implementation of a method that is already defined in its superclass. This allows the subclass to customize
or extend the behavior of the inherited method. Method overriding is a fundamental principle of
polymorphism, where objects of different classes can be treated as objects of a common superclass.

Here's an example of method overriding in Python:

class Animal:
def speak(self):
return "This is a generic animal sound."
 Page 191 of 343

class Dog(Animal):
def speak(self):
return "Woof!"

class Cat(Animal):
def speak(self):
return "Meow!"

# Creating instances of Dog and Cat

dog = Dog()
cat = Cat()

# Calling the speak method on Dog and Cat instances

print([Link]()) # Output: Woof!
print([Link]()) # Output: Meow!

Method overriding is a concept in object-oriented programming (OOP) where a subclass provides a specific
implementation of a method that is already defined in its superclass. This allows the subclass to customize
or extend the behavior of the inherited method. Method overriding is a fundamental principle of
polymorphism, where objects of different classes can be treated as objects of a common superclass.

Here's an example of method overriding in Python:

class Animal:
def speak(self):
return "This is a generic animal sound."

class Dog(Animal):
def speak(self):
return "Woof!"

class Cat(Animal):
def speak(self):
return "Meow!"

# Creating instances of Dog and Cat

dog = Dog()
cat = Cat()
 Page 192 of 343

# Calling the speak method on Dog and Cat instances

print([Link]()) # Output: Woof!
print([Link]()) # Output: Meow!

C3 Linearization Algorithm: The C3 Linearization algorithm is used to compute the

MRO for a class. It takes into account the method resolution order of the base classes,
ensuring that it respects the inheritance hierarchy while preventing ambiguities.

Superclass Linearization: The C3 Linearization algorithm constructs a linearization of

the class hierarchy, known as the C3 linearization list. This list defines the order in
which the base classes are considered during method lookup. Here's an example that
demonstrates the Method Resolution Order (MRO) in Python's multiple inheritance:

class A:

def speak(self):

return "A speaks"

class B(A):

def speak(self):

return "B speaks"

class C(A):

def speak(self):

return "C speaks"

class D(B, C):

pass

d = D()

# When we call the speak method on d, it follows the MRO to determine which method
to invoke.

# The MRO is calculated as D -> B -> C -> A (left-to-right, depth-first).

 Page 193 of 343

print([Link]()) # Output: B speaks

Hierarchical Inheritance:

Hierarchical inheritance is when multiple subclasses inherit from a single superclass.

class Vehicle:
def move(self):
pass

class Car(Vehicle):
def move(self):
return "Car is moving on the road."

class Boat(Vehicle):
def move(self):
return "Boat is sailing on the water."

car = Car()
boat = Boat()
print([Link]()) # Output: Car is moving on the road.
print([Link]()) # Output: Boat is sailing on the water.

Hybrid inheritance
Hybrid inheritance is a combination of different types of inheritance within a single program. It typically
involves multiple inheritance (where a class inherits from more than one superclass) along with other types
of inheritance like single inheritance or multilevel inheritance. In other words, hybrid inheritance is a mix
of various inheritance types to create a complex class hierarchy.

Here's an example of hybrid inheritance in Python:

class Animal:
def speak(self):
pass
class Mammal(Animal):
def give_birth(self):
pass

class Bird(Animal):
 Page 194 of 343

def lay_eggs(self):
pass

class Dog(Mammal):
def speak(self):
return "Woof!"

class Bat(Mammal, Bird):

def fly(self):
return "I can fly!"

class Sparrow(Bird):
def speak(self):
return "Chirp chirp!"

dog = Dog()
bat = Bat()
sparrow = Sparrow()

print([Link]()) # Output: Woof!

print([Link]()) # Output: I can fly!
print([Link]()) # Output: Chirp chirp!

Polymorphism
Basic Polymorphism
Polymorphism is the ability to perform an action on an object regardless of its type. This
is generally implemented by creating a base class and having two or more subclasses that
all implement methods with the same signature. Any other function or method that
manipulates these objects can call the same methods regardless of which type of object it
is operating on, without needing to do a type check first. In object-oriented terminology
when class X extend class Y, then Y is called super class or base class and X is called
subclass or derived class.
 Page 195 of 343

"""
raise NotImplemented

class
Square(Shape
): """
This is a subclass of the Shape class, and
represents a square """
side_length = 2 # in this example, the sides are 2 units long

def
calculate_area(
self): """
This method overrides Shape.calculate_area(). When
an object of type Square has its calculate_area()
method called, this is the method that will be called,
rather than the parent class' version.

It performs the calculation necessary for this

shape, a square, and returns the result.
"""
return self.side_length * 2

class
Triangle(Shap
e): """
This is also a subclass of the Shape class, and it
represents a triangle """
base_length = 4
height = 3

def
calculate_area(
self): """
This method also overrides Shape.calculate_area() and
performs the area calculation for a triangle, returning the
result.
 Page 196 of 343

"""

return 0.5 * self.base_length * [Link]

def
get_area(inpu
t_obj): """
This function accepts an input object, and will call
that object's calculate_area() method. Note that the
object type is not specified. It could be a Square,
Triangle, or Shape object.
"""

print(input_obj.calculate_area())

# Create one object of each class

shape_obj = Shape()
square_obj =
Square()
triangle_obj =
Triangle()

# Now pass each object, one at a time, to the get_area()

function and see the # result.
get_area(shape_
obj)
get_area(square
_obj)
get_area(triangl
e_obj)

We should see this output:

Non
e4
6.0

What happens without polymorphism?

Without polymorphism, a type check may be required before performing an action on an
object to determine the correct method to call. The following counter example performs
the same task as the previous code, but without the use of polymorphism, the
get_area() function has to do more work.
 Page 197 of 343

We should see this output:

4
6.0

Important Note
Note that the classes used in the counter example are "new style" classes and implicitly
inherit from the object class if Python 3 is being used. Polymorphism will work in both
Python 2.x and 3.x, but the polymorphism counterexample code will raise an exception if
run in a Python 2.x interpreter because type(input_obj).name will return "instance" instead
of the class name if they do not explicitly inherit from object, resulting in area never being
assigned to.
 Page 198 of 343

Duck Typing
Polymorphism without inheritance in the form of duck typing as available in Python due
to its dynamic typing system. This means that as long as the classes contain the same
methods the Python interpreter does not distinguish between them, as the only checking of
the calls occurs at run-time.

The output is:

Quaaaaaack!
The duck has white and gray
feathers. The person imitates a
duck.
The person takes a feather from the ground and shows it.

Magic/Dunder Methods
Magic (also called dunder as an abbreviation for double-underscore) methods in Python
serve a similar purpose to operator overloading in other languages. They allow a class to
define its behavior when it is used as an operand in unary or binary operator expressions.
They also serve as implementations called by some built-in functions.

Consider this implementation of two-dimensional vectors.

 Page 199 of 343

Now it is possible to naturally use instances of the Vector class in various expressions.
 Page 200 of 343

String representations of class instances: str and

repr methods
Motivation
So, you've just created your first class in Python, a neat little class that encapsulates a playing
card:

Elsewhere in your code, you create a few instances of this class:

You've even created a list of cards, in order to represent a "hand":

Now, during debugging, you want to see what your hand looks like, so you do what comes
naturally and write:

But what you get back is a bunch of gibberish:

Confused, you try just printing a single card:

And again, you get this weird output:

Have no fear. We're about to fix this.

 Page 201 of 343

First, however, it's important to understand what's going on here. When you wrote
print(ace_of_spades) you told Python you wanted it to print information about the
Card instance your code is calling ace_of_spades. And to be fair, it did.

That output is comprised of two important bits: the type of the object and the object's id.
The second part alone (the hexadecimal number) is enough to uniquely identify the
object at the time of the print call.[1]

What really went on was that you asked Python to "put into words" the essence of that
object and then display it to you. A more explicit version of the same machinery might be:

In the first line, you try to turn your Card instance into a string, and in the second you display
it.

The Problem

The issue you're encountering arises due to the fact that, while you told Python everything
it needed to know about the Card class for you to create cards, you didn't tell it how you
wanted Card instances to be converted to strings.

And since it didn't know, when you (implicitly) wrote str(ace_of_spades), it gave you what
you saw, a generic representation of the Card instance.

The Solution (Part 1)

But we can tell Python how we want instances of our custom classes to be converted to
strings. And the way we do this is with the str "dunder" (for double-underscore) or
"magic" method.

Whenever you tell Python to create a string from a class instance, it will look for a str
method on the class, and
call it.

Consider the following, updated version of our Card class:

 Page 202 of 343

Here, we've now defined the str method on our Card class which, after a simple
dictionary lookup for face cards, returns a string formatted however we decide.

(Note that "returns" is in bold here, to stress the importance of returning a string, and
not simply printing it. Printing it may seem to work, but then you'd have the card
printed when you did something like str(ace_of_spades), without even having a print
function call in your main program. So to be clear, make sure that
str returns a string.).

The str method is a method, so the first argument will be self and it should neither
accept, nor be passed additional arguments.

Returning to our problem of displaying the card in a more user-friendly manner, if we again
run:

We'll see that our output is much

better: Ace of Spades

So great, we're done, right?

Well just to cover our bases, let's double check that we've solved the first issue we
encountered, printing the list of
Card instances, the hand.

So we re-check the following code:

 Page 203 of 343

And, to our surprise, we get those funny hex codes again:

What's going on? We told Python how we wanted our Card instances to be displayed, why did
it apparently seem to forget?

The Solution (Part 2)

Well, the behind-the-scenes machinery is a bit different when Python wants to get the
string representation of items in a list. It turns out, Python doesn't care about str
for this purpose.

Instead, it looks for a different method, repr , and if that's not found, it falls back on the
"hexadecimal thing".[2]

So you're saying I have to make two methods to do the same thing? One for when I want to
print my card by itself and another when it's in some sort of container?

No, but first let's look at what our class would be like if we were to implement both str
and repr methods:

Here, the implementation of the two methods str and repr are exactly the same,
except that, to differentiate between the two methods, (S) is added to strings returned
by str and (R) is added to strings returned by repr .

Note that just like our str method, repr accepts no arguments and returns a string.

We can see now what method is responsible for each case:

 Page 204 of 343

As was covered, the str method was

called when we passed our Card instance to print and the repr
method was called when we passed a list of our instances to print.

At this point it's worth pointing out that just as we can explicitly create a string from a
custom class instance using str() as we did earlier, we can also explicitly create a string
representation of our class with a built-in function called repr().

For example:

And additionally, if defined, we could call the methods directly (although it seems a bit unclear
and unnecessary):

About those duplicated functions...

Python developers realized, in the case you wanted identical strings to be returned from
str() and repr() you might have to functionally-duplicate methods -- something
nobody likes.

So instead, there is a mechanism in place to eliminate the need for that. One I snuck you
past up to this point. It turns out that if a class implements the repr method but not the
str method, and you pass an instance of that class to
str() (whether implicitly or explicitly), Python will fallback on your repr
implementation and use that.

So, to be clear, consider the following version of the Card class:

 Page 205 of 343

Note this version only implements the repr method. Nonetheless, calls to str()
result in the user-friendly version:

as do calls to repr():

Summary

In order for you to empower your class instances to "show themselves" in user-friendly
ways, you'll want to consider implementing at least your class's repr method. If
memory serves, during a talk Raymond Hettinger said that ensuring classes implement
repr is one of the first things he looks for while doing
Python code reviews, and by now it should be clear why. The amount of information
you could have added to debugging statements, crash reports, or log files with a simple
method is overwhelming when compared to the paltry, and often less-than-helpful
(type, id) information that is given by default.

If you want different representations for when, for example, inside a container, you'll want to
implement both
repr and str methods. (More on how you might use these two methods differently
below).
 Page 206 of 343

Overloading
Operator overloading
Below are the operators that can be overloaded in classes, along with the method
definitions that are required, and an example of the operator in use within an expression.

N.B. The use of other as a variable name is not mandatory, but is considered the norm.

Operator Method Expression

+ Addition [index] Index add (self, other)
- Subtraction operator sub (self, other)
* Multiplication in In operator mul (self, other)
@ Matrix matmul (self, other)
Multiplication
div (self, other)
/ Division
truediv (self, other)
/ Division
floordiv (self, other)
// Floor Division
mod (self, other)
%
Modulo/Remainder pow (self, other[, modulo])
** Power lshift (self, other)
<< Bitwise Left Shift rshift (self, other)
>> Bitwise Right and (self, other)
Shift xor (self, other)
& Bitwise AND
or (self, other)
^ Bitwise XOR
neg (self)
| (Bitwise OR)
pos (self)
- Negation
(Arithmetic) invert (self)
+ Positive lt (self, other)

~ Bitwise NOT le (self, other)

< Less than
eq (self, other)
<= Less than or
Equal to ne (self, other)

== Equal to gt (self, other)

!= Not Equal to
ge (self, other)
> Greater than
getitem (self, index)
>= Greater than or
Equal to contains (self, other)
 Page 207 of 343

a a1 != a2
a1 + a2
1 a1 >
a1 - a2 a2
a1 >=
a1 * a2
< a2
(Python 3.5) a1 @
a2
< a1[ind
a1 / a2
(Python 2 only)
(Python 3) ex]
a1 / a2a2

a1 // a2 a in a1

a1 % 2
a2
a a1 >> a2
1 a1 &
a2
a1 ^
a2
* a1 |
a2
* -a1
+a1
a ~a1

2 a1 <
a2
a1 <= a2
a1 == a2
(*args, ...) call (self, *args, **kwargs) a1(*args, **kwargs)
Calling

The optional parameter modulo for pow is only used by the pow built-in function.

Each of the methods corresponding to a binary operator has a corresponding "right"

method which start with
r, for example
radd :
 Page 208 of 343

as well as a corresponding inplace version, starting with i:

Since there's nothing special about these methods, many other parts of the language, parts
of the standard library, and even third-party modules add magic methods on their own, like
methods to cast an object to a type or checking properties of the object. For example, the
builtin str() function calls the object's str method, if it exists.
Some of these uses are listed below.

Function Method Expression

Casting to int (self) int(a1)
int Absolute abs (self) abs(a1)
function str (self) str(a1)
Casting to str unicode (self) unicode(a1) (Python 2 only)

Casting to
unicode
String representation repr (self) repr(a1)
Casting to Ceiling

bool String
formatting
Hashing
Length
Revers
ed
Floor
 Page 209 of 343

nonzero (self) bool(a1)

format
(self,
formatstr)
"Hi
{:abc}".f
ormat(a1)
hash
(self) hash(a1)
len
(self) len(a1)
reversed (self) reversed(a1)
floor (self) [Link](a1)
ceil (self) [Link](a1)
 Page 210 of 343

There are also the special methods enter and exit for context managers, and many
more.
Hidden Features
Operator Overloading
Everything in Python is an object. Each object has some special internal methods which it
uses to interact with other objects. Generally, these methods follow the action
naming convention. Collectively, this is termed as the
Python Data Model
You can overload any of these methods. This is commonly used in operator overloading in
Python. Below is an example of operator overloading using Python's data model. The
Vector class creates a simple vector of two variables. We'll add appropriate support for
mathematical operations of two vectors using operator overloading.

class Vector(object):
def init
(self, x,
y): self.x =
x
self.y = y

def add (self, v):

# Addition with another vector.
return Vector(self.x + v.x, self.y + v.y)

def sub (self, v):

# Subtraction with another vector.
return Vector(self.x - v.x, self.y - v.y)

def mul (self, s):

# Multiplication with a scalar.
return Vector(self.x * s, self.y * s)

def div (self, s):

# Division with a scalar.
float_s = float(s)
return Vector(self.x / float_s, self.y / float_s)

def floordiv (self, s):

# Division with a scalar (value floored).
return Vector(self.x // s, self.y // s)
 Page 211 of 343

def repr (self):

# Print friendly representation of Vector
class. Else, it would # show up like, <
main .Vector instance at 0x01DDDDC8>.
return '<Vector (%f, %f)>' % (self.x,
self.y, )

a =
Vector(3,
5) b =
Vector(2,
7)

print a + b # Output: <Vector (5.000000,

12.000000)> print b - a # Output:
<Vector (-1.000000, 2.000000)> print
b * 1.3 # Output: <Vector (2.600000,
9.100000)>
print a // 17 # Output: <Vector (0.000000, 0.000000)>
print a / 17 # Output: <Vector (0.176471, 0.294118)>

The above example demonstrates overloading of basic numeric operators. A comprehensive

list can be found here.
Properties
Python classes support properties, which look like regular object variables, but with the
possibility of attaching custom behavior and documentation.
 Page 212 of 343

The objects of class MyClass will appear to have a property .string, however it's behavior is
now tightly controlled:

As well as the useful syntax as above, the property syntax allows for validation, or other
augmentations to be added to those attributes. This could be especially useful with public
APIs - where a level of help should be given to the user.

Another common use of properties is to enable the class to present 'virtual attributes' -
attributes which aren't actually stored but are computed only when requested.

# Make name read only by not providing a set method

@property
def name(self):
return [Link]

def take_damage(self,
damage): [Link] -=
damage
[Link] = 0 if [Link] <0 else [Link]

@property
def is_alive(self):
return [Link] != 0

@property
def is_wounded(self):
return [Link] < self.max_hp if [Link] > 0 else False
 Page 213 of 343

@property
def is_dead(self):
return not self.is_alive

bilbo = Character('Bilbo
Baggins', 100) [Link]
# out : 100
[Link] = 200
# out : AttributeError: can't
set attribute # hp attribute is
read only.

bilbo.is_ali
ve
# out :
True
bilbo.is_wou
nded # out :
False
bilbo.is_dea
d
# out : False
bilbo.take_da 5 )
mage( 0
[Link]
# out : 50
bilbo.is_ali
ve
# out :
True
bilbo.is_wou
nded # out :
True
bilbo.is_dea
d
# out : False
bilbo.take_da 5 )
mage( 0
[Link]
# out : 0
 Page 214 of 343

bilbo.is_ali
ve
# out :
False
bilbo.is_wou
nded # out :
False
bilbo.is_dea
d
# out : True

Property Objects
Using the @property decorator for read-writeproperties
If you want to use @property to implement custom behavior for setting and getting, use this
pattern:

To use this:

Using the @property decorator

The @property decorator can be used to define methods in a class which act like
attributes. One example where this can be useful is when exposing information which may
require an initial (expensive) lookup and simple retrieval thereafter.
 Page 215 of 343

Given some module [Link]:

Then

Overriding just a getter, setter or a deleter of aproperty object

When you inherit from a class with a property, you can provide a new implementation for
one or more of the property getter, setter or deleter functions, by referencing the
property object on the parent class:

You can also add a setter or deleter where there was not one on the base class before.
Using properties without decorators
While using decorator syntax (with the @) is convenient, it also a bit concealing. You can
use properties directly, without decorators. The following Python 3.x example shows this:
 Page 216 of 343

x, y, y2 = property (getX, setX), property (getY,

setY), property (getY2, setY2) t = property (getT, setT)
u = property (getU, setU)

A.q = 5678

class B:
def getZ (self):
return self.z_

def setZ (self,

value):
self.z_ =
value

z = property (getZ, setZ)

class C:
def init
(self):
[Link] =
 Page 217 of 343

1234

def getW (self):

return self.w_ + [Link]

def setW (self, value):

self.w_ = value -

[Link] w = property

(getW, setW)

a1 = A ()
a2 = A ()

a1.y2 = 1000
a2.y2 = 2000

a1.x = 5
a1.y = 6

a2.x = 7
a2.y = 8

a1.t = 77
a1.u = 88

print (a1.x, a1.y, a1.y2)

print (a2.x, a2.y, a2.y2)
print (a1.p, a2.p, a1.q, a2.q)
print (a1.t,
a1.u) b = B
()
c = C ()

b.z = 100100
c.z = 200200
c.w = 300300

print (a1.x, b.z, c.z, c.w)

c.w = 400400
c.z = 500500
b.z = 600600
 Page 218 of 343

Setters, Getters & Properties

For the sake of data encapsulation, sometimes you want to have an attribute which value
comes from other attributes or, in general, which value shall be computed at the moment.
The standard way to deal with this situation is to create a method, called getter or a setter.

In the example above, it's easy to see what happens if we create a new Book that contains a
title and a author. If all books we're to add to our Library have authors and titles, then we
can skip the getters and setters and use the dot notation. However, suppose we have some
books that do not have an author and we want to set the author to "Unknown". Or if they
have multiple authors and we plan to return a list of authors.

In this case we can create a getter and a setter for the author attribute.

This scheme is not recommended.

One reason is that there is a catch: Let's assume we have designed our class with the
public attribute and no methods. People have already used it a lot and they have
written code like this:
 Page 219 of 343

Now we have a problem. Because author is not an attribute! Python offers a solution to
this problem called properties. A method to get properties is decorated with the
@property before it's header. The method that we want to function as a setter is decorated
with @[Link] before it.

Keeping this in mind, we now have our new updated class.

Note, normally Python doesn't allow you to have multiple methods with the same name and
different number of parameters. However, in this case Python allows this because of the
decorators used.

If we test the code:

Abstract Base Classes (abc)

Setting the ABCMeta metaclass
Abstract classes are classes that are meant to be inherited but avoid implementing specific
methods, leaving behind only method signatures that subclasses must implement.

Abstract classes are useful for defining and enforcing class abstractions at a high level,
similar to the concept of interfaces in typed languages, without the need for method
implementation.

One conceptual approach to defining an abstract class is to stub out the class methods, and
 Page 220 of 343

then raise a NotImplementedError if accessed. This prevents children classes from

accessing parent methods without overriding them first. Like so:

Creating an abstract class in this way prevents improper usage of methods that are not
overridden, and certainly encourages methods to be defined in child classes, but it does not
enforce their definition. With the abc module we can prevent child classes from being
instantiated when they fail to override abstract class methods of their parents and
ancestors:

It is now possible to simply subclass and override:

Why/How to use ABCMeta and@abstractmethod

Abstract base classes (ABCs) enforce what derived classes implement particular methods from
the base class.
To understand how this works and why we should use it, let's take a look at an example
that Van Rossum would enjoy. Let's say we have a Base class "MontyPython" with two
methods (joke & punchline) that must be implemented by all derived classes.
 Page 221 of 343

When we instantiate an object and call it's two methods, we'll get an error (as expected) with
the punchline() method.

However, this still allows us to instantiate an object of the ArgumentClinic class without
getting an error. In fact we don't get an error until we look for the punchline().

This is avoided by using the Abstract Base Class (ABC) module. Let's see how this works with
the same example:

ABCMeta

ABCMeta

This time when we try to instantiate an object from the incomplete class, we immediately get a
TypeError!
 Page 222 of 343

In this case, it's easy to complete the class to avoid any TypeErrors:

This time when you instantiate an object it works!

Message passing
Passing an object of one class to another method involves calling a method of the second class
and passing an instance of the first class as an argument to that method. This allows the second
class's method to operate on the first class's object, potentially modifying its state or invoking
its methods. Here's a simple example to illustrate this concept:
class Student:
def _init_(self, name, roll_number):
[Link] = name
self.roll_number = roll_number

def display_info(self):
print(f"Name: {[Link]}")
print(f"Roll Number: {self.roll_number}")

class School:
def admit_student(self, student):
print(f"Admitted student with roll number {student.roll_number}")

# Create a Student object

student = Student("Alice", 101)

# Create a School object

school = School()

# Pass the Student object to the School's admit_student method

school.admit_student(student)
 Page 223 of 343

# Call the Student's display_info method

student.display_info()

Exceptions
Errors detected during execution are called exceptions and are not unconditionally fatal.
Most exceptions are not handled by programs; it is possible to write programs that handle
selected exceptions. There are specific features in Python to deal with exceptions and
exception logic. Furthermore, exceptions have a rich type hierarchy, all inheriting from
the BaseException type.

Catching Exceptions
Use try...except: to catch exceptions. You should specify as precise an exception as you
can:

The exception class that is specified - in this case, ZeroDivisionError - catches any
exception that is of that class or of any subclass of that exception.

For example, ZeroDivisionError is a subclass of ArithmeticError:

And so, the following will still catch the ZeroDivisionError:

 Page 224 of 343

Do not catch everything!

While it's often tempting to catch every Exception:

Or even everything (that includes BaseException and all its children including Exception):

In most cases it's bad practice. It might catch more than intended, such as
SystemExit, KeyboardInterrupt and MemoryError - each of which should generally be
handled differently than usual system or logic errors. It also means there's no clear
understanding for what the internal code may do wrong and how to recover properly
from that condition. If you're catching every error, you won't know what error
occurred or how to fix it.

This is more commonly referred to as 'bug masking' and should be avoided. Let your
program crash instead of silently failing or even worse, failing at deeper level of execution.
(Imagine it's a transactional system)

Usually, these constructs are used at the very outer level of the program, and will log the
details of the error so that the bug can be fixed, or the error can be handled more
specifically.

Re-raising exceptions
Sometimes you want to catch an exception just to inspect it, e.g., for logging purposes.
After the inspection, you want the exception to continue propagating as it did before.

In this case, simply use the raise statement with no parameters.

 Page 225 of 343

Keep in mind, though, that someone further up in the caller stack can still catch the
exception and handle it somehow. The done output could be a nuisance in this case
because it will happen in any case (caught or not caught). So, it might be a better idea to
raise a different exception, containing your comment about the situation as well as the
original exception:

But this has the drawback of reducing the exception trace to exactly this raise while the
raise without argument retains the original exception trace.

In Python 3 you can keep the original stack by using the raise-from syntax:

Catching multiple exceptions

There are a few ways to catch multiple exceptions.

The first is by creating a tuple of the exception types you wish to catch and handle in the
same manner. This example will cause the code to ignore KeyError and AttributeError
exceptions.

If you wish to handle different exceptions in different ways, you can provide a separate
exception block for each type. In this example, we still catch the KeyError and
AttributeError, but handle the exceptions in different manners.
 Page 226 of 343

Else
Code in an else block will only be run if no exceptions were raised by the code in the try
block. This is useful if you have some code you don’t want to run if an exception is
thrown, but you don’t want exceptions thrown by that code to be caught.

For example:

Note that this kind of else: cannot be combined with an if starting the else-clause to an
elif. If you have a following if it needs to stay indented below that else:

Raising Exceptions
If your code encounters a condition, it doesn't know how to handle, such as an incorrect
parameter, it should raise the appropriate exception.
 Page 227 of 343

Creating custom exception types

Create a class inheriting from Exception:

or another exception type:

Practical examples of exception handling

Imagine you want a user to enter a number via input. You want to ensure that the input is a
number. You can use
try/except for

this: Python 3.x

Version ≥ 3.0

Note: Python 2.x would use raw_input instead; the function input exists in Python 2.x
but has different semantics. In the above example, input would also accept expressions
 Page 228 of 343

such as 2 + 2 which evaluate to a number.

If the input could not be converted to an integer, a ValueError is raised. You can catch
it with except. If no exception is raised, break jumps out of the loop. After the loop, nb
contains an integer.

Dictionaries

Imagine you are iterating over a list of consecutive integers, like range(n), and you have a list
of dictionaries d that contains information about things to do when you encounter some
particular integers, say skip the d[i] next ones.

A KeyError will be raised when you try to get a value from a dictionary for a key that doesn’t
exist.
Chain exceptions with raise from
In the process of handling an exception, you may want to raise another exception. For
example, if you get an IOError while reading from a file, you may want to raise an
application-specific error to present to the users of your library, instead.

Python 3.x Version ≥ 3.0

You can chain exceptions to show how the handling of exceptions proceeded:
 Page 229 of 343

Raise Custom Errors /Exceptions

Python has many built-in exceptions which force your program to output an error
when something in it goes wrong. However, sometimes you may need to create
custom exceptions that serve your purpose.
In Python, users can define such exceptions by creating a new class. This exception class
has to be derived, either directly or indirectly, from Exception class. Most of the built-in
exceptions are also derived from this class.

Custom Exception
Here, we have created a user-defined exception called CustomError which is derived from
the Exception class. This new exception can be raised, like other exceptions, using the raise
statement with an optional error message.

Output:

Traceback (most recent call last):

File "error_custom.py", line 8, in
raise CustomError('This is custom error')
main .CustomError: This is custom error

Catch custom Exception

This example shows how to catch custom Exception

Output:

229
 Page 230 of 343

Files & Folders I/O

Parameter Details
filename the path to your file or, if the file is in the working directory, the
filename of your file access_mode a string value that determines how the
file is opened
buffering an integer value used for optional line buffering

When it comes to storing, reading, or communicating data, working with the files of an
operating system is both necessary and easy with Python. Unlike other languages where
file input and output requires complex reading and writing objects, Python simplifies the
process only needing commands to open, read/write and close the file. This topic explains
how Python can interface with files on the operating system.

File modes
There are different modes you can open a file with, specified by the mode parameter. These
include:

'r' - reading mode. The default. It allows you only to read the file, not to modify it.
When using this mode the file must exist.

'w' - writing mode. It will create a new file if it does not exist, otherwise will erase
the file and allow you to write to it.

'a' - append mode. It will write data to the end of the file. It does not erase the file,
and the file must exist for this mode.

'rb' - reading mode in binary. This is similar to r except that the reading is forced
in binary mode. This is also a default choice.

'r+' - reading mode plus writing mode at the same time. This allows you to read
and write into files at the same time without having to use r and w.

'rb+' - reading and writing mode in binary. The same as r+ except the data is in binary
'wb' - writing mode in binary. The same as w except the data is in binary.
'w+' - writing and reading mode. The exact same as r+ but if the file does not exist, a
new one is made. Otherwise, the file is overwritten.

'wb+' - writing and reading mode in binary mode. The same as w+ but the data is in
binary.
'ab' - appending in binary mode. Similar to a except that the data is in binary.

230
 Page 231 of 343

'a+' - appending and reading mode. Similar to w+ as it will create a new file if the
file does not exist. Otherwise, the file pointer is at the end of the file if it exists.

'ab+' - appending and reading mode in binary. The same as a+ except that the data is in
binary.

r r+ w w+ a a+
Read ✔ ✔ ✘ ✔ ✘ ✔
Write ✘ ✔ ✔ ✔ ✔ ✔
Creates ✘ ✘ ✔ ✔ ✔ ✔
file
Erases file ✘ ✘ ✔ ✔ ✘ ✘
Initial position Start Start Start Start End End

Python 3 added a new mode for exclusive creation so that you will not accidentally
truncate or overwrite and existing file.
'x' - open for exclusive creation, will raise FileExistsError if the file already
exists
'xb' - open for exclusive creation writing mode in binary. The same as x except the data
is in binary.
'x+' - reading and writing mode. Similar to w+ as it will create a new file if the file
does not exist. Otherwise, will raise FileExistsError.
'xb+' - writing and reading mode. The exact same as x+ but the data is binary
x x+
Read ✘ ✔
Write ✔ ✔
Creates file ✔ ✔
Erases file ✘ ✘
Initial position Start
Start
Allow one to write your file open code in a more

pythonic manner.

231
 Page 232 of 343

Reading a file line-by-line

The simplest way to iterate over a file line-by-line:

readline() allows for more granular control over line-by-line iteration. The example below
is equivalent to the one above:

Using the for loop iterator and readline() together is considered bad practice.

More commonly, the readlines() method is used to store an iterable collection of the file's
lines:

This would print the following:

Line 0: hello

Line 1: world

Iterate files (recursively)

232
 Page 233 of 343

To iterate all files, including in sub directories, use [Link]:

root_dir can be "." to start from current directory, or any other

path to start from. Python 3.x Version ≥ 3.5

If you also wish to get information about the file, you may use the more efficient method
[Link] like so:

Getting the full contents of a file

The preferred method of file i/o is to use the with keyword. This will ensure the file
handle is closed once the reading or writing has been completed.

or, to handle closing the file manually, you can forgo with and simply call close yourself:

Keep in mind that without using a with statement, you might accidentally keep the file open in
case an unexpected exception arises like so:

Writing to a file

233
 Page 234 of 343

If you open [Link], you will see that its contents are:

Line 1Line 2Line 3Line 4

Python doesn't automatically add line breaks, you need to do that manually:

Line 1
Line 2
Line 3
Line 4

Do not use [Link] as a line terminator when writing files opened in text mode (the
default); use \n instead. If you want to specify an encoding, you simply add the
encoding parameter to the open function:

It is also possible to use the print statement to write to a file. The mechanics are different in
Python 2 vs Python 3, but the concept is the same in that you can take the output that
would have gone to the screen and send it to a file instead.

234
 Page 235 of 343

Context Managers (“with”Statement)

While Python's context managers are widely used, few understand the purpose behind their
use. These statements, commonly used with reading and writing files, assist the application
in conserving system memory and improve resource management by ensuring specific
resources are only in use for certain processes. This topic explains and demonstrates the
use of Python's context managers.
Introduction to context managers and the withstatement
A context manager is an object that is notified when a context (a block of code) starts and
ends. You commonly use one with the with statement. It takes care of the notifying.

For example, file objects are context managers. When a context ends, the file object is closed
automatically:

The above example is usually simplified by using the as keyword:

Anything that ends execution of the block causes the context manager's exit method to be
called. This includes exceptions, and can be useful when an error causes you to
prematurely exit from an open file or connection. Exiting a script without properly closing
files/connections is a bad idea, that may cause data loss or other problems. By using a
context manager you can ensure that precautions are always taken to prevent damage or
loss in this way. This feature was added in Python 2.5.

User defined Modules

User-defined modules typically refer to custom code modules or libraries created by a
programmer or developer to encapsulate specific functionality or features within a
software application. These modules are separate from the core language or framework
and serve to organize and modularize code for better maintainability, reusability, and
readability. Here's a general overview of user-defined modules:
235
 Page 236 of 343

Definition: User-defined modules are created by users (developers) to extend the

functionality of a programming language or framework. They are essentially collections of
functions, classes, or variables that can be reused across different parts of a program.

Modularity: One of the primary purposes of user-defined modules is to promote

modularity in code. By breaking down a large program into smaller, more manageable
pieces (modules), developers can work on different parts of the program independently
and collaborate more effectively.
Reuse: User-defined modules allow developers to reuse code across different projects or
within the same project. This saves time and effort, as you don't have to recreate the same
functionality every time you need it.

Encapsulation: Modules can encapsulate related functionality, which means that the
internal details of how a module works are hidden from the rest of the program. This helps
in creating a clear and well-defined interface for using the module.
Organization: Modules help in organizing code logically. You can group related functions
and classes together in a module, making it easier to locate and understand specific pieces
of code.

Namespacing: Modules often serve as a way to create namespaces for functions and
variables. This helps prevent naming conflicts between different parts of a program.

Examples
In Python, for example, you can create user-defined modules by creating a .py file
containing functions, classes, or variables. You can then import and use these modules in
other Python scripts. In JavaScript, modules can be created using export and import
statements. Other programming languages have their own mechanisms for defining and
using user-defined modules.
Creating a module
A module is an importable file containing definitions
and statements. A module can be created by creating a
.py file.

Functions in a module can be used by importing the module.

236
 Page 237 of 343

For modules that you have made, they will need to be in the same directory as the file that
you are importing them into. (However, you can also put them into the Python lib
directory with the pre-included modules, but should be avoided if possible.)

Modules can be imported by other modules.

Specific functions of a module can be imported.

Modules can be aliased.

A module can be stand-alone runnable script.

Run it!

If the module is inside a directory and needs to be detected by python, the directory should
contain a file named
init .py.

Here's a simple Python example of a user-defined module:

237
 Page 238 of 343

# [Link]

def add(a, b):

return a + b
def subtract(a, b):

return a - b

You can then use this module in another Python script like this:

# [Link]

import mymodule

result1 = [Link](5, 3)

result2 = [Link](10, 2)

print(result1) # Output: 8

print(result2) # Output: 8

In this example, [Link] is a user-defined module that provides add and subtract
functions, which are then imported and used in the [Link] script.

In Python, you can create user-defined modules by writing code in a separate .py file and
then import and use that module in other Python scripts. The special variable _name_
plays a crucial role in determining whether a Python script is being run as the main
program or being imported as a module. This allows you to write reusable code within
your module.

Here's a step-by-step guide to creating a user-defined module and using the _name_
variable:

Create a User-Defined Module:

Let's say you want to create a module called my_module.py with some functions and
variables.

# my_module.py

def greet(name):

return f"Hello, {name}!"

pi = 3.14159265359
238
 Page 239 of 343

if _name_ == "_main_":

print("This is the main program.")

In this example, there's a greet function and a variable pi defined in the module. The if
_name_ == "_main_": block is used to specify code that should only run when the module
is executed directly as the main program.

Import the User-Defined Module:

You can import your user-defined module in another Python script (e.g., [Link]):
# [Link]

import my_module

result = my_module.greet("Alice")

print(result)
print(f"The value of pi is approximately {my_module.pi}")

In this script, my_module is imported, and you can use the functions and variables defined
in my_module.
Running the Code:

When you run [Link], it imports my_module and executes the code within it. The output
will be:

Hello, Alice!
The value of pi is approximately 3.14159265359

Executing the User-Defined Module Directly:

If you run my_module.py directly (not imported by another script), the code within the if
_name_ == "_main_": block will execute. For example:

This is the main program.

The _name_ variable allows you to distinguish whether a Python script is the main
program or an imported module. When a script is run as the main program, _name_ is set
to "_main". When a script is imported as a module, __name_ is set to the name of the
module.

This pattern of using if _name_ == "_main_": allows you to write code that can be both
239
 Page 240 of 343

used as a module in other programs and run as a standalone program for testing or
demonstration purposes.

The name special

variable
The name special variable is used to check whether a file has been imported as a module
or not, and to identify a function, class, module object by their name attribute.

name == ' main '

The special variable name is not set by the user. It is mostly used to check whether
or not the module is being run by itself or run because an import was performed. To
avoid your module to run certain parts of its code when it gets imported, check if
name == ' main '.

Let module_1.py be just one line long:

And let's see what happens, depending on

[Link] Situation 1

[Link]

Running [Link] will print hello

Running [Link] will print hello
Situation
2
module2.
py

Running [Link] will print

nothing Running [Link] will
print hello
240
 Page 241 of 343

Creating Python packages

Introduction
Every package requires a [Link] file which describes
the package. Consider the following directory structure
for a simple package:

The init .py contains only the line def foo(): return 100.

The following [Link] will define the package:

virtualenv is great to test package installs without modifying your other Python environments:
241
 Page 242 of 343

Making package executable

If your package isn't only a library, but has a piece of code that can be used either as a
showcase or a standalone application when your package is installed, put that piece of
code into main .py file.

Put the main .py in the package_name folder. This way you will be able to run it directly
from console:

If there's no main .py file available, the package won't run with this command and
this error will be printed:
python: No module named package_name. main ; 'package_name' is a
package and cannot be directly executed.

Difference between Module and Package

Modules
A module is a single Python file that can be imported. Using a module looks like this:
[Link]

my_script.py

in an interpreter

242
 Page 243 of 343

Packages
A package is made up of multiple Python files (or modules), and can even include libraries
written in C or C++. Instead of being a single file, it is an entire folder structure which
might look like this:

Folder package

init
.py
[Link]
[Link]

init .py

[Link]

[Link]

All Python packages must contain an init .py file. When you import a package in
your script (import package), the init .py script will be run, giving you access to
the all of the functions in the package. In this case, it allows you to use the [Link]
and [Link] functions.

243
 Page 244 of 343

The os Module
Parameter Details
Path A path to a file. The path separator may be determined
by [Link]. Mode The desired permission, in octal (e.g.
0700)

This module provides a portable way of using operating system dependent functionality.

makedirs - recursive directory creation

Given a local directory with the following contents:

dir1
subdir1
subdir2

We want to create the same subdir1, subdir2 under a new directory dir2, which does not exist
yet.

Running this results in

dir1
subdir1
subdir2
dir2
subdir1
subdir2

dir2 is only created the first time it is needed, for subdir1's creation.

If we had used [Link] instead, we would have had an exception because dir2 would not have
existed yet.

[Link] won't like it if the target directory exists already. If we re-run it again:

244
 Page 245 of 343

However, this could easily be fixed by catching the exception and checking that the directory
has been created.

Create a directory

If you need to specify permissions, you can use the optional mode argument:

Get current directory

Use the [Link]() function:

Determine the name of the operating system

The os module provides an interface to determine what type of operating system the code is
currently running on.

This can return one of the following in Python 3:

pos
ix
nt
ce
245
 Page 246 of 343

jav
a

More detailed information can be retrieved from [Link]

Remove a directory
Remove the directory at path:

You should not use [Link]() to remove a directory. That function is for files and
using it on directories will result in an OSError

Follow a symlink (POSIX)

Sometimes you need to determine the target of a symlink. [Link] will do this:

Change permissions on a file

where mode is the desired permission, in octal.

The dis module

What is Python bytecode?
Python is a hybrid interpreter. When running a program, it first assembles it into bytecode
which can then be run in the Python interpreter (also called a Python virtual machine).
The dis module in the standard library can be used to make the Python bytecode human-
readable by disassembling classes, methods, functions, and code objects.

0 LOAD_CONST

LOAD_CONST
8 RETURN_VALUE

246
 Page 247 of 343

The Python interpreter is stack-based and uses a first-in last-out system.

Each operation code (opcode) in the Python assembly language (the bytecode) takes a
fixed number of items from the stack and returns a fixed number of items to the stack. If
there aren't enough items on the stack for an opcode, the Python interpreter will crash,
possibly without an error message.

Constants in the dis module

ROT_THREE

Regular Expressions (Regex)

Python makes regular expressions available through the re module.

Regular expressions are combinations of characters that are interpreted as rules for
matching substrings. For instance, the expression 'amount\D+\d+' will match any string
composed by the word amount plus an integral number, separated by one or more non-
digits, such as:amount=100, amount is 3, amount is equal to: 33, etc.

247
 Page 248 of 343

Matching the beginning of a string

The first argument of [Link]() is the regular expression, the second is the string to match:

You may notice that the pattern variable is a string prefixed with r, which indicates that the
string is a raw string literal.

A raw string literal has a slightly different syntax than a string literal, namely a backslash \
in a raw string literal means "just a backslash" and there's no need for doubling up
backlashes to escape "escape sequences" such as newlines (\n), tabs (\t), backspaces (\),
form-feeds (\r), and so on. In normal string literals, each backslash must be doubled up to
avoid being taken as the start of an escape sequence.

Hence, r"\n" is a string of 2 characters: \ and n. Regex patterns also use backslashes, e.g.
\d refers to any digit character. We can avoid having to double escape our strings ("\\d") by
using raw strings (r"\d").

For instance:

Matching is done from the start of the string only. If you want to match anywhere use
[Link] instead:

248
 Page 249 of 343

Searching

Searching is done anywhere in the string unlike [Link]. You can also use [Link].

You can also search at the beginning of the string (use ^),

at the end of the string (use $),

or both (use both ^ and $):

249
 Page 250 of 343

Precompiled patterns

Compiling a pattern allows it to be reused later on in a program. However, note that Python
caches recently-used
expressions (docs, SO answer), so "programs that use only a few regular expressions at a time
needn’t worry about compiling regular expressions".

It can be used with [Link]().

Flags
For some special cases we need to change the behavior of the Regular Expression, this is
done using flags. Flags can be set in two ways, through the flags keyword or directly in the
expression.

Flags keyword

Below an example for [Link] but it works for most functions in the re module.

250
 Page 251 of 343

m None

IGNORECASE

IGNORECASE m

IGNORECASE

Common Flags

Flag Short Description

[Link], re.I Makes the pattern ignore the case
[Link], re.S Makes . match everything including
newlines [Link], re.M Makes ^ match the begin
of a line and $ the end of a line [Link] Turns on debug
information

For the complete list of all available flags check the docs

Inline flags

From the docs:

(?iLmsux) (One or more letters from the set 'i', 'L', 'm', 's', 'u', 'x'.)
The group matches the empty string; the letters set the corresponding flags: re.I (ignore
case), re.L (locale dependent), re.M (multi-line), re.S (dot matches all), re.U (Unicode
dependent), and re.X (verbose), for the entire regular expression. This is useful if you
wish to include the flags as part of the regular expression, instead of passing a flag
argument to the [Link]() function.

Note that the (?x) flag changes how the expression is parsed. It should be used first
in the expression string, or after one or more whitespace characters. If there are
non-whitespace characters before the flag, the results are undefined.

251
 Page 252 of 343

Replacing
Replacements can be made on strings using [Link].

Replacing strings
Using group references

Replacements with a small number of groups can be made as follows:

However, if you make a group ID like '10', this doesn't work: \10 is read as 'ID number 1
followed by 0'. So you have to be more specific and use the \g<i> notation:

Using a replacement function

Find All Non-Overlapping Matches

Note that the r before "[0-9]{2,3}" tells python to interpret the string as-is; as a "raw"
string.

You could also use [Link]() which works in the same way as [Link]() but
returns an iterator with
SRE_Match objects instead of a list of strings:

252
 Page 253 of 343

Checking for allowed characters

If you want to check that a string contains only a certain set of characters, in this case a-z,
A-Z and 0-9, you can do so like this,

You can also adapt the expression line from [^a-zA-Z0-9.] to [^a-z0-9.], to disallow
uppercase letters for example.

Splitting a string using regular expressions

You can also use regular expressions to split a string. For example,

Grouping
Grouping is done with parentheses. Calling group() returns a string formed of the
matching parenthesized subgroups.

253
 Page 254 of 343

Arguments can also be provided to group() to fetch a

particular subgroup. From the docs:

If there is a single argument, the result is a single string; if there are multiple
arguments, the result is a tuple with one item per argument.

Calling groups() on the other hand, returns a list of tuples containing the subgroups.

Named groups

Creates a capture group that can be referenced by name as well as by index.

254
 Page 255 of 343

Non-capturing groups

Using (?:) creates a group, but the group isn't captured. This means you can use it as a
group, but it won't pollute your "group space".

This example matches 11+22 or 11, but not 11+. This is since the + sign and the second
term are grouped. On the other hand, the + sign isn't captured.

Escaping Special Characters

Special characters (like the character class brackets [ and ] below) are not matched literally:

By escaping the special characters, they can be matched literally:

The [Link]() function can be used to do this for you:

The [Link]() function escapes all special characters, so it is useful if you are composing
a regular expression based on user input:

255
 Page 256 of 343

Iterating over matches using `re.finditer`

You can use [Link] to iterate over all matches in a string. This gives you (in
comparison to [Link] extra information, such as information about the match location
in the string (indexes):

Result:

Match "an" found at: [5,7]

Match "an" found at:
[20,22] Match "ant" found
at: [23,26]

PDB MODULE
1. To import we simply use import pdb in our code.
2. For debugging, we will use pdb.set_trace() method. Now, in Python
3.7 breakpoint() method is also available for this.
3. We run this on Python idle terminal (you can use any ide terminal to run).
Let’s begin with a simple example consisting of some lines of code.
Example:

# importing pdb
import pdb

# make a simple function to debug

def fxn(n):
for i in range(n):
print("Hello! ", i+1)

256
 Page 257 of 343

# starting point to debug

pdb.set_trace()
fxn(5)
Output:

Here, we can see that when the function call is done then pdb executes and ask for the next
command. We can use some commands here like
c -> continue execution
q -> quit the debugger/execution
n -> step to next line within the same function
s -> step to next line in this function or a called function
To know more about different commands you can type help and get the required information.

Now, we will execute our program further with the help of the n command.

257
 Page 258 of 343

In a similar way, we can use the breakpoint() method (which doesn’t need to import pdb).

 Python3
# a simple function
def fxn(n):
for i in range(n):
print("Hello! ", i+1)

# using breakpoint
breakpoint()
fxn(5)
Output:

Features provided by PDB Debugging

1. Printing Variables or expressions
When utilizing the print order p, you’re passing an articulation to be assessed by Python. On the off
chance that you pass a variable name, pdb prints its present worth. Notwithstanding, you can do
considerably more to examine the condition of your running application.

258
 Page 259 of 343

An application of PDB debugging in the recursion to check variables

In this example, we will define a recursive function with pdb trace and check the values of variables
at each recursive call. To the print the value of variable, we will use a simple print keyword with the
variable name.

 Python3
# importing pdb
import pdb

# define recursive function

def rec_fxn(r):
if r > 0:

# set trace
pdb.set_trace()
rec_fxn(r//2)
else:
print("recursion stops")
return

# set trace at start

pdb.set_trace()
rec_fxn(5)

Output:

259
 Page 260 of 343

An Example to check expressions

This example is similar to the above example, that prints the values of expressions after their
evaluation.

 Python3
# importing pdb
import pdb

# simple function
def fxn(n):
l=[]
for i in range(n):
[Link](i)

# set trace
pdb.set_trace()
return

fxn(5)
Output:

2. Moving in code by steps

This is the most important feature provided by pdb. The two main commands are used for this which
are given below:
n -> step to next line within the same function
s -> step to next line in this function or a called function
Let’s understand the working of these with the help of an example.

 Python3
# importing pdb
import pdb

260
 Page 261 of 343

# simple function
def fxn(n):
l = []
for i in range(n):
[Link](i)
return

# set trace
pdb.set_trace()
fxn(5)
Output Using n:

261
 Page 262 of 343

Output Using s:

262
 Page 263 of 343

3. Using Breakpoints

This feature helps us to create breakpoints dynamically at a particular line in the source code. Here,
in this example, we are creating breakpoint using command b which given below:
b(reak) [ ([filename:]lineno | function) [, condition] ]
Without argument, list all breaks.
With a line number argument, set a break at this line in the current file. With a function name, set a
break at the first executable line of that function. If a second argument is present, it is a string
specifying an expression that must evaluate to true before the breakpoint is honored.
The line number may be prefixed with a filename and a colon, to specify a breakpoint in another file
(probably one that hasn’t been loaded yet). The file is searched for on [Link]; the .py suffix may be
omitted.

 Python3
# importing pdb
import pdb

# simple function
def fxn(n):
l = []

for i in range(n):
[Link](i)
print(l)
return

# set trace
pdb.set_trace()
fxn(5)

263
 Page 264 of 343

Output:

4. Execute code until the specified line

Use unt to proceed with execution like c, however, stop at the following line more noteworthy than
the current line. Now and then unt is more helpful and faster to utilize and is actually what you need.
unt(il) [lineno]
Without argument, continue execution until the line with a number greater than the current one is
reached. With a line number, continue execution until a line with a number greater or equal to that is
reached. In both cases, also stop when the current frame returns.

 Python3
# importing pdb
import pdb

# simple function
def fxn(n):

# set trace
pdb.set_trace()
l = []

for i in range(n):
[Link](i)
print(l)
return

264
 Page 265 of 343

fxn(5)
Output:

Garbage Collection
Reuse of primitive objects
An interesting thing to note which may help optimize your applications is that primitives
are actually also refcounted under the hood. Let's take a look at numbers; for all integers
between -5 and 256, Python always reuses the same object:

Note that the refcount increases, meaning that a and b reference the same underlying object
when they refer to the
1 primitive. However, for larger numbers, Python actually doesn't reuse the underlying object:

265
 Page 266 of 343

Because the refcount for 999999999 does not change when assigning it to a and b we can
infer that they refer to two different underlying objects, even though they both are
assigned the same primitive.

Reference Counting
The vast majority of Python memory management is handled with reference counting.

Every time an object is referenced (e.g. assigned to a variable), its reference count is
automatically increased. When it is dereferenced (e.g. variable goes out of scope), its
reference count is automatically decreased.

When the reference count reaches zero, the object is immediately destroyed and the
memory is immediately freed. Thus for the majority of cases, the garbage collector is not
even needed.

266
 Page 267 of 343

To demonstrate further the concept of references:

Multiprocessing
Running Two Simple Processes
A simple example of using multiple processes would be two processes (workers) that are
executed separately. In the following example, two processes are started:

countUp() counts 1 up, every second.

countDown() counts 1 down, every second.

267
 Page 268 of 343

The output is as follows:

Up: 0
Down: 3
Up: 1
Up: 2
Down: 2
Up: 3
Down: 1
Down: 0

Using Pool and Map

268
 Page 269 of 343

Pool is a class which manages multiple Workers (processes) behind the scenes and lets you, the
programmer, use.

Pool(5) creates a new Pool with 5 processes, and [Link] works just like map but it uses
multiple processes (the amount defined when creating the pool).

Similar results can be achieved using map_async, apply and apply_async which can be found
in the documentation.
Multithreading
Threads allow Python programs to handle multiple functions at once as opposed to
running a sequence of commands individually. This topic explains the principles behind
threading and demonstrates its usage.

Basics of multithreading
Using the threading module, a new thread of execution may be started by creating a new
[Link] and assigning it a function to execute:

The target parameter references the function (or callable object) to be run. The thread
will not begin execution until start is called on the Thread object.

Starting a Thread

Now that my_thread has run and terminated, calling start again will produce a
RuntimeError. If you'd like to run your thread as a daemon, passing the daemon=True
kwarg, or setting my_thread.daemon to True before calling start(), causes your Thread
to run silently in the background as a daemon.

Joining a Thread

In cases where you split up one big job into several small ones and want to run them
concurrently, but need to wait for all of them to finish before continuing, [Link]() is
the method you're looking for.
269
 Page 270 of 343

For example, let's say you want to download several pages of a website and compile them
into a single page. You'd do this:

A closer look at how join() works can be found here.

Create a Custom Thread Class

Using [Link] class we can subclass new custom Thread class. we must override
run method in a subclass.

270
 Page 271 of 343

Processes and Threads

Most programs are executed line by line, only running a single process at a time. Threads
allow multiple processes to flow independent of each other. Threading with multiple
processors permits programs to run multiple processes simultaneously. This topic
documents the implementation and usage of threads in Python.

Global Interpreter Lock

Python multithreading performance can often suffer due to the Global Interpreter
Lock. In short, even though you can have multiple threads in a Python program, only
one bytecode instruction can execute in parallel at any one time, regardless of the
number of CPUs.

As such, multithreading in cases where operations are blocked by external events - like
network access - can be quite effective:

Note that even though each process took 2 seconds to execute, the four processes together
were able to effectively run in parallel, taking 2 seconds total.

However, multithreading in cases where intensive computations are being done in Python code
- such as a lot of computation - does not result in much improvement, and can even be slower
271
 Page 272 of 343

than running in parallel:

272
 Page 273 of 343

In the latter case, multiprocessing can be effective as multiple processes can, of course,
execute multiple instructions simultaneously:

Running in Multiple Threads

Use [Link] to run a function in another thread.

273
 Page 274 of 343

Running in Multiple Processes

Use [Link] to run a function in another process. The interface is similar
to [Link]:

Sharing State Between Threads

As all threads are running in the same process, all threads have access to the same data.

However, concurrent access to shared data should be protected with a lock to avoid
synchronization issues.

274
 Page 275 of 343

Sharing State Between Processes

Code running in different processes do not, by default, share the same data. However, the
multiprocessingmodule contains primitives to help share values across multiple processes.

275
 Page 276 of 343

Python concurrency
The multiprocessing module

Here, each function is executed in a new process. Since a new instance of Python VM is
running the code, there is no GIL and you get parallelism running on multiple cores.
The [Link] method launches this new process and run the function passed in the
target argument with the arguments args. The [Link] method waits for the end of
the execution of processes p1 and p2.

The new processes are launched differently depending on the version of python and the
platform on which the code is running e.g.:

Windows uses spawn to create the new process.

With unix systems and version earlier than 3.3, the processes are created using a fork.
Note that this method does not respect the POSIX usage of fork and thus leads to
unexpected behaviors, especially when interacting with other multiprocessing
libraries.
With unix system and version 3.4+, you can choose to start the new processes with
either fork, forkserver or spawn using multiprocessing.set_start_method at the
beginning of your program. forkserver and spawn methods are slower than
forking but avoid some unexpected behaviors.

276
 Page 277 of 343

POSIX fork usage:

After a fork in a multithreaded program, the child can safely call only async-
signal-safe functions until such time as it calls execve.
(see)

Using fork, a new process will be launched with the exact same state for all the current mutex
but only the
MainThread will be launched. This is unsafe as it could lead to race conditions e.g.:

If you use a Lock in MainThread and pass it to another thread which is supposed to
lock it at some point. If the fork occurs simultaneously, the new process will start
with a locked lock which will never be released as the second thread does not exist
in this new process.
Actually, this kind of behavior should not occurred in pure python as multiprocessing
handles it properly but if you are interacting with other library, this kind of behavior can
occurs, leading to crash of your system (for instance with numpy/accelerated on macOS).

The threading module

In certain implementations of Python such as CPython, true parallelism is not achieved

using threads because of using what is known as the GIL, or Global Interpreter Lock.

Here is an excellent overview of Python concurrency:

Passing data between multiprocessing processes

Because data is sensitive when dealt with between two threads (think concurrent read and
concurrent write can conflict with one another, causing race conditions), a set of unique
objects were made in order to facilitate the passing of data back and forth between threads.
277
 Page 278 of 343

Any truly atomic operation can be used between threads, but it is always safe to stick with
Queue.

Most people will suggest that when using queue, to always place the queue data in a try:
except: block instead of using empty. However, for applications where it does not matter if you
skip a scan cycle (data can be placed in the queue while it is flipping states from
[Link]==True to [Link]==False) it is usually better to place read and write
access in what I call an Iftry block, because an 'if' statement is technically more performant
than catching the exception.

278
 Page 279 of 343

Parallel computation
Using the multiprocessing module to parallelisetasks

As the execution of each call to fib happens in parallel, the time of execution of the full
example is 1.8× faster than if done in a sequential way on a dual processor.

Python 2.2+

Using a C-extension to parallelize tasks

The idea here is to move the computationally intensive jobs to C (using special macros),
independent of Python, and have the C code release the GIL while it's working.

Py_BEGIN_ALLOW_THREAD
S

279
 Page 280 of 343

Using Parent and Children scripts to executecode in parallel

[Link]

[Link]

This is useful for parallel, independent HTTP request/response tasks or Database

select/inserts. Command line arguments can be given to the [Link] script as well.
Synchronization between scripts can be achieved by all scripts regularly checking a
separate server (like a Redis instance).

Using PyPar module to parallelize

PyPar is a library that uses the message passing interface (MPI) to provide parallelism in
Python. A simple example in PyPar (as seen at [Link]
looks like this:

280
 Page 281 of 343

Accessing MySQL database using MySQLdb

The first thing you need to do is create a connection to the database using the connect
method. After that, you will need a cursor that will operate with that connection.

Use the execute method of the cursor to interact with the database, and every once in a
while, commit the changes using the commit method of the connection object.

Once everything is done, don't forget to close the cursor and the connection.

Here is a Dbconnect class with everything you'll need.

281
 Page 282 of 343

Interacting with the database is simple. After creating the object, just use the execute method.

If you want to call a stored procedure, use the following syntax. Note that the parameters list is
optional.

After the query is done, you can access the results multiple ways. The cursor object is a
generator that can fetch all the results or be looped.

If you want a loop using directly the generator:

If you want to commit changes to the database:

If you want to close the cursor and the connection:

Connection
Creating a connection
According to PEP 249, the connection to a database should be established using a
connect() constructor, which returns a Connection object. The arguments for this
constructor are database dependent. Refer to the database specific topics for the
relevant arguments.

This connection object has four methods:

282
 Page 283 of 343

1: close

Closes the connection instantly. Note that the connection is automatically closed if the
Connection. del
method is called. Any pending transactions will implicitly be rolled back.

2: commit

Commits any pending transaction to the database.

3: rollback

Rolls back to the start of any pending transaction. In other words: this cancels any non-
committed transaction to the database.

4: cursor

Returns a Cursor object. This is used to do transactions on the database.

Reading and Writing CSV
Using pandas
Write a CSV file from a dict or a DataFrame.

Read a CSV file as a DataFrame and convert it to a dict:

283
 Page 284 of 343

Writing a TSV file

Python

Output file

Data Visualization withPython

Matplotlib
Matplotlib is a mathematical plotting library for Python that provides a variety of different
plotting functionality.

The matplotlib documentation can be found here, with the SO Docs being available here.

Matplotlib provides two distinct methods for plotting, though they are interchangeable for the
most part:

Firstly, matplotlib provides the pyplot interface, direct and simple-to-use interface
that allows plotting of complex graphs in a MATLAB-like style.
Secondly, matplotlib allows the user to control the different aspects (axes, lines,
ticks, etc) directly using an object-based system. This is more difficult but allows
complete control over the entire plot.

Below is an example of using the pyplot interface to plot some generated data:

284
 Page 285 of 343

Note that [Link]() is known to be problematic in some environments due to running

[Link] in interactive mode, and if so, the blocking behaviour can be overridden
explicitly by passing in an optional argument, [Link](block=True), to alleviate the issue.
285
 Page 286 of 343

Plotly
Plotly is a modern platform for plotting and data visualization. Useful for producing a
variety of plots, especially for data sciences, Plotly is available as a library for Python, R,
JavaScript, Julia and, MATLAB. It can also be used as a web application with these
languages.

Users can install plotly library and use it offline after user authentication. The installation
of this library and offline authentication is given here. Also, the plots can be made in
Jupyter Notebooks as well.

Usage of this library requires an account with username and password. This gives the
workspace to save plots and data on the cloud.

The free version of the library has some slightly limited features and designed for
making 250 plots per day. The paid version has all the features, unlimited plot
downloads and more private data storage. For more details, one can visit the main page
here.
For documentation and examples, one can go here

A sample plot from the documentation examples:

286
 Page 287 of 343

N = 100

287
 Page 288 of 343

The Interpreter (CommandLine Console)

Getting general help
If the help function is called in the console without any arguments, Python presents an
interactive help console, where you can find out about Python modules, symbols,
keywords and more.

Referring to the last expression

To get the value of the last result from your last expression in the console, use an underscore _.

This magic underscore value is only updated when using a python expression that results
in a value. Defining functions or for loops does not change the value. If the expression
raises an exception there will be no changes to
_.

288
 Page 289 of 343

Remember, this magic variable is only available in the interactive python interpreter.
Running scripts will not do this.

Sockets
Parameter Description
socket.AF_UNIX UNIX Socket
socket.AF_INET IPv4
socket.AF_INET6
IPv
6
socket.SOCK_STREA
M TCP
socket.SOCK_DGRA
M UDP

Many programming languages use sockets to communicate across processes or between

devices. This topic explains proper usage the sockets module in Python to facilitate sending
and receiving data over common networking protocols.

Raw Sockets on Linux

First you disable your network card's automatic checksumming:

Then send your packet, using a SOCK_RAW socket:

289
 Page 290 of 343

Sending data via UDP

UDP is a connectionless protocol. Messages to other processes or computers are sent
without establishing any sort of connection. There is no automatic confirmation if your
message has been received. UDP is usually used in latency sensitive applications or in
applications sending network wide broadcasts.

The following code sends a message to a process listening on localhost port 6667 using UDP

Note that there is no need to "close" the socket after the send, because UDP is connectionless.

Receiving data via UDP

UDP is a connectionless protocol. This means that peers sending messages do not require
establishing a connection before sending messages. [Link] returns a tuple
(msg [the message the socket received], addr [the address of the sender])

A UDP server using solely the socket module:

Below is an alternative implementation using [Link]:

290
 Page 291 of 343

By default, sockets block. This means that execution of the script will wait until the socket
receives data.

Sending data via TCP

Sending data over the internet is made possible using multiple modules. The sockets
module provides low-level access to the underlying Operating System operations
responsible for sending or receiving data from other computers or processes.

The following code sends the byte string b'Hello' to a TCP server listening on port 6667
on the host localhost and closes the connection when finished:

Socket output is blocking by default, that means that the program will wait in the connect
and send calls until the action is 'completed'. For connect that means the server actually
accepting the connection. For send it only means that the operating system has enough
buffer space to queue the data to be send later.

Sockets should always be closed after use.

Multi-threaded TCP Socket Server

When run with no arguments, this program starts a TCP socket server that listens for
connections to [Link] on
port 5000. The server handles each connection in a separate thread.

When run with the -c argument, this program connects to the server, reads the client list,
and prints it out. The client list is transferred as a JSON string. The client name may be
specified by passing the -n argument. By passing different names, the effect on the client
list may be observed.

client_list.py

import
argparse
import json
import
291
 Page 292 of 343

socket
import
threading

def handle_client(client_list,
conn, address): name =
[Link](1024)
entry = dict(zip(['name', 'address', 'port'], [name,
address[0], address[1]])) client_list[name] = entry
[Link]([Link](clien
t_list))
[Link](socket.SHUT_RD
WR) [Link]()

def server(client_list):
print "Starting server..."
s = [Link](socket.AF_INET,
socket.SOCK_STREAM) [Link](socket.SOL_SOCKET,
socket.SO_REUSEADDR, 1)
[Link](('[Link]', 5000))
[Link](5)
while True:
(conn, address) = [Link]()
t = [Link](target=handle_client,
args=(client_list, conn, address)) [Link] = True
[Link]()

def client(name):
s = [Link](socket.AF_INET,
socket.SOCK_STREAM) [Link](('[Link]', 5000))
[Link](name)
data =
[Link](1024) result
= [Link](data)
print [Link](result, indent=4)

def parse_arguments():
parser = [Link]()
parser.add_argument('-c', dest='client',
action='store_true') parser.add_argument('-n',
dest='name', type=str, default='name') result
= parser.parse_args()
return result

292
 Page 293 of 343

def main():
client_list =
dict() args =
parse_arguments()
if [Link]:
client([Link])
else:
try:
server(client_list)
except KeyboardInterrupt:
print "Keyboard interrupt"

if name == ' main ':

Server Output

Client Output

The receive buffers are limited to 1024 bytes. If the JSON string representation of the client
list exceeds this size, it will be truncated. This will cause the following exception to be
raised:

Websockets
Simple Echo with aiohttp
aiohttp provides asynchronous

websockets. Python 3.x Version ≥ 3.5

293
 Page 294 of 343

Wrapper Class with aiohttp

[Link] may be used as a parent for a custom

WebSocket class. Python 3.x Version ≥ 3.5

294
 Page 295 of 343

Using Autobahn as a Websocket Factory

The Autobahn package can be used for Python web socket

server factories. Python Autobahn package documentation

To install, typically one would simply use the terminal command

(For Linux):

(For Windows):

Then, a simple echo server can be created in a Python script:

295
 Page 296 of 343

the payload contains binary data.

I typically elsewise assume that
the payload is a string.
In this example, the payload is returned to sender verbatim.'''
[Link](payload,isBinary)
if name ==' main ':
try:
importasyncio
except ImportError:
'''Trollius = 0.3 was renamed'''
import trollius as asyncio
from [Link]
factory=WebSocketServerFactory()
'''Initialize the websocket factory, and set the
protocol to the above defined protocol(the class
that inherits from
[Link]
Protocol)''' [Link]=MyServerProtocol
'''This above line can be thought of as "binding" the methods
onConnect, onMessage, et-c that were described in the
MyServerProtocol class to the server, setting the servers
functionality, ie, protocol'''
loop=asyncio.get_event_loop()
coro=loop.create_server(factory,'[Link]',9000)
server=loop.run_until_complete(coro)
'''Run the server in an infinite loop'''
try:
loop.run_forever()
except KeyboardInterrupt:
pa
ss
final
ly:
[Link]
se()
[Link]
e()

In this example, a server is being created on the localhost ([Link]) on port 9000. This is
the listening IP and port. This is important information, as using this, you could identify
your computer's LAN address and port forward from your modem, though whatever
routers you have to the computer. Then, using google to investigate your WAN IP, you
could design your website to send WebSocket messages to your WAN IP, on port 9000 (in
296
 Page 297 of 343

this example).

It is important that you port forward from your modem back, meaning that if you have
routers daisy chained to the modem, enter into the modem's configuration settings, port
forward from the modem to the connected router, and so forth until the final router your
computer is connected to is having the information being received on modem port
9000 (in this example) forwarded to it.

Sockets And Message Encryption/Decryption Between Client andServer

Cryptography is used for security purposes. There are not so many examples of
Encryption/Decryption in Python using IDEA encryption MODE CTR. Aim of this
documentation :

Extend and implement of the RSA Digital Signature scheme in station-to-station

communication. Using Hashing for integrity of message, that is SHA-1. Produce simple
Key Transport protocol. Encrypt Key with IDEA encryption. Mode of Block Cipher is
Counter Mode

Server side Implementation

import
socket
import
hashlib
import os
import
time
import
itertools
import
threading
import sys
import
[Link] as
AES from
[Link] import
RSA from
[Link] import
IDEA

#server address and port number input from admin

host= raw_input("Server
297
 Page 298 of 343

Address - > ") port =

int(input("Port - > "))
#boolean for checking
server and port check =
False
done = False

def animate():
for c in [Link](['....','.......','..........','. ']):
if done:
break
[Link]('\rCHECKING IP ADDRESS AND
NOT USED PORT '+c) [Link]()
[Link](0.1)
[Link]('\r -----SERVER STARTED. WAITING FOR CLIENT \n')
try:
#setting up socket
server =
[Link](socket.AF_INET,socket.SOCK_STREAM)
[Link]((host,port))
[Link](5
) check =
True
except BaseException:
print "-----Check Server Address or Port "
check = False

if check is True:
# server Quit
shutdown = False
# printing "Server Started
Message" thread_load =
[Link](target=animate)
thread_load.start()

[Link](4)
done = True
#binding client and address
client,address = [Link]()
print ("CLIENT IS CONNECTED. CLIENT'S ADDRESS ->",address)
print ("\n-----WAITING FOR PUBLIC KEY & PUBLIC KEY HASH \n")

#client's message(Public Key)

getpbk = [Link](2048)
298
 Page 299 of 343

#conversion of string to KEY

server_public_key = [Link](getpbk)

#hashing the public key in server side for validating the hash from client
hash_object =
hashlib.sha1(getpbk)
hex_digest =
hash_object.hexdigest()

if getpbk != "":
print
(getpbk)
[Link]("
YES")
gethash = [Link](1024)
print ("\n-----HASH OF PUBLIC KEY \n"+gethash)
if hex_digest == gethash:
# creating session
key key_128 =
[Link](16)
#encrypt CTR MODE
session key
en = [Link](key_128,AES.MODE_CTR,counter =
lambda:key_128) encrypto = [Link](key_128)
#hashing sha1
en_object =
hashlib.sha1(encrypto)
en_digest =
en_object.hexdigest()

print ("\n-----SESSION KEY \n"+en_digest)

#encrypting session key and public key

E = server_public_key.encrypt(encrypto,16)
print ("\n-----ENCRYPTED PUBLIC KEY AND SESSION KEY
-------------------------------------------------------- \n"+str(E))
print ("\n-----HANDSHAKE COMPLETE ")
[Link](str(E))
while True:
#message from client
newmess = [Link](1024)
#decoding the message from HEXADECIMAL to decrypt the encrypted
version of the message only
299
 Page 300 of 343

decoded = [Link]("hex")
#making en_digest(session_key) as the key
key = en_digest[:16]
print ("\nENCRYPTED MESSAGE FROM CLIENT -> "+newmess)
#decrypting message from the client
ideaDecrypt = [Link](key, IDEA.MODE_CTR,
counter=lambda: key) dMsg =
[Link](decoded)
print ("\n**New Message**
"+[Link]([Link]()) +" >
"+dMsg+"\n") mess = raw_input("\nMessage To
Client -> ")
if mess != "":
ideaEncrypt = [Link](key, IDEA.MODE_CTR,
counter=lambda : key) eMsg =
[Link](mess)
eMsg = [Link]("hex").upper()
if eMsg != "":
print ("ENCRYPTED MESSAGE TO CLIENT-> " + eMsg)
[Link](eMsg)
[Link]()
else:
print ("\n-----PUBLIC KEY HASH DOESNOT MATCH \n")
Client side Implementation
import
time
import
socket
import
threading
import
hashlib
import
itertools
import sys
from Crypto import Random
from [Link] import RSA
from [Link] import IDEA

#animating loading
done = False
def animate():
for c in [Link](['....','.......','..........','. ']):
300
 Page 301 of 343

if done:
break
[Link]('\rCONFIRMING CONNECTION TO
SERVER '+c) [Link]()
[Link](0.1)

#public key and private key

random_generator = [Link]().read
key =
[Link](1024,random_gen
erator) public =
[Link]().exportKey()
private = [Link]()

#hashing the public key

hash_object =
hashlib.sha1(public) hex_digest
= hash_object.hexdigest()

#Setting up socket
server = [Link](socket.AF_INET,socket.SOCK_STREAM)

#host and port input user

host = raw_input("Server Address To Be
Connected -> ") port = int(input("Port of
The Server -> "))
#binding the address and port
[Link]((host, port))
# printing "Server Started
Message" thread_load =
[Link](target=animate)
thread_load.start()

[Link]
p(4)
done =
True

def send(t,name,key):
mess = raw_input(name
+ " : ") key = key[:16]
#merging the message and the name
whole = name+" : "+mess
ideaEncrypt = [Link](key, IDEA.MODE_CTR,
301
 Page 302 of 343

counter=lambda : key) eMsg =

[Link](whole)
#converting the encrypted message to HEXADECIMAL to readable
eMsg = [Link]("hex").upper()
if eMsg != "":
print ("ENCRYPTED MESSAGE TO SERVER-> "+eMsg)
[Link](eMsg)
def recv(t,key):
newmess = [Link](1024)
print ("\nENCRYPTED MESSAGE FROM SERVER-> " + newmess)
key = key[:16]
decoded = [Link]("hex")
ideaDecrypt = [Link](key, IDEA.MODE_CTR,
counter=lambda: key) dMsg =
[Link](decoded)
print ("\n**New Message From Server** " + [Link]([Link]())
+ " : " + dMsg + "\n")

while True:
[Link](public)
confirm = [Link](1024)
if confirm == "YES":
[Link](hex_digest)

#connected msg
msg =
[Link](1024)
en = eval(msg)
decrypt = [Link](en)
# hashing sha1
en_object =
hashlib.sha1(decrypt)
en_digest =
en_object.hexdigest()

print ("\n-----ENCRYPTED PUBLIC KEY AND SESSION KEY FROM SERVER

------------------------------------------------------------------------ ")
print (msg)
print ("\n-----DECRYPTED SESSION KEY ")
print (en_digest)
print ("\n-----HANDSHAKE COMPLETE \n")
alais = raw_input("\nYour Name -> ")

while True:
302
 Page 303 of 343

thread_send = [Link](target=send,args=("------Sending
Message------
",alais,en_digest))
thread_recv = [Link](target=recv,args=("------Receiving
Message------
",en_digest))
thread_send.start()
thread_recv.start()

thread_send.j
oin()
thread_recv.
join()
[Link](0.
5)
[Link](60
)
[Link](
)

Python Networking
Creating a Simple Http Server
To share files or to host simple websites(http and javascript) in your local network, you can use
Python's builtin SimpleHTTPServer module. Python should be in your Path variable. Go to the
folder where your files are and type:

For python 2:

For python 3:

If port number is not given 8000 is the default port. So the output will be:

Serving HTTP on [Link] port 8000 ...

You can access to your files through any device connected to the local network by typing
303
 Page 304 of 343

[Link]

hostipaddress is your local IP address which probably starts

with 192.168.x.x. To finish the module simply press

ctrl+c.

Creating a TCP server

You can create a TCP server using the socketserver library. Here's a
simple echo server. Server side

Client side

socketserver makes it relatively easy to create simple TCP servers. However, you should
be aware that, by default, the servers are single threaded and can only serve one client at a
time. If you want to handle multiple clients, either instantiate a ThreadingTCPServer
instead.

304
 Page 305 of 343

Creating a UDP Server

A UDP server is easily created using the
socketserver library. a simple time server:

Testing:

Start Simple HttpServer in a thread and openthe browser

Useful if your program is outputting web pages along the way.

305
 Page 306 of 343

The simplest Python socket client-serverexample

Server side:

Client Side:

First run the [Link], and make sure the server is ready to listen/receive sth Then the
client send info to the server; After the server received sth, it terminates.
Python HTTP Server
Running a simple HTTP server

Python 2.x Version ≥ 2.3

Python 3.x Version ≥ 3.0

Running this command serves the files of the current directory at port 9000.

If no argument is provided as port number then server will run on

default port 8000. The -m flag will search [Link] for the

corresponding .py file to run as a module.

If you want to only serve on localhost you'll need to write a custom Python program such as:

306
 Page 307 of 343

Serving files
Assuming you have the following directory of files:

You can setup a web server to serve these files

as follows: Python 2.x Version ≥ 2.3

307
 Page 308 of 343

Python 3.x Version ≥ 3.0

The SocketServer module provides the classes and functionalities to setup a network server.

SocketServer's TCPServer class sets up a server using the TCP protocol. The constructor
accepts a tuple representing the address of the server (i.e. the IP address and port) and the
class that handles the server requests.

The SimpleHTTPRequestHandler class of the SimpleHTTPServer module allows the

files at the current directory to be served.

Save the script at the same directory

and run it. Run the HTTP Server :

Python 2.x Version ≥ 2.3

python -m

SimpleHTTPServer 8000

Python 3.x Version ≥ 3.0

python -m [Link] 8000

The '-m' flag will search '[Link]' for the corresponding '.py' file to run

308
 Page 309 of 343

as a module. Open localhost:8000 in the browser, it will give you the following:

Basic handling of GET, POST, PUT usingBaseHTTPRequestHandler

from [Link] import BaseHTTPRequestHandler, HTTPServer # python3

class HandleRequests(BaseHTTPRequestHandler):
def _set_headers(self):
self.send_response(200)
self.send_header('Content-type',
'text/html') self.end_headers()

def
do_GET(self)
:
self._set_hea
ders()
[Link]("received get request")

def do_POST(self):
'''Reads post request body'''
self._set_headers()
content_len =
int([Link]('content-length', 0))
post_body = [Link](content_len)
[Link]("received post
request:<br>{}".format(post_body))

def
do_PUT(se
lf):
self.do_P
OST()

host =
'' port
= 80
HTTPServer((host, port), HandleRequests).serve_forever()

Example output using curl:

309
 Page 310 of 343

Programmatic API of SimpleHTTPServer

What happens when we execute python -m SimpleHTTPServer 9000?

To answer this question we should understand the construct of SimpleHTTPServer

([Link] and
BaseHTTPServer([Link]

Firstly, Python invokes the SimpleHTTPServer module with 9000 as an argument. Now
observing the SimpleHTTPServer code,

The test function is invoked following request handlers and ServerClass. Now
[Link] is invoked

310
 Page 311 of 343

Hence here the port number, which the user passed as argument is parsed and is
bound to the host address. Further basic steps of socket programming with given port
and protocol is carried out. Finally socket server is initiated.

This is a basic overview of inheritance from SocketServer class to other classes:

+------------+
| BaseServer |
+------------+
|
v
+-----------+ +------------------+
| TCPServer |------->| UnixStreamServer |
+-----------+ +------------------+
|
v
+-----------+ +--------------------+
| UDPServer |------->| UnixDatagramServer |
+-----------+ +--------------------+

Flask
Flask is a Python micro web framework used to run major websites including Pinterest, Twilio,
and LinkedIn. This topic explains and demonstrates the variety of features Flask offers for
both front and back end web development.

Files and Templates

Instead of typing our HTML markup into the return statements, we can use the
render_template() function:

311
 Page 312 of 343

This will use our template file [Link]. To ensure our application can find this file
we must organize our directory in the following format:

Most importantly, references to these files in the HTML must look like this:

<link rel="stylesheet" type="text/css", href="{{url_for('static',

filename='styles/about- [Link]')}}">

which will direct the application to look for [Link] in the styles folder under
the static folder. The same format of path applies to all references to images, styles,
scripts, or files.

The basics
The following example is an example of a basic server:

312
 Page 313 of 343

Running this script (with all the right dependencies installed) should start up a local
server. The host is [Link] commonly known as localhost. This server by default runs on
port 5000. To access your webserver, open a web browser and enter the URL
localhost:5000 or [Link]:5000 (no difference). Currently, only your computer can
access the webserver.

[Link]() has three parameters, host, port, and debug. The host is by default [Link], but
setting this to
[Link] will make your web server accessible from any device on your network using
your private IP address in the URL. the port is by default 5000 but if the parameter is set
to port 80, users will not need to specify a port number as browsers use port 80 by default.
As for the debug option, during the development process (never in production) it helps to
set this parameter to True, as your server will restart when changes made to your Flask
project.

Routing URLs
With Flask, URL routing is traditionally done using decorators. These decorators can be
used for static routing, as well as routing URLs with parameters. For the following
example, imagine this Flask script is running the website [Link].

With that last route, you can see that given a URL with /users/ and the profile name, we
could return a profile. Since it would be horribly inefficient and messy to include a
@[Link]() for every user, Flask offers to take parameters from the URL:
313
 Page 314 of 343

HTTP Methods
The two most common HTTP methods are GET and POST. Flask can run different code
from the same URL dependent on the HTTP method used. For example, in a web service
with accounts, it is most convenient to route the sign in page and the sign in process
through the same URL. A GET request, the same that is made when you open a URL in
your browser should show the login form, while a POST request (carrying login data)
should be processed separately. A route is also created to handle the DELETE and PUT
HTTP method.

To simplify the code a bit, we can import the request package from flask.

To retrieve data from the POST request, we must use the request package:

314
 Page 315 of 343

Jinja Templating
Similar to [Link], Flask integrates well with front end templating services. Flask uses by
default Jinja Templating. Templates allow small snippets of code to be used in the HTML file
such as conditionals or loops.
When we render a template, any parameters beyond the template file name are passed into
the HTML templating service. The following route will pass the username and joined date
(from a function somewhere else) into the HTML.

315
 Page 316 of 343

When this template is rendered, it can use the variables passed to it from the
render_template() function. Here are the contents of [Link]:

The following delimiters are used for different interpretations:

{% ... %} denotes a statement

{{ ... }} denotes an expression where a template is outputted
{# ... #} denotes a comment (not included in template output)
{# ... ## implies the rest of the line should be interpreted as a statement
The Request Object
The request object provides information on the request that was made to the route. To
utilize this object, it must be imported from the flask module:

URL Parameters

In previous examples [Link] and [Link] were used, however we can also
use the [Link]
property to retrieve a dictionary of the keys/values in the URL parameters.

To correctly authenticate in this context, the following URL would be needed (replacing
the username with any username:

[Link]/api/users/guido-van-rossum?key=pa55w0Rd

316
 Page 317 of 343

File Uploads

If a file upload was part of the submitted form in a POST request, the files can be handled
using the request object:

Cookies

The request may also include cookies in a dictionary similar to the URL parameters.

Introduction to RabbitMQusing AMQPStorm

How to consume messages from RabbitMQ
Start with importing the library.

When consuming messages, we first need to define a function to handle the incoming
messages. This can be any callable function, and has to take a message object, or a
message tuple (depending on the to_tuple parameter defined in start_consuming).

Besides processing the data from the incoming message, we will also have to Acknowledge
or Reject the message. This is important, as we need to let RabbitMQ know that we
properly received and processed the message.

317
 Page 318 of 343

Next we need to set up the connection to the RabbitMQ server.

After that we need to set up a channel. Each connection can have multiple
channels, and in general when performing multi-threaded tasks, it's
recommended (but not required) to have one per thread.

Once we have our channel set up, we need to let RabbitMQ know that we want to start
consuming messages. In this case we will use our previously defined on_message function
to handle all our consumed messages.

The queue we will be listening to on the RabbitMQ server is going to be simple_queue,

and we are also telling RabbitMQ that we will be acknowledging all incoming messages
once we are done with them.

Finally we need to start the IO loop to start processing messages delivered by the RabbitMQ
server.

318
 Page 319 of 343

How to publish messages to RabbitMQ

Start with importing the library.

Next we need to open a connection to the RabbitMQ server.

After that we need to set up a channel. Each connection can have multiple
channels, and in general when performing multi-threaded tasks, it's
recommended (but not required) to have one per thread.

Once we have our channel set up, we can start to prepare our message.

Now we can publish the message by simply calling publish and providing a routing_key. In
this case we are going to send the message to a queue called simple_queue.

How to create a delayed queue in RabbitMQ

First we need to set up two basic channels, one for the main queue, and one for the delay
queue. In my example at the end, I include a couple of additional flags that are not
required, but makes the code more reliable; such as confirm delivery, delivery_mode
and durable. You can find more information on these in the RabbitMQ manual.

After we have set up the channels we add a binding to the main channel that we can use to
send messages from the delay channel to our main queue.

319
 Page 320 of 343

Next we need to configure our delay channel to forward messages to the main queue once they
have expired.

x-message-ttl (Message - Time To Live)

This is normally used to automatically remove old messages in the queue after a specific
duration, but by
adding two optional arguments we can change this behaviour, and instead have this
parameter determine in milliseconds how long messages will stay in the delay queue.

x-dead-letter-routing-key

This variable allows us to transfer the message to a different queue once they have
expired, instead of the default behaviour of removing it completely.

x-dead-letter-exchange

This variable determines which Exchange used to transfer the message from hello_delay
to hello queue.

Publishing to the delay queue

When we are done setting up all the basic Pika parameters you simply send a message to
the delay queue using basic publish.

Once you have executed the script you should see the following queues created in your
RabbitMQ management module.

320
 Page 321 of 343

Example.

from amqpstorm import Connection

connection = Connection('[Link]', 'guest', 'guest')

# Create normal 'Hello World'

type channel. channel =
[Link]()
channel.confirm_deliveries()
[Link](queue='hello',
durable=True)

# We need to bind this channel to an exchange, that

will be used to transfer # messages from our delay
queue.
[Link](exchange='[Link]', routing_key='hello',
queue='hello')

# Create our delay channel.

delay_channel =
[Link]()
delay_channel.confirm_deliv
eries()

# This is where we declare the delay, and routing for our delay channel.
delay_channel.[Link](queue='hello_delay', durable=True,
arguments={
'x-message-ttl': 5000, # Delay until the message is transferred in
milliseconds.
'x-dead-letter-exchange': '[Link]', # Exchange used to transfer the
message from A to B.
'x-dead-letter-routing-key': 'hello' # Name of the queue we want the
message transferred to.

321
 Page 322 of 343

Descriptor
Simple descriptor
There are two different types of descriptors. Data descriptors are defined as objects that
define both a get () and a set ()
method, whereas non-data descriptors only define a get () method. This distinction
is important when considering overrides and the namespace of an instance's dictionary. If a
data descriptor and an entry in an instance's dictionary share the same name, the data
descriptor will take precedence. However, if instead a non-data descriptor and an entry in
an instance's dictionary share the same name, the instance dictionary's entry will take
precedence.

To make a read-only data descriptor, define both get() and set() with the set() raising an
AttributeError when called. Defining the set() method with an exception raising placeholder
is enough to make it a data descriptor.

322
 Page 323 of 343

An implemented example:

Two-way conversions
Descriptor objects can allow related object attributes to react to changes automatically.

Suppose we want to model an oscillator with a given frequency (in Hertz) and period (in
seconds). When we update the frequency we want the period to update, and when we
update the period we want the frequency to update:

323
 Page 324 of 343

We pick one of the values (frequency, in Hertz) as the "anchor," i.e. the one that can be set
with no conversion, and write a descriptor class for it:

The "other" value (period, in seconds) is defined in terms of the anchor. We write a descriptor
class that does our conversions:

Now we can write the Oscillator class:

324
 Page 325 of 343

tempfileNamedTemporaryFile
param description
mode mode to open file,
default=w+b delete To delete file
on closure, default=True suffix
filename suffix, default=''
prefix filename prefix, default='tmp'
dir dirname to place tempfile,
default=None buffsize default=-1,
(operating system default used)

Create (and write to a) known, persistenttemporary file

You can create temporary files which has a visible name on the file system which can be
accessed via the name property. The file can, on unix systems, be configured to delete on
closure (set by delete param, default is True) or can be reopened later.

The following will create and open a named temporary file and write 'Hello World!' to
that file. The filepath of the temporary file can be accessed via name, in this example it is
saved to the variable path and printed for the user. The file is then re-opened after closing
the file and the contents of the tempfile are read and printed for the user.

Output:

325
 Page 326 of 343

Input, Subset and OutputExternal Data Files using Pandas

This section shows basic code for reading, sub-setting and writing external data files using
pandas.

Basic Code to Import, Subset and Write ExternalData Files Using Pandas
# Print the working directory
import os
print [Link]()
# C:\Python27\Scripts

# Set the working directory

[Link]('C:/Users/general1/Documents/simpl
e Python files') print [Link]()
# C:\Users\general1\Documents\simple Python files

# load pandas
import pandas as pd

# read a csv data file named 'small_dataset.csv' containing 4 lines and 3

variables
my_data =
pd.read_csv("small_dataset.csv")
my_data
# x y z
#0 1 2 3
#1 4 5 6
#2 7 8 9
# 3 1 12
1 1
0

my_data.shape # number of rows and

columns in data set # (4, 3)

my_data.shape[0] # number of
rows in data set # 4

my_data.shape[1] # number of
columns in data set # 3

# Python uses 0-based indexing. The first row or column

in a data set is located # at position 0. In R the first row
or column in a data set is located
326
 Page 327 of 343

# at position 1.

# Select the first two rows

my_data[0:2]
# x y z
#0 1 2 3
#1 4 5 6

# Select the second and third rows

my_data[
1:3] #
x y z
# 1 4
5 6
# 2 7 8 9

# Select the third row

my_data[2:3]
# x y z
#2 7 8 9

327
 Page 328 of 343

# x
# 0 1
# 1 4

# y z
#0 2 3
#1 5 6
#2 8 9

328
 Page 329 of 343

Logging
Introduction to Python Logging
This module defines functions and classes which implement a flexible event logging
system for applications and libraries.

The key benefit of having the logging API provided by a standard library module is that all
Python modules can participate in logging, so your application log can include your own
messages integrated with messages from third- party modules.

So, let's start:

Example Configuration Directly in Code

Output example:

2016-07-26 18:53:55,332 root DEBUG this is a debug test

Example Configuration via an INI File

Assuming the file is named logging_confi[Link]. More details for the file format are in the
logging configuration section of the logging tutorial.

329
 Page 330 of 343

[loggers]
keys=root

[handlers]
keys=stream_handler

[formatters]
keys=formatter

[logger_root]
level=DEBUG
handlers=stream_handler

[handler_stream_handler]
class=StreamHandler
level=DEBUG
formatter=formatter
args=([Link],)

[formatter_formatter]

format=%(asctime)s %(name)-12s %(levelname)-8s %(message)s

Then use [Link]() in the code:

makes

Example Configuration via a Dictionary

As of Python 2.7, you can use a dictionary with configuration details. PEP 391 contains a
list of the mandatory and optional elements in the configuration dictionary.

330
 Page 331 of 343

makes

Logging exceptions
If you want to log exceptions you can and should make use of the [Link](msg)
method:

Do not pass the exception as argument:

As [Link](msg) expects a msg arg, it is a common pitfall to pass the
exception into the logging call like this:

331
 Page 332 of 343

While it might look as if this is the right thing to do at first, it is actually problematic due
to the reason how exceptions and various encoding work together in the logging
module:

Trying to log an exception that contains unicode chars, this way will fail miserably. It will
hide the stacktrace of the original exception by overriding it with a new one that is
raised during formatting of your [Link](e) call.

Obviously, in your own code, you might be aware of the encoding in exceptions.
However, 3rd party libs might handle this in a different way.

Correct Usage:

If instead of the exception you just pass a message and let python do its magic, it will work:

332
 Page 333 of 343

As you can see we don't actually use e in that case, the call to [Link](...)
magically formats the most recent exception.

Logging exceptions with non ERROR log level

If you want to log an exception with another log level than ERROR, you can use the
exc_info argument of the default loggers:

Accessing the exception's message

Be aware that libraries out there might throw exceptions with messages as any of unicode
or (utf-8 if you're lucky) byte-strings. If you really need to access an exception's text, the
only reliable way, that will always work, is to use repr(e) or the %r string formatting:

Mixins
Mixin
A Mixin is a set of properties and methods that can be used in different classes, which
don't come from a base class. In Object Oriented Programming languages, you typically
use inheritance to give objects of different classes the same functionality; if a set of objects
have some ability, you put that ability in a base class that both objects inherit from.

For instance, say you have the classes Car, Boat, and Plane. Objects from all of
these classes have the ability to travel, so they get the function travel. In this
scenario, they all travel the same basic way, too; by getting a route, and moving
along it. To implement this function, you could derive all of the classes from
Vehicle, and put the function in that shared class:

333
 Page 334 of 343

With this code, you can call travel on a car ([Link]("Montana")), boat
([Link]("Hawaii")), and plane ([Link]("France"))

However, what if you have functionality that's not available to a base class? Say, for
instance, you want to give Car a radio and the ability to use it to play a song on a radio
station, with play_song_on_station, but you also have a Clock that can use a radio too.
Car and Clock could share a base class (Machine). However, not all machines can play
songs; Boat and Plane can't (at least in this example). So how do you accomplish without
duplicating code? You can use a mixin. In Python, giving a class a mixin is as simple as
adding it to the list of subclasses, like this

Foo will inherit all of the properties and methods of main_super, but also those of mixin as
well.

So, to give the classes Car and clock the ability to use a radio, you could
override Car from the last example and write this:

334
 Page 335 of 343

Now you can call car.play_song_on_station(98.7) and

clock.play_song_on_station(101.3), but not something like
boat.play_song_on_station(100.5)
The important thing with mixins is that they allow you to add functionality to much
different objects, that don't share a "main" subclass with this functionality but still
share the code for it nonetheless. Without mixins, doing something like the above
example would be much harder, and/or might require some repetition.

Pandas Transform: Preform operations on groups and concatenate the

results
Simple transform
First, Let's create a dummy dataframe
We assume that a customer can have n orders, an order can have m items, and items
can be ordered more multiple times

# customer_id order_id item

# 0 1 1 apples
# 1 1 1 chocolate
# 2 1 1 chocolate
# 3 1 2 coffee
# 4 1 2 coffee
# 5 2 3 apples
. # 6 2 3 bananas
. # 7 3 4 coffee
# 8 3 5 milkshake
# 9 3 6 chocolate
# 10 3 6 strawberry
# 11 3 6 strawberry

335
 Page 336 of 343

Now, we will use pandas transform function to count the number of orders per customer

# customer_id order item number_of_orders_

_id per_cient
#0 1 1 apples 2
#1 1 1 chocola 2
te
#2 1 1 chocola 2
te
#3 1 2 coffee 2
#4 1 2 coffee 2
# 5 2 3 apples 1
# 6 2 3 bananas 1
# 7 3 4 coffee 3
# 8 3 5 milksha 3
ke
#9 3 6 chocola 3
te
# 10 3 6 strawbe 3
rry
# 11 3 6 strawbe 3
rry

Multiple results per group

Using transform functions that return sub-calculations per group

In the previous example, we had one result per client. However, functions returning
different values for the group can also be applied.

# Create a dummy dataframe

orders_df = [Link]()
orders_df['customer_id'] = [1,1,1,1,1,2,2,3,3,3,3,3]
orders_df['order_id'] = [1,1,1,2,2,3,3,4,5,6,6,6]
336
 Page 337 of 343

orders_df['item'] = ['apples', 'chocolate', 'chocolate', 'coffee',

'coffee', 'apples', 'bananas', 'coffee', 'milkshake',
'chocolate', 'strawberry', 'strawberry']

# Let's try to see if the items were ordered more

than once in each orders # First, we define a
function that will be applied per group
def multiple_items_per_order(_items):
# Apply .duplicated, which will return True is the item
occurs more than once. multiple_item_bool =
_items.duplicated(keep=False)
return(multiple_item_bool)

# Then, we transform each group according to the defined function

orders_df['item_duplicated_per_order'] = ( # Put the results into a new
column
orders_df # Take the orders dataframe
.groupby(['order_id'])['item'] # Create a
separate group for each order_id & select the item
.transform(multiple_items_per_order)) # Apply the defined
function to each
group separately

# Inspecting the results ...

print(orders_df)
# customer order item item_duplicated_p
_id _id er_order
#0 1 1 apples False
#1 1 1 chocola True
te
#2 1 1 chocola True
te
#3 1 2 coffee True
#4 1 2 coffee True
#5 2 3 apples False
#6 2 3 bananas False
#7 3 4 coffee False
#8 3 5 milksha False
ke
#9 3 6 chocola False
te
# 10 3 6 strawbe True
rry
# 11 3 6 strawbe True
rry

337
 Page 338 of 343

Unit Testing
Test Setup and Teardown within [Link]
Sometimes we want to prepare a context for each test to be run under. The setUp method
is run prior to each test in the class. tearDown is run at the end of every test. These
methods are optional. Remember that TestCases are often used in cooperative multiple
inheritance so you should be careful to always call super in these methods so that base
class's setUp and tearDown methods also get called. The base implementation of
TestCase provides empty setUp and tearDown methods so that they can be called
without raising exceptions:

Note that in python2.7+, there is also the addCleanup method that registers functions to be
called after the test is run. In contrast to tearDown which only gets called if setUp
succeeds, functions registered via addCleanup will be called even in the event of an
unhandled exception in setUp. As a concrete example, this method can frequently be seen
removing various mocks that were registered while the test was running:

338
 Page 339 of 343

Another benefit of registering cleanups this way is that it allows the programmer to put
the cleanup code next to the setup code and it protects you in the event that a subclasser
forgets to call super in tearDown.

Asserting on Exceptions
You can test that a function throws an exception with the built-in unittest through two
different methods.
Using a context manager

This will run the code inside of the context manager and, if it succeeds, it will fail the test
because the exception was not raised. If the code raises an exception of the correct type, the
test will continue.

You can also get the content of the raised exception if you want to execute additional assertions
against it.

339
 Page 340 of 343

By providing a callable function

The exception to check for must be the first parameter, and a callable function must be
passed as the second parameter. Any other parameters specified will be passed directly to
the function that is being called, allowing you to specify the parameters that trigger the
exception.

Testing Exceptions
Programs throw errors when for instance wrong input is given. Because of this, one needs
to make sure that an error is thrown when actual wrong input is given. Because of that we
need to check for an exact exception, for this example we will use the following
exception:

This exception is raised when wrong input is given, in the following context where we
always expect a number as text input.

340
 Page 341 of 343

To check whether an exception has been raised, we use assertRaises to check for that
exception. assertRaises
can be used in two ways:

1. Using the regular function call. The first argument takes the exception type,
second a callable (usually a function) and the rest of arguments are passed to this
callable.
2. Using a with clause, giving only the exception type to the function. This has as
advantage that more code can be executed, but should be used with care since
multiple functions can use the same exception which can be problematic. An example:
with [Link](WrongInputException): convert2number("not a number")

This first has been implemented in the following test case:

There also may be a need to check for an exception which should not have been thrown.
However, a test will automatically fail when an exception is thrown and thus may not be
necessary at all. Just to show the options, the second test method shows a case on how one
can check for an exception not to be thrown. Basically, this is catching the exception and
then failing the test using the fail method.

Choosing Assertions Within Unittests

While Python has an assert statement, the Python unit testing framework has better
assertions specialized for tests: they are more informative on failures, and do not depend
on the execution's debug mode.
 Page 342 of 343

Perhaps the simplest assertion is assertTrue, which can be used like this:

This will run fine, but replacing the line above with

will fail.
The assertTrue assertion is quite likely the most general assertion, as anything tested can be
cast as some boolean condition, but often there are better alternatives. When testing for
equality, as above, it is better to write

When the former fails, the message is

======================================================================

FAIL: test ( main .TruthTest)

Traceback (most recent call last):

File "[Link]", line 6, in test

[Link](1 + 1 == 3)

AssertionError: False is not true

but when the latter fails, the message is

======================================================================

FAIL: test ( main .TruthTest)

Traceback (most recent call last):

File "[Link]", line 6, in test

[Link](1 + 1, 3)
AssertionError: 2 != 3
 Page 343 of 343

which is more informative (it actually evaluated the result of the left hand side).

You can find the list of assertions in the standard documentation. In general, it is a good
idea to choose the assertion that is the most specifically fitting the condition. Thus, as
shown above, for asserting that 1 + 1 == 2 it is better to use assertEqual than
assertTrue. Similarly, for asserting that a is None, it is better to use assertIsNone than
assertEqual.

Note also that the assertions have negative forms. Thus assertEqual has its negative
counterpart assertNotEqual, and assertIsNone has its negative counterpart
assertIsNotNone. Once again, using the negative counterparts when appropriate, will
lead to clearer error messages.
used as the key.