Reference Types

Python Iterators

An iterator is an object that contains a countable number of values.

An iterator is an object that can be iterated upon, meaning that you can traverse through
all the values.

Technically, in Python, an iterator is an object which implements the iterator protocol,

which consist of the methods __iter__() and __next__().

Iterator vs Iterable
Lists, tuples, dictionaries, and sets are all iterable objects. They are
iterable containers which you can get an iterator from.

All these objects have a iter() method which is used to get an iterator:

Example
Return an iterator from a tuple, and print each value:

mytuple = ("apple", "banana", "cherry")

myit = iter(mytuple)

print(next(myit))
print(next(myit))
print(next(myit))

Even strings are iterable objects, and can return an iterator:

Example
Strings are also iterable objects, containing a sequence of characters:
mystr = "banana"
myit = iter(mystr)

print(next(myit))
print(next(myit))
print(next(myit))
print(next(myit))
print(next(myit))
print(next(myit))

Looping Through an Iterator

We can also use a for loop to iterate through an iterable object:

Example
Iterate the values of a tuple:

mytuple = ("apple", "banana", "cherry")

for x in mytuple:
print(x)

Example
Iterate the characters of a string:

mystr = "banana"

for x in mystr:
print(x)

The for loop actually creates an iterator object and executes the next() method for each
loop.

Python Infinite Iterators

An infinite iterator is an iterator that never ends, meaning that it will continue
to produce elements indefinitely.

Here is an example of how to create an infinite iterator in Python using

the count() function from the itertools module,

from itertools import count

# create an infinite iterator that starts at 1 and increments by 1 each time

infinite_iterator = count(1)

# print the first 5 elements of the infinite iterator

for i in range(5):
print(next(infinite_iterator))
Output
1
2
3
4
5
Here, we have created an infinite iterator that starts at 1 and increments
by 1 each time.
And then we printed the first 5 elements of the infinite iterator using
the for loop and the next() method.

Create an Iterator
To create an object/class as an iterator you have to implement the
methods __iter__() and __next__() to your object.

As you have learned in the Python Classes/Objects chapter, all classes have a function
called __init__(), which allows you to do some initializing when the object is being created.

The __iter__() method acts similar, you can do operations (initializing etc.), but must
always return the iterator object itself.
The __next__() method also allows you to do operations, and must return the next item in
the sequence.

Example
Create an iterator that returns numbers, starting with 1, and each sequence will increase
by one (returning 1,2,3,4,5 etc.):

class MyNumbers:
def __iter__(self):
self.a = 1
return self

def __next__(self):
x = self.a
self.a += 1
return x

myclass = MyNumbers()
myiter = iter(myclass)

print(next(myiter))
print(next(myiter))
print(next(myiter))
print(next(myiter))
print(next(myiter))

Building Custom Iterators

Building an iterator from scratch is easy in Python. We just have to implement

the __iter__() and the __next__() methods,
● __iter__() returns the iterator object itself. If required, some initialization

can be performed.
● __next__() must return the next item in the sequence. On reaching the

end, and in subsequent calls, it must raise StopIteration.

Let's see an example that will give us the next power of 2 in each iteration.
Power exponent starts from zero up to a user set number,

class PowTwo:
"""Class to implement an iterator
of powers of two"""

def __init__(self, max=0):

[Link] = max

def __iter__(self):
self.n = 0
return self

def __next__(self):
if self.n <= [Link]:
result = 2 ** self.n
self.n += 1
return result
else:
raise StopIteration

# create an object
numbers = PowTwo(3)

# create an iterable from the object

i = iter(numbers)

# Using next to get to the next iterator element

print(next(i)) # prints 1
print(next(i)) # prints 2
print(next(i)) # prints 4
print(next(i)) # prints 8
print(next(i)) # raises StopIteration exception

Output
1
2
4
8
Traceback (most recent call last):
File "<string>", line 32, in <module>
File "<string>", line 18, in __next__
StopIteration
We can also use a for loop to iterate over our iterator class.
for i in PowTwo(3):
print(i)
Output

1
2
4
8
To learn more about object-oriented programming

StopIteration
The example above would continue forever if you had enough next() statements, or if it
was used in a for loop.

To prevent the iteration from going on forever, we can use the StopIteration statement.

In the __next__() method, we can add a terminating condition to raise an error if the
iteration is done a specified number of times:

Example
Stop after 20 iterations:

class MyNumbers:
def __iter__(self):
self.a = 1
return self
 def __next__(self):
if self.a <= 20:
x = self.a
self.a += 1
return x
else:
raise StopIteration

myclass = MyNumbers()
myiter = iter(myclass)

for x in myiter:
print(x)

Python Generators
In Python, a generator is a function that returns an iterator that produces a
sequence of values when iterated over.
Generators are useful when we want to produce a large sequence of values,
but we don't want to store all of them in memory at once.

Create Python Generator

In Python, similar to defining a normal function, we can define a generator

function using the def keyword, but instead of the return statement we use
the yield statement.
def generator_name(arg):
# statements
yield something
Here, the yield keyword is used to produce a value from the generator.
When the generator function is called, it does not execute the function body
immediately. Instead, it returns a generator object that can be iterated over to
produce the values.

Example: Python Generator

Here's an example of a generator function that produces a sequence of

numbers,

def my_generator(n):

# initialize counter
value = 0

# loop until counter is less than n

while value < n:

# produce the current value of the counter

yield value

# increment the counter

value += 1

# iterate over the generator object produced by my_generator

for value in my_generator(3):

# print each value produced by generator

print(value)
Output
0
1
2
In the above example, the my_generator() generator function takes an
integer n as an argument and produces a sequence of numbers from 0 to n-1.
The yield keyword is used to produce a value from the generator and pause the
generator function's execution until the next value is requested.
The for loop iterates over the generator object produced by my_generator(), and
the print statement prints each value produced by the generator.
We can also create a generator object from the generator function by calling
the function like we would any other function as,

generator = my_range(3)
print(next(generator)) # 0
print(next(generator)) # 1
print(next(generator)) # 2

Python Generator Expression

In Python, a generator expression is a concise way to create a generator
object.

It is similar to a list comprehension, but instead of creating a list, it creates a

generator object that can be iterated over to produce the values in the
generator.
Generator Expression Syntax

A generator expression has the following syntax,

(expression for item in iterable)

Here, expression is a value that will be returned for each item in the iterable.
The generator expression creates a generator object that produces the values
of expression for each item in the iterable, one at a time, when iterated over.
Example 2: Python Generator Expression

# create the generator object

squares_generator = (i * i for i in range(5))

# iterate over the generator and print the values

for i in squares_generator:
print(i)
Output
0
1
4
9
16
Here, we have created the generator object that will produce the squares of
the numbers 0 through 4 when iterated over.
And then, to iterate over the generator and get the values, we have used
the for loop.

Use of Python Generators

There are several reasons that make generators a powerful implementation.

1. Easy to Implement

Generators can be implemented in a clear and concise way as compared to

their iterator class counterpart. Following is an example to implement a
sequence of power of 2 using an iterator class.
class PowTwo:
def __init__(self, max=0):
self.n = 0
[Link] = max

def __iter__(self):
return self

def __next__(self):
if self.n > [Link]:
raise StopIteration

result = 2 ** self.n
self.n += 1
return result
The above program was lengthy and confusing. Now, let's do the same using a
generator function.

def PowTwoGen(max=0):
n = 0
while n < max:
yield 2 ** n
n += 1
Since generators keep track of details automatically, the implementation was
concise and much cleaner.

2. Memory Efficient

A normal function to return a sequence will create the entire sequence in

memory before returning the result. This is an overkill, if the number of items
in the sequence is very large.

Generator implementation of such sequences is memory friendly and is

preferred since it only produces one item at a time.

3. Represent Infinite Stream

Generators are excellent mediums to represent an infinite stream of data.
Infinite streams cannot be stored in memory, and since generators produce
only one item at a time, they can represent an infinite stream of data.

The following generator function can generate all the even numbers (at least
in theory).

def all_even():
n = 0
while True:
yield n
n += 2
4. Pipelining Generators

Multiple generators can be used to pipeline a series of operations. This is best

illustrated using an example.

Suppose we have a generator that produces the numbers in the Fibonacci

series. And we have another generator for squaring numbers.

If we want to find out the sum of squares of numbers in the Fibonacci series,
we can do it in the following way by pipelining the output of generator
functions together.

def fibonacci_numbers(nums):
x, y = 0, 1
for _ in range(nums):
x, y = y, x+y
yield x

def square(nums):
for num in nums:
yield num**2

print(sum(square(fibonacci_numbers(10))))
# Output: 4895
This pipelining is efficient and easy to read (and yes, a lot cooler!).

Python Closures
Python closure is a nested function that allows us to access variables of the
outer function even after the outer function is closed.

Before we learn about closure, let's first revise the concept of nested functions
in Python.

Understanding Inner and Outer Functions

Let us see a simple example to work around inner and outer functions −

Example
def outerFunc(a):
# the enclosing function
def innerFunc():
# accessing outer function’s variable from inner function
print(a)
return innerFunc

# calling the enclosing function

demoFunc = outerFunc('Hello')
demoFunc()

Output
Hello
To understand Python Closures, lets first understand what’s nested function and python
class.

Nested function in Python

In Python, we can create a function inside another function. This is known as a
nested function. For example,

def greet(name):
# inner function
def display_name():
print("Hi", name)

# call inner function

display_name()

# call outer function

greet("John")

# Output: Hi John

In the above example, we have defined the display_name() function inside

the greet() function.
Here, display_name() is a nested function. The nested function works similar to
the normal function. It executes when display_name() is called inside the
function greet().

Python Closures
As we have already discussed, closure is a nested function that helps us
access the outer function's variables even after the outer function is closed.
For example,

def greet():
# variable defined outside the inner function
name = "John"

# return a nested anonymous function

return lambda: "Hi " + name

# call the outer function

message = greet()

# call the inner function

print(message())

# Output: Hi John
In the above example, we have created a function named greet() that returns a
nested anonymous function.
Here, when we call the outer function,

message = greet()
The returned function is now assigned to the message variable.
At this point, the execution of the outer function is completed, so
the name variable should be destroyed. However, when we call the anonymous
function using
print(message())
we are able to access the name variable of the outer function.
It's possible because the nested function now acts as a closure that closes the
outer scope variable within its scope even after the outer function is executed.

Let's see one more example to make this concept clear.

Example: Print Odd Numbers using Python Closure
def calculate():
num = 1
def inner_func():
nonlocal num
num += 2
return num
return inner_func

# call the outer function

odd = calculate()

# call the inner function

print(odd())
print(odd())
print(odd())

# call the outer function again

odd2 = calculate()
print(odd2())

Output
3
5
7
3
In the above example,

odd = calculate()
This code executes the outer function calculate() and returns a closure to the
odd number. T
That's why we can access the num variable of calculate() even after completing
the outer function.
Again, when we call the outer function using

odd2 = calculate()
a new closure is returned. Hence, we get 3 again when we call odd2().

When to use closures?

So what are closures good for?

Closures can be used to avoid global values and provide data hiding, and can
be an elegant solution for simple cases with one or few methods.

However, for larger cases with multiple attributes and methods, a class
implementation may be more appropriate.

def make_multiplier_of(n):
def multiplier(x):
return x * n
return multiplier

# Multiplier of 3
times3 = make_multiplier_of(3)

# Multiplier of 5
times5 = make_multiplier_of(5)

# Output: 27
print(times3(9))

# Output: 15
print(times5(3))

# Output: 30
print(times5(times3(2)))

Python Decorators make extensive use of closures as well.

On a concluding note, it is good to point out that the values that get enclosed
in the closure function can be found out.

All function objects have a __closure__ attribute that returns a tuple of cell
objects if it is a closure function.
Referring to the example above, we know times3 and times5 are closure
functions.

Use __closure__ attribute

To get more information we can use __closure__ attribute and cell objects −

Example
def closureFunc(start):
def incrementBy(inc):
return start + inc
return incrementBy

a= closureFunc(9)
b = closureFunc(90)

print ('type(a)=%s' %(type(a)))

print ('a.__closure__=%s' %(a.__closure__))
print ('type(a.__closure__[0])=%s' %(type(a.__closure__[0])))
print ('a.__closure__[0].cell_contents=%s' %(a.__closure__[0].cell_contents))

print ('type(b)=%s' %(type(b)))

print ('b.__closure__=%s' %(b.__closure__))
print ('type(b.__closure__[0])=%s' %(type(b.__closure__[0])))
print ('b.__closure__[0].cell_contents=%s' %(b.__closure__[0].cell_contents))
Output
type(a)=<class 'function'>
a.__closure__=<cell at 0x7efdfb4683a8: int object at 0x7efdfc7324c0>
type(a.__closure__[0])=<class 'cell'>
a.__closure__[0].cell_contents=9
type(b)=<class 'function'>
b.__closure__=<cell at 0x7efdfb3f9888: int object at 0x7efdfc732ee0>
type(b.__closure__[0])=<class 'cell'>
b.__closure__[0].cell_contents=90

Python Decorators
In Python, a decorator is a design pattern that allows you to modify the
functionality of a function by wrapping it in another function.

The outer function is called the decorator, which takes the original function as
an argument and returns a modified version of it.

Prerequisites for learning decorators

Before we learn about decorators, we need to understand a few important
concepts related to Python functions. Also, remember that everything in
Python is an object, even functions are objects.

Nested Function
We can include one function inside another, known as a nested function. For
example,

def outer(x):
def inner(y):
return x + y
return inner

add_five = outer(5)
result = add_five(6)
print(result) # prints 11

# Output: 11

Here, we have created the inner() function inside the outer() function.

Pass Function as Argument

We can pass a function as an argument to another function in Python. For
Example,

def add(x, y):

return x + y

def calculate(func, x, y):

return func(x, y)

result = calculate(add, 4, 6)

print(result) # prints 10
Output
10
In the above example, the calculate() function takes a function as its argument.
While calling calculate(), we are passing the add() function as the argument.
In the calculate() function, arguments: func, x, y become add, 4, and 6 respectively.
And hence, func(x, y) becomes add(4, 6) which returns 10.
Return a Function as a Value
In Python, we can also return a function as a return value. For example,

def greeting(name):
def hello():
return "Hello, " + name + "!"
return hello

greet = greeting("Atlantis")
print(greet()) # prints "Hello, Atlantis!"

# Output: Hello, Atlantis!

In the above example, the return hello statement returns the inner hello() function.
This function is now assigned to the greet variable.
That's why, when we call greet() as a function, we get the output.

Python Decorators
As mentioned earlier, A Python decorator is a function that takes in a function
and returns it by adding some functionality.

In fact, any object which implements the special __call__() method is termed
callable. So, in the most basic sense, a decorator is a callable that returns a
callable.
Basically, a decorator takes in a function, adds some functionality and returns
it.

def make_pretty(func):
 def inner():
print("I got decorated")
func()
return inner

def ordinary():
print("I am ordinary")

# Output: I am ordinary

Here, we have created two functions:

● ordinary() that prints "I am ordinary"

● make_pretty() that takes a function as its argument and has a nested

function named inner(), and returns the inner function.

We are calling the ordinary() function normally, so we get the output "I am ordinary".
Now, let's call it using the decorator function.

def make_pretty(func):
# define the inner function
def inner():
# add some additional behavior to decorated function
print("I got decorated")

# call original function

func()
# return the inner function
return inner

# define ordinary function

def ordinary():
print("I am ordinary")

# decorate the ordinary function

decorated_func = make_pretty(ordinary)
# call the decorated function
decorated_func()

Output
I got decorated
I am ordinary

In the example shown above, make_pretty() is a decorator. Notice the code,

decorated_func = make_pretty(ordinary)
● We are now passing the ordinary() function as the argument to
the make_pretty().
● The make_pretty() function returns the inner function, and it is now
assigned to the decorated_func variable.

decorated_func()
Here, we are actually calling the inner() function, where we are printing
@ Symbol With Decorator

Instead of assigning the function call to a variable, Python provides a much

more elegant way to achieve this functionality using the @ symbol. For
Example,
def make_pretty(func):

def inner():
print("I got decorated")
func()
return inner

@make_pretty
def ordinary():
print("I am ordinary")

ordinary()
Output
I got decorated
I am ordinary
Here, the ordinary() function is decorated with the make_pretty() decorator using
the @make_pretty syntax, which is equivalent to calling ordinary =
make_pretty(ordinary).

Decorating Functions with Parameters

The above decorator was simple and it only worked with functions that did not
have any parameters. What if we had functions that took in parameters like:

def divide(a, b):

return a/b
This function has two parameters, a and b. We know it will give an error if we
pass in b as 0.
Now let's make a decorator to check for this case that will cause the error.

def smart_divide(func):
def inner(a, b):
print("I am going to divide", a, "and", b)
if b == 0:
print("Whoops! cannot divide")
return

return func(a, b)

return inner

@smart_divide
def divide(a, b):
print(a/b)

divide(2,5)
divide(2,0)

Output
I am going to divide 2 and 5
0.4
I am going to divide 2 and 0
Whoops! cannot divide

Here, when we call the divide() function with the arguments (2,5),
the inner() function defined in the smart_divide() decorator is called instead.
This inner() function calls the original divide() function with the
arguments 2 and 5 and returns the result, which is 0.4.
Similarly, When we call the divide() function with the arguments (2,0),
the inner() function checks that b is equal to 0 and prints an error message
before returning None.

Chaining Decorators in Python

Multiple decorators can be chained in Python.

To chain decorators in Python, we can apply multiple decorators to a single

function by placing them one after the other, with the most inner decorator
being applied first.

def star(func):
def inner(*args, **kwargs):
print("*" * 15)
func(*args, **kwargs)
print("*" * 15)
return inner

def percent(func):
def inner(*args, **kwargs):
 print("%" * 15)
func(*args, **kwargs)
print("%" * 15)
return inner

@star
@percent
def printer(msg):
print(msg)

printer("Hello")

Output
***************
%%%%%%%%%%%%%%%
Hello
%%%%%%%%%%%%%%%
***************
The above syntax of,

@star
@percent
def printer(msg):
print(msg)
is equivalent to

def printer(msg):
print(msg)
printer = star(percent(printer))
The order in which we chain decorators matter. If we had reversed the order
as,

@percent
@star
def printer(msg):
print(msg)
The output would be:

%%%%%%%%%%%%%%%
***************
Hello
***************
%%%%%%%%%%%%%%%