The following post contains a summary of the book titled Treading on Python II by Matt Harrison

Programming Styles

Python supports three types of programming paradigms
- Imperative/Procedural
- Object Oriented
- Declarative/Functional

Iterator Protocol

iter is a global built-in function that calls the object’s dunder method __iter__
Writing a for loop based on iterators

test = [1, 2, 4]
for i in test:
    print(i)
iterator = iter(test)
while True:
try:
x = iterator.next()
print(x)
except StopIteration as e:
break

Each loop is converted in to byte code and this byte code is run by the interpreter
The actual iterator is not the object that is being iterated. list and string have separate iterator objects to iterate upon them
StringIO class implements the iterator protocol
Iterator protocol defines the process of iterating the objects in a container utilizing the methods __iter__ and __next__

Iterable vs. Iterator

What is an iterable ? An iterable is any object that allows iteration
- This object must implement __iter__ method and must return an iterator object. This iterator object can be the same object or a completely different object
- This object must also implement __next__ method
Iterators are good for one pass over the values. This means that iterators are stateful
range(10) returns an rangeiterator object that implements __iter__ and __next__ methods
A class is called a self-iterator if its __iter__ method returns the same instance on which the dunder method has been invoked
Most iterable objects are not self-iterators. They return a different object when their __iter__ method is invoked
If the datatype is a self-iterator, then there could be problems in nested loops. Here is a nice example

class Counter(object):
    def __init__(self, size):
        self.size = size
        self.start = 0
    def __iter__(self):
        return self
    def __next__(self):
        if self.start < self.size:
            self.start +=1
            return self.start
        raise StopIteration
x = Counter(2)
y = Counter(3)
for i in x:
for j in y:
print(i,j)

The above code does not work as desired as the iter returns the same instance and the inner loop goes through only once and never gets repeated. The solution to this problem is to make sure that the iter method returns a different object as compared to the original object on which the method was invoked

StringIO is a self-iterator and hence allows a single traversal through the data
One can modify a class to be an iterable and not an iterator by modifying its __iter__ method to return a fresh instance of the class
Self-iterators will exhaust. If that is an issue, make objects that are only iterable, but not iterators themselves
Never thought about iterables and iterators in this detail, until now. This chapter has been super awesome as it talks about the perils of making a data instance in to a self-iterator
One can easily create an object spitting out an infinite sequence using an iterator
What did I learn from this chapter ?

Iterators are different from Iterable. Any object that implements __iter__ and __next__ method are iterators. Iterable are objects that need to implement __iter__- method. Iterables are used in for loops, while loops etc. This is exactly the object we are looking for. Iterables have __iter__ method. This method can return an instance of iterator whenever

Generators

Generators were introduced in Python 2.3
Iterators have two problems
- They must track their state within the iterator instance in order to provide the correct elements across multiple calls
- If a list is returned, it could potentially consume large amounts of memory
What is generator according to Python documentation ?
- A function which returns an iterator is a generator
- It looks like a normal function but returns a yield statement to produce a series of values usable in a for-loop or can be retrieved one at a time using next() method
- Each yield suspends processing, remembering the location execution state
generators are not invoked when they are created. They are invoked only when they are iterated over
The differences between function and generator are
- Generators are not executed when they are invoked. They are invoked only when they are iterated over
- Generators can be iterated over whereas functions cannot be
- Generators freeze their state after a yield statement
Any return statement inside a generator function is treated as raising a StopIteration exception
Because generators exhaust, they do not serve well for re-use in nested loops
Invoking __iter__ on a generator will return the same generator object instance
Generators are self-generators
return in a generator causes the generator to stop and exit

Object Generators

Not only functions, but methods in a class can be generator functions
Object generators are reusable if they do not attach state to the instance
Every time a generator is iterated over, Python creates a separate object to iterate over

Generators in Practice

The main pattern to look for is accumulation in to a list during the loop.
Generators exhaust. They are not good for reuse
A nice way of debugging generators is to convert them in to list
pdb will not step in to the generator unless it is actually iterated over
Generators do not index or slice
itertools module contains function to slice generators
Generators have no inherent length
It is possible that Generators are slower than the normal list iteration for smaller chunks of data
Generators may consume less memory
Generators always evaluate to True. This is an area where generators are not a drop in replacement for lists
The OrderedDict class in the collections.py module in the Standard Library uses a generator in its __iter__ method

List or Generator

If the access to items is repeated, then it is better to use list
If data fits in memory, use lists
If operations on whole sequence are performed, then using generator is not a good idea
A list can be converted to generator but not vice-versa
More of Python objects are becoming lazy since Python 3

Real World Uses

Database chunking
Recursive generators

Functions

Functions are first class citizens. It means that functions can be passed in to other functions, returned from other functions and assigned to variable
Java language has no concept of passing functions around
How to check whether a function is callable ?

1
2
3
4
5
6


import collections
def foo():
    return 1
callable(foo)
hasattr(foo,'__call__')
isinstance(foo, collections.Callable)

Instances of classes in Python have attributes. Functions, being instances of function, likewise have attributes
__name__ attribute of the function stores the function name
__doc__ attribute of the function stores the doc string
__defaults__ attribute of the function stores the defaults
Default parameter for functions only use non-mutable types
locals and globals will return a mapping of names to objects that their respective namespaces contain
Any nested function has read/write access to globals and built-ins
A free variable is any variable that is neither a local variable nor passed as an argument

Function Parameters

Python supports four different types of parameters for functions
- Normal parameters
- Keyword parameters
- Variable parameters(*args)
- Variable keyword parameters (**kwargs)
If you pass a splat operator, it will be passed as a tuple in to the function
* splat operator can be used to flatten an input and pass the input to a function that expects normal arguments
** operator can be used to flatten a dictionary and pass the input in to a function the expects normal arguments
The order of parameters in any function should be normal, keyword, variable, variable keyword

Closures

They are used to keep a common interface, eliminate code duplication and to delay execution of a function
They enable generation of functions and conforming to an interface
When you define a function using def, it creates a function object and a variable name

Decorators

A decorator is a method for altering a callable. Closure enables the creation of decorators
Python 2.4 introduced syntactic sugar to write decorators for a function
Parametrized decorator is useful to customize the specific function it is wrapping
common uses of decorators
- function arguments
- function being wrapped
- results of a function
common instances where decorators are used
- caching expensive computations
- retrying a function that might fail
- logging the amount of time spent in a function
- timing out a function call
- access control

Alternative Decorator implementations

One cannot have a lambda function in another lambda function ? Why ? Lambdas support only expression in their body. Another lambda would be compound statement
you can write classes that serve as decorator instances
Decorating a decorator looks complicated. I guess unless I use this in the code, I will not be able to internalize the lesson

Functional Constructs in Python

The biggest drawback to lambda expressions is that they support a single expression in their body
lambda expressions are considered pure
map operator can only operate on finite sequences. It cannot operate on infinite generators or iterators
In Python 3, reduce is a part of functools module
filter was converted to lazy class in Python 3
Tail Call Optimization refers to an optimization that applies to certain recursive functions where they call themselves as the last operation
Python interpreter comes with rails to protect against using too much memory by creating a lot of stacks. By default it is set to 1000
Generator expressions return a generator object, which follows the iterator protocol
A generator expression can be iterated only once whereas a list comprehension can be run through multiple times
An object that allows for single iteration will have __iter__ and __next__ method
An iterable object will have __iter__ method that gives an iterator every time it is called
Dictionary comprehensions are not lazy and are evaluated upon creation
Set comprehensions are not lazy and are evaluated upon creation
operator module is super useful when programming in a functional style or using comprehension constructs

Takeaway

I had downloaded this book in June 2019 and somehow had never found time to go through this book. Thanks to my Python immersion in this lock-down period, I have managed to read through the entire book in a day. With out spending time going through Python Workout and Tiny Python Projects, I would not have had followed the book in its entirety in one day. Since I had been working through the exercises in the previous two books, I could follow most of the stuff mentioned in the book. Needless to say, the book has valuable content that would be helpful to understand iterators, generators, decorators and list comprehensions. Super useful book and I am glad I have managed to read it. This was my HIGHLIGHT for the day that I had planned. I am happy that I have managed get it done.

Treading on Python - II - Book Summary

Contents