Iteration

Iterables & Iteration

Comprehensions

  • Comprehensions can process more than one input sequence
  • Multiple input sequences in comprehensions work like nested for-loops
  • Comprehensions can also have multiple if-clauses interspersed with the for-clauses
  • Later clauses in a comprehension can reference variables bound in earlier clauses
  • Comprehension can also appear in the result expression of a comprehension, resulting in nested sequences

Example

## Comprehensions    
>>> points
[(0, 0), (0, 1), (0, 2), (0, 3), (1, 0), (1, 1), (1, 2), (1, 3), (2, 0), (2, 1), (2, 2), (2, 3)]
>>> points_c=[(x,y) for x in range(3) for y in range(4) ]
>>> points_c
[(0, 0), (0, 1), (0, 2), (0, 3), (1, 0), (1, 1), (1, 2), (1, 3), (2, 0), (2, 1), (2, 2), (2, 3)]
>>> points_d=[(x,y)
...     for x in range(3)
...     if x > 0
...     for y in range(4)
...     if y > x ]
>>> points_d
[(1, 2), (1, 3), (2, 3)]
>>> points_e = [ [ y-1 for y in range(x)  ] for x in range(3)]
>>> points_e
[[], [-1], [-1, 0]]

Functional-style tools

  • Python provides a number of functional-style tools for working with iterators

Map

  • map() calls a function for each element in its input sequences
  • map() returns an iterable object, not a fully-evaluated collection
  • map() results are lazily evaluated, meaning that you must access them to force their calculation
  • map() results are typically evaluated through the use of iteration constructs such as for-loops
  • You must provide as many input sequences to map() as the callable argument has parameters
  • map() takes one element from each input sequence for each output element it produces
  • map() stops producing output when its shortest input sequence is exhausted
  • map() can be used to implement the same behavior as comprehensions in some cases
Example
>>> sizes =['small','medium', 'large']
>>> colors = ['lavender', 'teal', 'burnt orange']
>>> animals = ['koala', 'platypus', 'salamander']
>>> def combine( size, color, animal):
...     return '{} {} {}'.format(size, color, animal)
...
>>> list(map(combine, sizes, colors, animals ))
['small lavender koala', 'medium teal platypus', 'large burnt orange salamander']
>>> import itertools
>>> def combine2(quantity, size, color, animal):
...     return '{} - {} {} {}'.format(quantity, size, color, animal)
...
>>> list(map(combine2, itertools.count(), sizes, colors, animals ))
['0 - small lavender koala', '1 - medium teal platypus', '2 - large burnt orange salamander']

Filter

  • filter() selects values from an input sequence which match a specified criteria
  • filter() passes each element in its input sequence to the function argument
  • filter() returns an iterable over the input elements for which the function argument is truthy
  • Like map(), filter() produces its output lazily If you pass None as the first argument to filter(), it yields the input values which evaluate to True in a boolean context reduce() cumulatively applies a function to the elements of an input sequence
Example
## filter
>>> negs = filter(lambda x: x<0 , [3, -1, 2,-4,0,9,-33])
>>> list(negs)
[-1, -4, -33]
>>> notnones = filter(None, [0,1, False, True, [], () , {}, (1,2,), [1,2], '', 'yes'])
>>> list(map(type, list(notnones)))
[<class 'int'>, <class 'bool'>, <class 'tuple'>, <class 'list'>, <class 'str'>]

Reduce

  • reduce() calls the input function with two arguments: the accumulated result so far, and the next element in the sequence
  • reduce() is a generalization of summation
  • reduce() returns the accumulated result after all of the input has been processed
  • If you pass an empty sequence to reduce() it will raise a TypeError
  • reduce() accepts an optional initial value argument
  • This initial value is conceptually added to the front of the input sequence
  • The initial value is returned if the input sequence is empty
  • The map() and reduce() functions in Python are related to the ideas in the map-reduce algorithm
Example
## reduce
>>> from functools import reduce
>>> import operator
>>> reduce(operator.add, [1,2,3,4,5])
15

### x is interim result , y is the next sequence value
>>> def mul(x, y):
...     print(' mul {} {} '.format(x,y))
...     return x * y
...
>>> reduce( mul, range(1, 5))
mul 1 2
mul 2 3
mul 6 4
24

The next() function

  • Python’s next() function calls __next__() on its argument
  • Iterators in Python must support the __next__() method
  • __next__() should return the next item in the sequence, or raise StopIteration if it is exhausted
  • Python’s iter() function calls __iter__() on its argument
  • Iterable objects in Python must support the __iter__() method
  • __iter__() should return an iterator for the iterable object
  • Objects with a __getitem__() method that accepts consecutive integer indices starting at zero are also iterables
  • Iterables implemented via __getitem__() must raise IndexError when they are exhausted
Example of Iterator
class ExampleIterator:
def __init__(self, data):
    self.index = 0
    self.data = data

def __iter__(self):
    return self

def __next__(self):
    if self.index >= len(self.data):
        raise StopIteration()

    rslt = self.data[self.index]
    self.index += 1
    return rslt
Example of Iterable
class ExampleIterable:
    def __init__(self):
        self.data = [1, 2, 3]

    def __iter__(self):
        return ExampleIterator(self.data)
Example of AnotherIterable
class AnotherIterable:
    def __init__(self):
        self.data = [1, 2, 3]

    def __getitem__(self, idx):
        return self.data[idx]

The iter() function

  • The extended form of iter() accepts a zero-argument callable and a sentinel value
  • Extended iter() repeatedly calls the callable argument until it returns the sentinel value
  • The values produced by extended iter() are those returned from the callable
Example
>>> ts = iter(datetime.datetime.now, None)
>>> next(ts)
datetime.datetime(2017, 7, 14, 14, 38, 10, 752761)
>>> next(ts)
datetime.datetime(2017, 7, 14, 14, 38, 13, 373613)
>>> next(ts)
datetime.datetime(2017, 7, 14, 14, 38, 14, 754588)

## Read file 
## Content of the file file.txt
## You are reading 
## the file
## you won't read 
## the 
## END
## but not see the END above

>>> with open('file.txt', 'rt') as f:
...     for line in iter(lambda: f.readline().strip(), 'END'):
...             print(line)
...
You are reading
the file
you won't read
the
  • One use case for extended iter() is to iterate using simple functions
  • Protocol conforming iterators must also be iterable
Example of Sensor
import datetime
import itertools
import random
import time

class Sensor:
    def __iter__(self):
        return self

    def __next__(self):
        return random.random()

sensor = Sensor()
timestamps = iter(datetime.datetime.now, None)

for stamp, value in itertools.islice(zip(timestamps, sensor), 10):
    print(stamp, value)
    time.sleep(1)