Python - List Comprehension

Python features functional programming tools like map and filter for mapping operations over sequences and collecting results.

Since this is such a common task in Python coding, Python made a new expression: the list comprehension which is more flexible than map and filter. List comprehensions apply an arbitrary expression to items in an iterable rather than applying function. It provides a compact way of mapping a list into another list by applying a function to each of the elements of the list.

Python's built-in ord returns the ASCII integer code of a character:

>>> ord('A')
65

If we want to collect the ASCII codes of all characters in a string, the most straightforward method is using a for loop and append the results to a list:

>>> result = []
>>> for x in 'Dostoyevsky':
	result.append(ord(x))
	
>>> result
[68, 111, 115, 116, 111, 121, 101, 118, 115, 107, 121]
>>> 

If we use map, we can get the same result with a single function call:

>>> result = list(map(ord,'Dostoyevsky'))
>>> result
[68, 111, 115, 116, 111, 121, 101, 118, 115, 107, 121]
>>>

But, we can get the similar result from a list comprehension expression. While map maps a function over a sequence, list comprehensions map an expression over a sequence:

>>> result = [ord(x) for x in 'Dostoyevsky']
>>> result
[68, 111, 115, 116, 111, 121, 101, 118, 115, 107, 121]
>>> 

List comprehensions collect the result of applying an arbitrary expression to a sequence and return them in a new list. The effect is similar to that of the for loop and the map call. List comprehensions become more convenient when we need to apply an arbitrary expression to a sequence:

>>> [x ** 3 for x in range(5)]
[0, 1, 8, 27, 64]

If we had to use map, we would need to write a function to implement the square operation, probably, lambda instead of using a def:

>>> list(map((lambda x: x ** 2),range(5)))
[0, 1, 4, 9, 16]

This does the job. It's only a little bit longer that the list comprehension. For more advance kinds of expressions, however, list comprehensions will often require considerably less typing.

If we use if with list compression, it is almost equivalent to the filter built-in.

Let's make examples using both schemes.

>>> 
>>> [x for x in range(10) if x % 2 == 0]
[0, 2, 4, 6, 8]

>>> list(filter((lambda x: x % 2 == 0), range(10)))
[0, 2, 4, 6, 8]

>>> result = []
>>> for x in range(10):
	if x % 2 == 0:
		result.append(x)
	
>>> result
[0, 2, 4, 6, 8]
>>> 

All of these use the modulus operator %, to extract even numbers. The filter call here is not much longer that the list comprehension either. But we can combine an if and a map, in a single expression:

>>> [x ** 2 for x in range(10) if x % 2 == 0]
[0, 4, 16, 36, 64]
>>> 

We collect the squares of the even numbers from 0 to 9. The for loop skips numbers which the if on the right is false. The expression on the left computes the squares. The equivalent map requires a lot more work. We have to combine filter with map iteration:

>>> list( map((lambda x: x ** 2), filter((lambda x: x % 2== 0),range(10))) )
[0, 4, 16, 36, 64]

Actually, list comprehensions are more general. We can code any number of nested for loop in a list comprehension, and each may have an optional associated if test. When for loop are nested within a list comprehension, they work like equivalent for loop statement:

>>> result = []
>>> result = [ x ** y for x in [10, 20, 30] for y in [2, 3, 4]]
>>> result
[100, 1000, 10000, 400, 8000, 160000, 900, 27000, 810000]
>>> 

More verbose version is:

>>> result = []
>>> for x in [10, 20, 30]:
	for y in [2, 3, 4]:
		result.append(x ** y)

		
>>> result
[100, 1000, 10000, 400, 8000, 160000, 900, 27000, 810000]
>>> 

Though list comprehensions construct lists, they can iterate over any sequence:

>>> [x + y for x in 'ball' for y in 'boy']
['bb', 'bo', 'by', 'ab', 'ao', 'ay', 'lb', 'lo', 'ly', 'lb', 'lo', 'ly']
>>> 

Here is a much more complicated list comprehension example:

>>> [(x,y) for x in range(5) if x % 2 == 0 for y in range(5) if y % 2 == 1]
[(0, 1), (0, 3), (2, 1), (2, 3), (4, 1), (4, 3)]

The expression permutes even numbers from 0 through 4 with odd numbers from 0 through 4. Here is an equivalent version which is much more verbose:

>>> result = []
>>> for x in range(5):
	if x % 2 == 0:
		for y in range(5):
			if y % 2 == 1:
				result.append((x,y))

				
>>> result
[(0, 1), (0, 3), (2, 1), (2, 3), (4, 1), (4, 3)]
>>> 

Let's see how the list comprehension works with Matrixes. For example, we have two 3 x 3 matrixes as lists of nested list:

>>> M1 = [[1, 2, 3],
          [4, 5, 6],
          [7, 8, 9]]
>>> M2 = [[9, 8, 7],
          [6, 5, 4],
          [3, 2, 1]]

We can index rows and columns within rows:

>>> M1[2]
[7, 8, 9]
>>> M1[2][2]
9

List comprehensions are powerful tools for processing such structures because they automatically scan rows and columns for us. For example, though this structure stores the matrix by rows, to collect the second column we can simply iterate across the rows and pull out the desired column. We can also iterate through positions in the rows and index as we go:

>>> [r[2] for r in M1]
[3, 6, 9]

Here, we pulled out column 3 from M1. We can get the same result from the following list comprehension.

>>> [M1[r][2] for r in (0, 1, 2)]
[3, 6, 9]

We typically should use simple for loops when getting started with Python, and map. Use comprehension where they are easy to apply. However, there is a substantial performance advantage to use list comprehension. The map calls are roughly twice as fast as equivalent for loops. List comprehensions are usually slightly faster than map calls. This speed difference is largely due to the fact that map and list comprehensions run at C language speed inside the interpreter. It is much faster that stepping through Python for loop code within the Python Virtual Machine (PVM).

However, for loops make logic more explicit, we may want to use them on the grounds of simplicity. On the other hand, map and list comprehensions are worth knowing and using for simpler kinds of iterations if the speed of application is an important factor.