# Effective Python 2E
### Chapter 4: Comprehensions and Generators
[David Ye](https://dwye.dev/) @ Houzz
---
## Outline
- [Comprehensions (Items 27-29)](#/2)
- [Generators (Items 30-35)](#/3)
- [`itertools` (Item 36)](#/4)
---
## Comprehensions
```python=
> [x ** 3 for x in range(4)]
[0, 1, 8, 27]
> {x: x ** 3 for x in range(4)}
{0: 0, 1: 1, 2: 8, 3: 27}
> {x for x in range(10) if x % 3 != 0}
{1, 2, 4, 5, 7, 8}
```
----
### 27: Use Comprehensions Instead of `map` and `filter`
```python=
map(lambda x: x ** 2, filter(lambda x: x % 2 == 0, other_list))
[x ** 2 for x in other_list if x % 2 == 0]
# [0, 4, 16, 36, 64, 100, 144, 196, 256, 324]
```
- more clear
- easily skip items w/o `filter`
- Dictionaries / Sets
----
Multiple conditions at the same loop level have an implicit `and` expression.
```python=
a = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
> [x for x in a if x > 4 if x % 2 == 0]
[6, 8, 10]
> [x for x in a if x > 4 and x % 2 == 0]
[6, 8, 10]
```
----
### 28: Avoid More Than Two Control Subexpressions in Comprehensions
```python=
[[x for x in row if x % 3 == 0] for row in matrix if sum(row) >= 10]
```
\>= 2 control subexpressions: difficult to read.
```python=
alt = []
for row in matrix:
if sum(row) >= 10:
alt.append(
[x for x in row if x % 3 == 0]
)
```
----
### 29: Avoid Repeated Work in Comprehensions by Using Assignment Expressions
Assignment Expressions / Walrus Operator `:=`
```python=
stock = {
'nails': 125,
'screws': 35,
'wingnuts': 8,
'washers': 24,
}
> {
> name: tenth for name, count in stock.items()
> if (tenth := count // 10) > 10
> }
{'nails': 12}
```
----
`:=` will **leak** the loop variable into the containing scope
```python=
> half = [(last := count // 2) for count in stock.values()]
> f'Last item of {half} is {last}'
'Last item of [62, 17, 4, 12] is 12'
```
`for` loop also leak the variable:
```python=
> for count in stack.values():
> pass
> f'last item of {list(stock.values())} is {count}'
'last item of [125, 35, 8, 24] is 24'
```
but not for the `for` statement in comprehension:
```python=
> half = [count // 2 for count in stock.values()]
> f'last item of {stack} is {count}'
# NameError: name 'count' is not defined
```
- recommend using assignment expressions only in the condition part
---
## Generators
a generator function returns generator iterator when called
```python=
def range(n):
i = 0
while i < n:
yield i
i += 1
for i in range(10):
print(i) # 0 1 2 3 4 ... 9
print(range(3))
# <generator object range at 0x1088aae40>
```
----
### 30: Consider Generators Instead of Returning Lists
```python=
def index_words(text):
result = []
if text:
result.append(0)
for index, letter in enumerate(text):
if letter == ' ':
result.append(index + 1)
return result
```
```python=
def index_words_iter(text):
if text:
yield 0
for index, letter in enumerate(text):
if letter == ' ':
yield index + 1
> index_words_iter('Four score and seven years ago...')
[0, 5, 11, 15, 21, 27]
```
- bounded memory requirements
- for infinite generators
- works with `itertools`
----
### 31: Be Defensive When Iterating Over Arguments
iterator are stateful, can only be iterated once:
- convert to `list`
- construct another iterator
- check input:
```python=
def defensive_iteration(container):
if iter(container) is container: # iterator.__iter__ returns itself
raise TypeError('Must apply a container')
```
```python=
from collections.abc import Iterator # Abstract Base Classes
def normalize_defensive2(numbers):
if isinstance(numbers, Iterator):
raise TypeError('Must apply a container')
```
----
### 32: Consider Generator Expressions for Large List Comprehensions
```python=
[len(x) for x in open('my_file.txt')] # list
(len(x) for x in open('my_file.txt')) # generator expression
```
----
### 33: Compose Multiple Generators with `yield from`
`yield from <other generator>` iterate and yield results in that generator
```python=
def move(period, speed):
for _ in range(period):
yield speed
def pause(delay):
for _ in range(delay):
yield 0
def animate_composed():
yield from move(4, 5.0)
yield from pause(3)
yield from move(2, 3.0)
```
```python=
> run(animate_composed)
Delta: 5.0
Delta: 5.0
Delta: 5.0
Delta: 5.0
Delta: 0.0
Delta: 0.0
Delta: 0.0
Delta: 3.0
Delta: 3.0
```
<aside class="notes">
yield from 可以從別的 generator 抓結果，該 generator 結束後，才往下走<br/>
rendor 藏起來了，總之就是會調控速度
</aside>
----
### 34: Avoid Injecting Data into Generators with `send`
`send` upgrades `yield` into a 2-way channel
```python=
# send.py
def my_generator():
received = yield 1
print(f'received: {received}')
it = my_generator()
output = it.send(None) # 1st valid value of `send` is `None`
print(f'output: {output}')
try:
while (output := it.send('hello')):
print('will not come here')
except StopIteration:
pass
```
```
$ python send.py
output: 1
received: hello
```
<aside class="notes">
我猜八成人不知道 send.... 這邊就是要先跟你說，有這種用法，然後叫你不要用 XD<br />
第一個 send 必須要給 None，因為 generator 剛初始化時，還沒有碰到任何的 yield，所以無法接收變數<br />
第一個 send 會讓 generator 跑到第一個 yield，而第二次 send 才會傳給第一次 yield<br />
沒有 send 直接 next() ，就當拿到 None
</aside>
----
with composed generator: unexcepted `None`?
```python=
def step_mul(steps):
amplitude = yield
for step in range(steps):
amplitude = yield amplitude * step
def complex_step_mul():
yield from step_mul(3)
yield from step_mul(4)
def run_modulating(it):
amplitudes = [None, 7, 7, 7, 2, 2, 2, 2]
for amplitude in amplitudes:
output = it.send(amplitude)
print(f'Output: {output}')
```
```python=
> run_modulating(complex_step_mul())
Output: None
Output: 0
Output: 7
Output: 14
Output: None # ??
Output: 0
Output: 2
Output: 4
```
<aside class="notes">
surprise! compose 後還是會出現 none
</aside>
----
solution: pass another "amplitude" iterator to generator function:
```python=
def step_mul(amplitude_it, steps):
for step in range(steps):
amplitude = next(amplitude_it)
yield amplitude * step
def complex_step_mul(amplitude_it):
yield from step_mul(amplitude_it, 3)
yield from step_mul(amplitude_it, 4)
def run_cascading():
amplitudes = [7, 7, 7, 2, 2, 2, 2]
it = complex_step_mul(iter(amplitudes))
for _ in amplitudes:
output = next(it)
print(f'Output: {output}')
```
```python=
> run_cascading()
Output: 0
Output: 7
Output: 14
Output: 0
Output: 2
Output: 4
Output: 6
```
<aside class="notes">
用另一個 iterator 來做雙向的 channel
</aside>
----
### 35: Avoid Causing State Transitions in Generators with `throw`
`iterator.throw()` raise exception on next yield
```python=
class MyError(Exception):
pass
def my_generator():
yield 1
yield 2
yield 3
```
```python=
it = my_generator()
next(it) # Yield 1
next(it) # Yield 2
it.throw(MyError('test error')) # MyError: test error
```
----
catch exceptions to make state transition: (harder to read)
```python=
def timer(period):
current = period
while current:How
current -= 1
try:
yield current
except Reset:
current = period
```
instead, write a container class with `__iter__`:
```python=
class Timer:
def __init__(self, period):
self.current = period
self.period = period
# reset timer by call reset method
def reset(self):
self.current = self.period
def __iter__(self):
while self.current:
self.current -= 1
yield self.current
```
<aside class="notes">
可以 try catch 抓 error type 做 state transition 但不要這樣做 XD
</aside>
---
## `itertools`
https://docs.python.org/3/library/itertools.html
### 36: Consider `itertools` for Working with Iterators and Generators
----
### Linking iterators
```python=
it1 = [1, 2, 3, 4]
it2 = [5, 6, 7, 8]
it3 = [5, 6, 7]
```
```python=
> itertools.chain(it1, it2)
[1, 2, 3, 4, 5, 6, 7, 8]
> itertools.repeat('hello', 3)
['hello', 'hello', 'hello']
# cycle: repeat itself forever
> it = itertools.cycle(it1)
> [next(it) for _ in range(9)]
[1, 2, 3, 4, 1, 2, 3, 4, 1]
```
```python=
# zip_longest: zip with padding
> zip(it1, it3)
[(1, 5), (2, 6), (3, 7)]
> itertools.zip_longest(it1, it3, fillvalue='NA')
[(1, 5), (2, 6), (3, 7), (4, 'NA')]
# tee: parallel iterators
> it4, it5 = itertools.tee(it1, 2)
> it4
[1, 2, 3, 4]
> it5
[1, 2, 3, 4]
```
(all `list` are dropped)
----
### Filtering items
```python=
values = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
```
```python=
# islice
> itertools.islice(values, 5) # end (inclusive!)
[1, 2, 3, 4, 5]
> itertools.islice(values, 2, 4) # start + end
[3, 4]
> itertools.islice(values, 1, 7, 2) # start + end + step
[2, 4, 6]
```
```python=
# takewhile / dropwhile
> itertools.takewhile(lambda x: x < 4, values)
[1, 2, 3]
> itertools.dropwhile(lambda x: x > 4, values)
[4, 5, 6, 7, 8, 9, 10]
# filterfalse
> filter(lambda x: x % 3 == 0, values)
[3, 6, 9]
> itertools.filterfalse(lambda x: x % 3 == 0, values)
[1, 2, 4, 5, 7, 8, 10]
```
----
### Producing combinations of items from iterators
```python=
# accumulate
> itertools.accumulate(values)
[1, 3, 6, 10, 15, 21, 28, 36, 45, 55]
> itertools.accumulate(values, lambda x, y: (x + y) % 20)
[1, 3, 6, 10, 15, 1, 8, 16, 5, 15]
# product: Cartesian product
> it1 = 'abc'
> it2 = [1, 2]
> itertools.product(it1, it2)
[('a', 1), ('a', 2), ('b', 1), ('b', 2), ('c', 1), ('c', 2)]
> itertools.product(it2, repeat=3) # self prodcut
[(1, 1, 1), (1, 1, 2), (1, 2, 1), (1, 2, 2), (2, 1, 1), (2, 1, 2), (2, 2, 1), (2, 2, 2)]
```
```python=
# Permutations and Combinations
> itertools.permutations(it1, 2)
[('a', 'b'), ('a', 'c'), ('b', 'a'), ('b', 'c'), ('c', 'a'), ('c', 'b')]
> itertools.combinations(it1, 2)
[('a', 'b'), ('a', 'c'), ('b', 'c')]
> itertools.combinations_with_replacement(it1, 2)
[('a', 'a'), ('a', 'b'), ('a', 'c'), ('b', 'b'), ('b', 'c'), ('c', 'c')]
```
<aside class="notes">
排列組合三大天王 P / C / H
</aside>
---
# The End
Thanks for listening!
- [back to outline](#/1)
{"metaMigratedAt":"2023-06-16T19:51:26.866Z","metaMigratedFrom":"YAML","title":"Effective Python Chp 4: Comprehensions and Generators","breaks":true,"slideOptions":"{\"height\":1000,\"width\":1500,\"theme\":\"white\"}","contributors":"[{\"id\":\"915f29e1-3f9c-4908-bbd4-a58795589e48\",\"add\":11125,\"del\":706}]"}