What Is the Difference Between Chain and Chain. From_Iterable in Itertools?
I Could Not Find Any Valid Example on the Internet Where I Can See the Difference Between Them and Why to Choose One over the Other. 6 Answers the First Takes...
I could not find any valid example on the internet where I can see the difference between them and why to choose one over the other.
6 Answers
The first takes 0 or more arguments, each an iterable, the second one takes one argument which is expected to produce the iterables:
from itertools import chain
chain(list1, list2, list3)
iterables = [list1, list2, list3]
chain.from_iterable(iterables)
but iterables can be any iterator that yields the iterables:
def gen_iterables():
for i in range(10):
yield range(i)
itertools.chain.from_iterable(gen_iterables())
Using the second form is usually a case of convenience, but because it loops over the input iterables lazily, it is also the only way you can chain an infinite number of finite iterators:
def gen_iterables():
while True:
for i in range(5, 10):
yield range(i)
chain.from_iterable(gen_iterables())
The above example will give you a iterable that yields a cyclic pattern of numbers that will never stop, but will never consume more memory than what a single range() call requires.
I could not find any valid example ... where I can see the difference between them [
chainandchain.from_iterable] and why to choose one over the other
The accepted answer is thorough. For those seeking a quick application, consider flattening several lists:
list(itertools.chain(["a", "b", "c"], ["d", "e"], ["f"]))
# ['a', 'b', 'c', 'd', 'e', 'f']
You may wish to reuse these lists later, so you make an iterable of lists:
iterable = (["a", "b", "c"], ["d", "e"], ["f"])
Attempt
However, passing in an iterable to chain gives an unflattened result:
list(itertools.chain(iterable))
# [['a', 'b', 'c'], ['d', 'e'], ['f']]
Why? You passed in one item (a tuple). chain needs each list separately.
Solutions
When possible, you can unpack an iterable:
list(itertools.chain(*iterable))
# ['a', 'b', 'c', 'd', 'e', 'f']
list(itertools.chain(*iter(iterable)))
# ['a', 'b', 'c', 'd', 'e', 'f']
More generally, use .from_iterable (as it also works with infinite iterators):
list(itertools.chain.from_iterable(iterable))
# ['a', 'b', 'c', 'd', 'e', 'f']
g = itertools.chain.from_iterable(itertools.cycle(iterable))
next(g)
# "a"
They do very similar things. For small number of iterables itertools.chain(*iterables) and itertools.chain.from_iterable(iterables) perform similarly.
The key advantage of from_iterables lies in the ability to handle large (potentially infinite) number of iterables since all of them need not be available at the time of the call.
Another way to see it:
chain(iterable1, iterable2, iterable3, ...) is for when you already know what iterables you have, so you can write them as these comma-separated arguments.
chain.from_iterable(iterable) is for when your iterables (like iterable1, iterable2, iterable3) are obtained from another iterable.
Extending @martijn-pieters answer
Although the access to the inner items in the iterable remains the same, and implementation wise,
itertools_chain_from_iterable(i.e.chain.from_iterablein Python) andchain_new(i.e.chainin Python)
in the CPython implementation, are both duck-types of chain_new_internal
Are there any optimization benefits from using chain.from_iterable(x), where x is an iterable of iterable; and the main purpose is to ultimately consume the flatten list of items?
We can try benchmarking it with:
import random
from itertools import chain
from functools import wraps
from time import time
from tqdm import tqdm
def timing(f):
@wraps(f)
def wrap(*args, **kw):
ts = time()
result = f(*args, **kw)
te = time()
print('func:%r args:[%r, %r] took: %2.4f sec' % (f.__name__, args, kw, te-ts))
return result
return wrap
def generate_nm(m, n):
# Creates m generators of m integers between range 0 to n.
yield iter(random.sample(range(n), n) for _ in range(m))
def chain_star(x):
# Stores an iterable that will unpack and flatten the list of list.
chain_x = chain(*x)
# Consumes the items in the flatten iterable.
for i in chain_x:
pass
def chain_from_iterable(x):
# Stores an iterable that will unpack and flatten the list of list.
chain_x = chain.from_iterable(x)
# Consumes the items in the flatten iterable.
for i in chain_x:
pass
@timing
def versus(f, n, m):
f(generate_nm(n, m))
P/S: Benchmark running... Waiting for the results.
Results
Must Read
chain_star, m=1000, n=1000
for _ in range(10):
versus(chain_star, 1000, 1000)
[out]:
func:'versus' args:[(<function chain_star at 0x7f5c7188ef28>, 1000, 1000), {}] took: 0.6494 sec
func:'versus' args:[(<function chain_star at 0x7f5c7188ef28>, 1000, 1000), {}] took: 0.6603 sec
func:'versus' args:[(<function chain_star at 0x7f5c7188ef28>, 1000, 1000), {}] took: 0.6367 sec
func:'versus' args:[(<function chain_star at 0x7f5c7188ef28>, 1000, 1000), {}] took: 0.6350 sec
func:'versus' args:[(<function chain_star at 0x7f5c7188ef28>, 1000, 1000), {}] took: 0.6296 sec
func:'versus' args:[(<function chain_star at 0x7f5c7188ef28>, 1000, 1000), {}] took: 0.6399 sec
func:'versus' args:[(<function chain_star at 0x7f5c7188ef28>, 1000, 1000), {}] took: 0.6341 sec
func:'versus' args:[(<function chain_star at 0x7f5c7188ef28>, 1000, 1000), {}] took: 0.6381 sec
func:'versus' args:[(<function chain_star at 0x7f5c7188ef28>, 1000, 1000), {}] took: 0.6343 sec
func:'versus' args:[(<function chain_star at 0x7f5c7188ef28>, 1000, 1000), {}] took: 0.6309 sec