Apache Beam: Flatmap vs Map?
I Want to Understand in Which Scenario That I Should Use Flatmap or Map. the Documentation Did Not Seem Clear to Me. I Still Do Not Understand in Which...
I want to understand in which scenario that I should use FlatMap or Map. The documentation did not seem clear to me.
I still do not understand in which scenario I should use the transformation of FlatMap or Map.
Could someone give me an example so I can understand their difference?
I understand the difference of FlatMap vs Map in Spark, and however not sure if there any similarity?
3 Answers
These transforms in Beam are exactly same as Spark (Scala too).
A Map transform, maps from a PCollection of N elements into another PCollection of N elements.
A FlatMap transform maps a PCollections of N elements into N collections of zero or more elements, which are then flattened into a single PCollection.
As a simple example, the following happens:
beam.Create([1, 2, 3]) | beam.Map(lambda x: [x, 'any'])
# The result is a collection of THREE lists: [[1, 'any'], [2, 'any'], [3, 'any']]
Whereas:
beam.Create([1, 2, 3]) | beam.FlatMap(lambda x: [x, 'any'])
# The lists that are output by the lambda, are then flattened into a
# collection of SIX single elements: [1, 'any', 2, 'any', 3, 'any']
Let me show you one example
import apache_beam as beam
def categorize_explode(text):
result = text.split(':')
category = result[0]
elements = result[1].split(',')
return list(map(lambda x: (category, x), elements))
with beam.Pipeline() as pipeline:
things = (
pipeline
| 'Categories and Elements' >> beam.Create(["Vehicles:Car,Jeep,Truck,BUS,AIRPLANE","FOOD:Burger,SANDWICH,ICECREAM,APPLE"])
| 'Explode' >> beam.FlatMap(categorize_explode)
| beam.Map(print)
)
As you can see categorize_explode function splits the strings into categories and corresponding elements and returns iterator like [('Vehicles','Car'),('Vehicles','Jeep'),...]
FlatMap takes each element in this iterator and treats each element as a separate element in PCollection.
So the result would be:
('Vehicles', 'Car')
('Vehicles', 'Jeep')
('Vehicles', 'Truck')
('Vehicles', 'BUS')
('Vehicles', 'AIRPLANE')
('FOOD', 'Burger')
('FOOD', 'SANDWICH')
('FOOD', 'ICECREAM')
('FOOD', 'APPLE')
While Map performs one to one mapping. i.e. this iterator [('Vehicles','Car'),('Vehicles','Jeep'),...] would be returned as it is.
So the result would be for Map:
[('Vehicles', 'Car'), ('Vehicles', 'Jeep'), ('Vehicles', 'Truck'), ('Vehicles', 'BUS'), ('Vehicles', 'AIRPLANE')]
[('FOOD', 'Burger'), ('FOOD', 'SANDWICH'), ('FOOD', 'ICECREAM'), ('FOOD', 'APPLE')]
The approach I have used is somewhat similar to spark explode transform.
Hope this helps!!!
In simplest word,
Map transformation is "one to one" mapping on each element of list/collection. Ex -
{"Amar", "Akabar", "Anthony"} -> {"Mr.Amar", "Mr.Akabar", "Mr.Anthony"}
FlatMap transformation is usually on collection like "list of list", and this collection gets flattened to single list and transformation/mapping is applied on each element of "list of list"/collection.
FlatMap transformation Ex -
{ {"Amar", "Akabar"}, "Anthony"} -> {"Mr.Amar", "Mr.Akabar", "Mr.Anthony"}
This concept remains same across programming language and across platform.
Hope it helps.
Marcus Vance is a cybersecurity auditor and technology writer dedicated to educating the public about online safety, data privacy regulations, enterprise security, and emerging cyber threats.
