Explode Array Data into Rows in Spark [Duplicate]

Question

I have a dataset in the following way:

FieldA    FieldB    ArrayField
1         A         {1,2,3}
2         B         {3,5}

I would like to explode the data on ArrayField so the output will look in the following way:

FieldA    FieldB    ExplodedField
1         A         1
1         A         2
1         A         3
2         B         3
2         B         5

I mean I want to generate an output line for each item in the array the in ArrayField while keeping the values of the other fields.

How would you implement it in Spark. Notice that the input dataset is very large.

Accepted Answer

The explode function should get that done.

pyspark version:

>>> df = spark.createDataFrame([(1, "A", [1,2,3]), (2, "B", [3,5])],["col1", "col2", "col3"])
>>> from pyspark.sql.functions import explode
>>> df.withColumn("col3", explode(df.col3)).show()
+----+----+----+
|col1|col2|col3|
+----+----+----+
|   1|   A|   1|
|   1|   A|   2|
|   1|   A|   3|
|   2|   B|   3|
|   2|   B|   5|
+----+----+----+

Scala version

scala> val df = Seq((1, "A", Seq(1,2,3)), (2, "B", Seq(3,5))).toDF("col1", "col2", "col3")
df: org.apache.spark.sql.DataFrame = [col1: int, col2: string ... 1 more field]

scala> df.withColumn("col3", explode($"col3")).show()
+----+----+----+
|col1|col2|col3|
+----+----+----+
|   1|   A|   1|
|   1|   A|   2|
|   1|   A|   3|
|   2|   B|   3|
|   2|   B|   5|
+----+----+----+

Answer 2

You can use explode function Below is the simple example for your case

import org.apache.spark.sql.functions._
import spark.implicits._

  val data = spark.sparkContext.parallelize(Seq(
    (1, "A", List(1,2,3)),
    (2, "B", List(3, 5))
  )).toDF("FieldA", "FieldB", "FieldC")

    data.withColumn("ExplodedField", explode($"FieldC")).drop("FieldC")

Hope this helps!

Answer 3

explode does exactly what you want. Docs:

Also, here is an example from a different question using it:

Explode Array Data into Rows in Spark [Duplicate]

Must Read

3 Answers

Sarah Jenkins