Convert a Spark Dataframe to Pandas Df
Is There a Way to Convert a Spark Df (Not Rdd) to Pandas Df I Tried the Following: Var Some_Df = Seq( ("A", "No"), ("B", "Yes"), ("B", "Yes"), ("B", "No") )...
Is there a way to convert a Spark Df (not RDD) to pandas DF
I tried the following:
var some_df = Seq(
("A", "no"),
("B", "yes"),
("B", "yes"),
("B", "no")
).toDF(
"user_id", "phone_number")
Code:
%pyspark
pandas_df = some_df.toPandas()
Error:
NameError: name 'some_df' is not defined
Any suggestions.
3 Answers
following should work
some_df = sc.parallelize([
("A", "no"),
("B", "yes"),
("B", "yes"),
("B", "no")]
).toDF(["user_id", "phone_number"])
pandas_df = some_df.toPandas()
In my case the following conversion from spark dataframe to pandas dataframe worked:
pandas_df = spark_df.select("*").toPandas()
Converting spark data frame to pandas can take time if you have large data frame. So you can use something like below:
spark.conf.set("spark.sql.execution.arrow.enabled", "true")
pd_df = df_spark.toPandas()
I have tried this in DataBricks.
Chloe Bennett explores the intersection of pop culture, streaming entertainment, digital trends, and contemporary lifestyle. Her weekly commentary reaches thousands of culture enthusiasts.
