Pyspark: Create More Than 1 Dataframe Fails
I'd Like to Convert Several Large Pandas Dataframes into Spark Dataframes and Then Manipulate and Merge Them, as Follows: Import Pandas as Pd from Pyspark...
I'd like to convert several large Pandas dataframes into Spark dataframes and then manipulate and merge them, as follows:
import pandas as pd
from pyspark import SparkContext,SQLContext
df1 = pd.read_csv('data1.cat',delim_whitespace=True)
df2 = pd.read_csv('data2.cat',delim_whitespace=True)
sc = SparkContext()
sql = SQLContext(sc)
spark_df1 = sql.createDataFrame(df1)
spark_df2 = sql.createDataFrame(df2)
But something goes wrong, and I get the following error:
File "/home/user/anaconda3/envs/conda_py3.6.8/lib/python3.6/site-packages/pyspark/sql/context.py", line 307, in createDataFrame
return self.sparkSession.createDataFrame(data, schema, samplingRatio, verifySchema)
File "/home/user/anaconda3/envs/conda_py3.6.8/lib/python3.6/site-packages/pyspark/sql/session.py", line 724, in createDataFrame
data = self._convert_from_pandas(data, schema, timezone)
File "/home/user/anaconda3/envs/conda_py3.6.8/lib/python3.6/site-packages/pyspark/sql/session.py", line 487, in _convert_from_pandas
np_records = pdf.to_records(index=False)
File "/home/user/anaconda3/envs/conda_py3.6.8/lib/python3.6/site-packages/pandas/core/frame.py", line 1839, in to_records
return np.rec.fromarrays(arrays, dtype={"names": names, "formats": formats})
File "/home/user/.local/lib/python3.6/site-packages/numpy/core/records.py", line 617, in fromarrays
descr = sb.dtype(dtype)
ValueError: name already used as a name or title
Is it possible to create multiple Spark dataframes in the same session like this?
1 Answer
The error is thrown because your pandas DataFrame (df1 of df2) has the same column name for 2 or more columns. Either change the columns name or drop it.
Thanks, Sagar