Pyspark: Create More Than 1 Dataframe Fails

I'd like to convert several large Pandas dataframes into Spark dataframes and then manipulate and merge them, as follows:

import pandas as pd
from pyspark import SparkContext,SQLContext

df1 = pd.read_csv('data1.cat',delim_whitespace=True)
df2 = pd.read_csv('data2.cat',delim_whitespace=True)

sc = SparkContext()
sql = SQLContext(sc)
spark_df1 = sql.createDataFrame(df1)
spark_df2 = sql.createDataFrame(df2)

But something goes wrong, and I get the following error:

  File "/home/user/anaconda3/envs/conda_py3.6.8/lib/python3.6/site-packages/pyspark/sql/context.py", line 307, in createDataFrame
    return self.sparkSession.createDataFrame(data, schema, samplingRatio, verifySchema)
  File "/home/user/anaconda3/envs/conda_py3.6.8/lib/python3.6/site-packages/pyspark/sql/session.py", line 724, in createDataFrame
    data = self._convert_from_pandas(data, schema, timezone)
  File "/home/user/anaconda3/envs/conda_py3.6.8/lib/python3.6/site-packages/pyspark/sql/session.py", line 487, in _convert_from_pandas
    np_records = pdf.to_records(index=False)
  File "/home/user/anaconda3/envs/conda_py3.6.8/lib/python3.6/site-packages/pandas/core/frame.py", line 1839, in to_records
    return np.rec.fromarrays(arrays, dtype={"names": names, "formats": formats})
  File "/home/user/.local/lib/python3.6/site-packages/numpy/core/records.py", line 617, in fromarrays
    descr = sb.dtype(dtype)
ValueError: name already used as a name or title

Is it possible to create multiple Spark dataframes in the same session like this?

3

1 Answer

The error is thrown because your pandas DataFrame (df1 of df2) has the same column name for 2 or more columns. Either change the columns name or drop it.

Thanks, Sagar

1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Chloe Bennett

Chloe Bennett

Culture, Media & Entertainment Columnist

Chloe Bennett explores the intersection of pop culture, streaming entertainment, digital trends, and contemporary lifestyle. Her weekly commentary reaches thousands of culture enthusiasts.

Share this article
Twitter Facebook Pinterest