Error: Attributeerror: 'Dataframe' Object Has No Attribute '_Jdf'
I Want to Perform K-Fold Cross Validation Using Pyspark to Finetune the Parameters and I'm Using Pyspark. Ml. I Am Getting Attribute Error. Attributeerror...
I want to perform k-fold cross validation using pyspark to finetune the parameters and I'm using pyspark.ml. I am getting Attribute Error.
AttributeError: 'DataFrame' object has no attribute '_jdf'
I have tried initially using pyspark.mllib but was not able to succeed in performing k-fold cross validation
import pandas as pd
from pyspark import SparkConf, SparkContext
from pyspark.ml.classification import DecisionTreeClassifier
data=pd.read_csv("file:///SparkCourse/wdbc.csv", header=None)
type(data)
print(data)
conf = SparkConf().setMaster("local").setAppName("SparkDecisionTree")
sc = SparkContext(conf = conf)
# Create initial Decision Tree Model
dt = DecisionTreeClassifier(labelCol="label", featuresCol="features",
maxDepth=3)
# Train model with Training Data
dtModel = dt.fit(data)
# I expect the model to be trained but I'm getting the following error
AttributeError: 'DataFrame' object has no attribute '_jdf'
Note: I'm able to print the data. Error is in dtModel
3 Answers
Convert Panadas to Spark
from pyspark.sql import SQLContext
sc = SparkContext.getOrCreate()
sqlContext = SQLContext(sc)
spark_dff = sqlContext.createDataFrame(panada_df)
If a metric evaluation error you probably:
- Transformed using Spark on test set properly, then peeked using Pandas DF.
# Spark model, transformed test, converted to pandas df
predictions = model.transform(test)
predDF = predictions.toPandas()
predDF.head()
- Then tried:
eval_acc = MulticlassClassificationEvaluator(
labelCol='Label_index',
predictionCol='prediction',
metricName='accuracy'
)
# Evaluate Performance
acc = eval_acc.evaluate(predDF) # Error
print(f"accuracy: {acc}")
I forgot predDF is a Pandas DataFrame. Needed predictions because its a Spark Dataframe.
acc = eval_acc.evaluate(predictions) # Works
print(f"accuracy: {acc}")
I think it's because you need to use: spark.read, try this:
data = spark.read.option("header", True).csv(
"file:///SparkCourse/wdbc.csv"
)
Chloe Bennett explores the intersection of pop culture, streaming entertainment, digital trends, and contemporary lifestyle. Her weekly commentary reaches thousands of culture enthusiasts.
