Using Python and Spark Machine Learning to Do Classification

We’ve been writing about how to use Spark ML with the Scala programming language. But not many programmers know Scala. Python has moved ahead of Java in terms of number of users, largely based on the strength of machine learning. So, let’s turn our attention to using Spark ML with Python.

You could say that Spark is Scala-centric. Scala has both Python and Scala interfaces and command line interpreters. Scala is the default one. The Python one is called pyspark. The most examples given by Spark are in Scala and in some cases no examples are given in Python.

(This tutorial is part of our Apache Spark Guide. Use the right-hand menu to navigate.)

Apache Atom

Python is the preferred language to use for data science because of NumPy, Pandas, and matplotlib, which are tools that make working with arrays and drawing charts easier and can work with large arrays of data efficiently. But Spark is designed to work with enormous amount of data, spread across a cluster. It’s good practice to use both tools, switching back and forth, perhaps, as the demand warrants it.

But as we will see, because Spark dataframe is not the same as a Pandas dataframe, there is not 100% compatibility among all of these objects. You must convert Spark dataframes to lists and arrays and other structures in order to plot them with matplotlib. Because you can’t slice arrays using the familiar [:,4], it takes more code to do the same operation.

But the other issue is performance. Apache Atom exists to efficiently convert objects in java processes to python processes and vice versa. Spark is written in Java and Scala. Scala rides atop Java. Python, of course, runs in a Python process.

Arrow speeds up operations with as the conversion of Spark dataframes to Pandas dataframes and with column wise operations such as .withcolumn().

Spark discusses some of the issues around this and the config change you need to make in Spark to take advantage of this boost in performance in their Apache Arrow documentation.

Heart patient data

Download the data from the University of São Paolo data set, available here. If you are curious, see this discussion.

The columns are:

1. Age
2. Sex
3. Chest pain type (4 values)
4. Resting blood pressure
5. Serum cholesterol in mg/dl
6. Fasting blood sugar > 120 mg/dl
7. Resting electrocardiographic results (values 0,1,2)
8. Maximum heart rate achieved
9. Exercise induced angina
10. Oldpeak = ST depression induced by exercise relative to rest
11. Slope of the peak exercise ST segment
12. Number of major vessels (0-3) colored by fluoroscopy
13. Thal: 3 = normal; 6 = fixed defect; 7 = reversable defect