CODEWITHSUNDEEP
apache-sparkpyspark
Kelo
Kelo

Reputation: 1893

Unable to run pyspark 3.0.0 or pyspark 2.4.6 on MacOSx (with code that works on linux)

The python code I'm running:

# read input data -- search data for departure months (columns) across shopping days (rows)
df = pd.read_csv('test_arima_states_input.csv', header=0)

# format date columns
df['date'] = pd.to_datetime(df.date, format='%m/%d/%Y')
df['date'] = df['date'].dt.strftime('%m/%d/%Y')
state_data = df[['State', 'date', 'Jul-20']]

spark_df = spark.createDataFrame(state_data)

results = spark_df.groupby('State').apply(run_arima_fcst)

results.show()

This error persistently comes up no matter what I try.

py4j.protocol.Py4JJavaError: An error occurred while calling o59.showString.
: java.lang.IllegalAccessError: class io.netty.buffer.UnsafeDirectLittleEndian cannot access 
its superclass io.netty.buffer.WrappedByteBuf

I've reinstalled python and pyspark numerous times and all of the pip packages. There aren't any additional JDK's on my machine other than the Java 8 version.

Upvotes: 0

Views: 143

Answers (1)

Graeme Wallace
Graeme Wallace

Reputation: 36

Turns out the way to fix it, is to wipe your Mac clean and reinstall from scratch....bit excessive.

Upvotes: 1

Related Questions