CODEWITHSUNDEEP
javapythonapache-sparkpyspark
milva
milva

Reputation: 161

How to fix Py4JJavaError while using .toPandas() function?

I'm new at pyspark and I'm trying to use word_tokenize() function. This is my code:

import nltk
from nltk import word_tokenize
import pandas as pd

df_pd = df2.select("*").toPandas()
df2.select('text').apply(word_tokenize)
df_pd.show()

I use JDK 1.8, Python 3.7, spark 2.4.3.

Could you please tell me what I'm doing wrong? How to fix it? The code below that piece is working well, without any errors.

I got such message: 

Py4JJavaError: An error occurred while calling o106.collectToPython.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 14.0 failed 1 times, most recent failure: Lost task 0.0 in stage 14.0 (TID 330, localhost, executor driver): java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:3236)
    at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
    at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
    at org.apache.spark.util.ByteBufferOutputStream.write(ByteBufferOutputStream.scala:41)
    at java.io.ObjectOutputStream$BlockDataOutputStream.write(ObjectOutputStream.java:1853)
    at java.io.ObjectOutputStream.write(ObjectOutputStream.java:709)
    at org.apache.spark.util.Utils$.writeByteBuffer(Utils.scala:260)
    at org.apache.spark.scheduler.DirectTaskResult$$anonfun$writeExternal$1.apply$mcV$sp(TaskResult.scala:50)
    at org.apache.spark.scheduler.DirectTaskResult$$anonfun$writeExternal$1.apply(TaskResult.scala:48)
    at org.apache.spark.scheduler.DirectTaskResult$$anonfun$writeExternal$1.apply(TaskResult.scala:48)
    at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1326)
    at org.apache.spark.scheduler.DirectTaskResult.writeExternal(TaskResult.scala:48)
    at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1459)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1430)
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
    at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
    at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:43)
    at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:517)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)


 and more....

Upvotes: 0

Views: 1500

Answers (1)

Dafni Argyro Krystallidou
Dafni Argyro Krystallidou

Reputation: 81

toPandas is optimised for use with smaller datasets. As suggested it might be due to low memory, that you are getting the error.

try to limit your datasetsize: df_pd = df2.limit(10).select("*").toPandas()

apply your function, and then run .head(10) to eliminate your problem to a memory error.

Upvotes: 2

Related Questions