CODEWITHSUNDEEP
apache-sparkpysparkapache-spark-sql
Sophie Dinka
Sophie Dinka

Reputation: 83

object has no attribute 'map' error in pyspark 2.4.4

i am running spark 2.4.4 with python 2.7 and IDE is pycharm.

The Input file contain encoded value in some column like given below.

.ʽ|!3-2-704A------------ (dotted line is space)

i am trying to get result like

3-2-704A

I tried below code.

from pyspark.sql import SparkSession
spark = SparkSession \
    .builder \
    .appName("Python Spark") \
    .config("spark.some.config.option", "some-value") \
    .getOrCreate()

df = spark.read.csv("Customers_v01.csv",header=True,sep=",");
myres = df.map(lambda x :x[1].decode('utf-8'))

print(myres.collect())

Error:

    myres = df.map(lambda x :x[1].decode('utf-8'))
  File "C:\spark\python\pyspark\sql\dataframe.py", line 1301, in __getattr__
    "'%s' object has no attribute '%s'" % (self.__class__.__name__, name))
AttributeError: 'DataFrame' object has no attribute 'map'

i am not sure what cause this error.... kindly help. is there any other way to do it.

Upvotes: 0

Views: 372

Answers (1)

oetzi
oetzi

Reputation: 1052

map is available on Resilient Distributed Dataset (RDD)

from pyspark.sql import SparkSession




spark = SparkSession.builder.appName("Python Spark").getOrCreate()

df = spark.read.csv("Customers_v01.csv", header=True, sep=",", encoding='utf-8')
myres = df.rdd.map(lambda x: x[1].encode().decode('utf-8'))

print(myres.collect())

Upvotes: 1

Related Questions