CODEWITHSUNDEEP
pythondataframepysparkapache-spark-sql
Amine H
Amine H

Reputation: 21

pyspark : select columns with special characters from map type column

in a query to select some columns from dataframe, I have a column type : map, which has multiple attributes. I'm trying to select only few attributes from this column, but this is returning to me an error since some attributes contains special characters like '-' , other attributes of this column are working fine

example : 

sqlContext.sql("select colA, colB.attribute1 from schema.table")

however if i select an attribute that contains - , it fails

sqlContext.sql("select colA, colB.4-TRANSPORT from schema.table")

=> fails 

18/08/29 15:21:34 INFO ParseDriver: Parse Completed
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/hdp/2.5.3.16-1/spark/python/pyspark/sql/context.py", line 580, in sql
    return DataFrame(self._ssql_ctx.sql(sqlQuery), self)
  File "/usr/hdp/2.5.3.16-1/spark/python/lib/py4j-0.9-src.zip/py4j/java_gateway.py", line 813, in __call__
  File "/usr/hdp/2.5.3.16-1/spark/python/pyspark/sql/utils.py", line 51, in deco
    raise AnalysisException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.AnalysisException: u"cannot resolve 'TRANSPORT'

i tried double quote it etc , no luck so far ...

Upvotes: 2

Views: 3193

Answers (1)

vvg
vvg

Reputation: 6385

Backquotes will help.

sqlContext.sql("select colA, colB.`4-TRANSPORT` from schema.table")

updated based on comment.

Upvotes: 2

Related Questions