CODEWITHSUNDEEP
pythonapache-sparkhivepysparkapache-spark-sql
Aravind Krishnakumar
Aravind Krishnakumar

Reputation: 2777

Convert Hive Row Object to Integer Spark

I am trying to convert the output of a hive column into key value pair. 

sqlContext = HiveContext(sc)
id1 = sqlContext.sql("select instance_id from temp_table")
pairs1 = id1.map(lambda s: (int(s), 'Configuration'))

I am getting the below error

TypeError: int() argument must be a string or a number, not 'Row'

I am not sure how to typecast the Hive Row object into a integer so that I can apply the map function to that

For example, the id1 is a dataframe and when I apply collect() to that it returns

[Row(_c0=12616821)]

I need the extract the value from the Row Object. Please let me know if any solution related to this problem

Upvotes: 0

Views: 3066

Answers (1)

Aravind Krishnakumar
Aravind Krishnakumar

Reputation: 2777

I figured out a way to get the integer value from the Row Object. Initially I thought of applying typecast and converting it into int and few other methods.But it seems we can get the value by applying the index as simple as that

>> id1 = sqlContext.sql("select int(id) as id  from temp_table limit 1")
>> temp = df1.select('id').collect()

>> temp
[Row(id = 9331413)]

>> temp[0][0]
9331413

Upvotes: 5

Related Questions