CODEWITHSUNDEEP
pythonpandashadooputf-8pyodbc
Alexis.Rolland
Alexis.Rolland

Reputation: 6363

Python Dictionary Contains Encoded Values

I have a pandas data frame oParameterData which I have built querying on Hadoop using Hive ODBC connection. I am using it to populate a Python dictionary called oParameter

import pyodbc
import pandas

oConnexionString = 'Driver={ClouderaHive};[...]'
oConnexion = pyodbc.connect(oConnexionString, autocommit=True)
oConnexion.setencoding(encoding='utf-8')
oQueryParameter = "select * from my_db.my_table;"
oParameterData = pandas.read_sql(oQueryParameter, oConnexion)
oCursor = oConnexion.cursor()

for oRow in oParameterData.index:
    oParameter = {}
    oParameter['pTableName'] = oParameterData.loc[oRow,'game']
    oParameter['pDataPartition'] = oParameterData.loc[oRow,'partition']
    oParameter['pDataLocation'] = oParameterData.loc[oRow,'data_path']
    oParameter['pAvroSchemaURL'] = oParameterData.loc[oRow,'schema_path']

When I print the whole dictionary I have the following:

>>> print(oParameter)
>>> {'pDataLocation': '/\x00d\x00a\x00t\x00a\x00/\x00d\x00a\x00t\x00a\x00l\x00a\x00k\x00e\x00/\x00t\x00m\x00p\x00/\x00k\x00a\x00f\x00k\x00a\x00d\x00u\x00m\x00p\x00e\x00r\x00/\x00d\x00a\x00t\x00a\x00/\x00H\x00e\x00r\x00o\x00/\x00c\x00o\x00n\x00t\x00e\x00x\x00t\x00.\x00s\x00t\x00a\x00r\x00t\x00.\x00G\x00a\x00m\x00e\x00M\x00o\x00d\x00e\x00\x00/\x00v\x00=\x001\x00.\x00x\x00', 'pAvroSchemaURL': '/\x00d\x00a\x00t\x00a\x00/\x00d\x00a\x00t\x00a\x00l\x00a\x00k\x00e\x00/\x00t\x00m\x00p\x00/\x00k\x00a\x00f\x00k\x00a\x00d\x00u\x00m\x00p\x00e\x00r\x00/\x00d\x00a\x00t\x00a\x00/\x00H\x00e\x00r\x00o\x00/\x00c\x00o\x00n\x00t\x00e\x00x\x00t\x00.\x00s\x00t\x00a\x00r\x00t\x00.\x00G\x00a\x00m\x00e\x00M\x00o\x00d\x00e\x00\x00/\x00c\x00o\x00n\x00t\x00e\x00x\x00t\x00.\x00s\x00t\x00a\x00r\x00t\x00.\x00G\x00a\x00m\x00e\x00M\x00o\x00d\x00e\x00_\x001\x00.\x00x\x00.\x00a\x00v\x00s\x00c\x00', 'pTableName': 'h\x00e\x00r\x00o\x00_c\x00o\x00n\x00t\x00e\x00x\x00t\x00', 'pDataPartition': 'd\x00t\x00'}

But when I print Keys and Values one by one they display properly:

>>> print(oParameter['pTableName'])
>>> 'hero_game_context_gamemode'
>>> print(oParameter['pDataPartition'])
>>> 'dt'

Could you please explain why and how to have the dictionary properly encoded? I am using these parameters in subsequent queries described here: Hive ParseException in Drop Table Statement and I am guessing the queries fail due to this encoding issue.

Upvotes: 3

Views: 234

Answers (1)

Alexis.Rolland
Alexis.Rolland

Reputation: 6363

After investigating further, I found out the encoding was not correctly set when connecting to Hadoop using pyodbc.

I was connecting like this:

import pyodbc
import pandas

oConnexionString = 'Driver={ClouderaHive};[...]'
oConnexion = pyodbc.connect(oConnexionString, autocommit=True)
oConnexion.setencoding(encoding='utf-8')

I changed to connect like this:

import pyodbc
import pandas

oConnexionString = 'Driver={ClouderaHive};[...]'
oConnexion = pyodbc.connect(oConnexionString, autocommit=True)
oConnexion.setdecoding(pyodbc.SQL_CHAR, encoding='utf-8')
oConnexion.setdecoding(pyodbc.SQL_WCHAR, encoding='utf-8')
oConnexion.setencoding(encoding='utf-8')

Now when I build my dictionary from the data frame it displays properly.

Upvotes: 1

Related Questions