CODEWITHSUNDEEP
pythonhadoopapache-sparkibm-cloud
Suyeon Byun
Suyeon Byun

Reputation: 3

IBM Bluemix set_hadoop_config error

Whenever I try the setup procedure for Apache spark data analytics, I get this error.

In

def set_hadoop_config(credentials):
    prefix = "fs.swift.service." + credentials['name'] 
    hconf = sc._jsc.hadoopConfiguration()
    hconf.set(prefix + ".auth.url", credentials['auth_url']+'/v3/auth/tokens')
    hconf.set(prefix + ".auth.endpoint.prefix", "endpoints")
    hconf.set(prefix + ".tenant", credentials['project_id'])
    hconf.set(prefix + ".username", credentials['user_id'])
    hconf.set(prefix + ".password", credentials['password'])
    hconf.setInt(prefix + ".http.port", 8080)
    hconf.set(prefix + ".region", credentials['region'])
    hconf.setBoolean(prefix + ".public", True)

In

credentials['name'] = 'keystone'
set_hadoop_config(credentials)

Out

---------------------------------------------------------------------------
 NameError                                 Traceback (most recent call last)
<ipython-input-6-976c35e1d85e> in <module>()
----> 1 credentials['name'] = 'keystone'
      2 set_hadoop_config(credentials)

NameError: name 'credentials' is not defined

Does anyone know how to solve this problem? I am stuck

Upvotes: 0

Views: 106

Answers (1)

NSHUKLA
NSHUKLA

Reputation: 81

I think you are missing the credentials dictionary i.e you should pass the values for parameters of accessing Object Store service as follows: 

credentials = 
{
  'auth_uri':'',
  'global_account_auth_uri':'',
  'username':'admin_b055482b7febbd287d9020d65cdd55f5653d0ffb',
  'password':"XXXXXX",
  'auth_url':'https://identity.open.softlayer.com',
  'project':'object_storage_e5e45537_ea14_4d15_b90a_5fdd271ea402',
  'project_id':'7d7e5f2a83fe47e586b91f459d47169f',
  'region':'dallas',
  'user_id':'001c394e06d74b86a76a786615e358e2',
  'domain_id':'2df6373c549e49f8973fb6d22ab18c1a',
  'domain_name':'639347',
  'filename':'2015_SQL.csv',
  'container':'notebooks',
  'tenantId':'s322-e1e9acad6196b9-a1259eb961e2'
 }

If you are using the notebooks, you can get the above using the "Insert to Code" for the file listed under the data sources panel (right hand side).

To access the file , you will need Swift URI as follows:

raw_data = sc.textFile("swift://" + credentials['container'] + "." + credentials['name'] + "/" + credentials['filename'])

Upvotes: 1

Related Questions