CODEWITHSUNDEEP
pythonibm-cloudobject-storagedata-science-experience
Ross Lewis
Ross Lewis

Reputation: 785

How do I create a connection to Object Storage on Bluemix in a Data Science Experience Project?

I'm attempting to set up a connection to a Bluemix Object Storage for a project that is different than the default one the project created. This is a problem because:

1) When I go to add a new connection, the Object Storage instance I want to use is not under data service.

2) When I go to add a Softlayer Object Storage, the credentials I'm asked for are (Login URL, Access Key, and Secret Key), but the credentials I have for my instance are ("auth_url":"project":"projectId":"region":"userId":"username":"password":"domainId":"domainName":"role")

3) I have a good interface to a placeholder object storage, but I would like to replace it with the other instance.

Please help me access the data in a different Bluemix Object Storage instance other than the one attached to the project by default.

Upvotes: 1

Views: 760

Answers (4)

Trent Gray-Donald
Trent Gray-Donald

Reputation: 2346

I strongly recommend you look at https://github.com/ibm-cds-labs/ibmos2spark (which works for Python, R and Scala).

For Python + SoftLayer credentials it would specifically be this code:

slos = ibmos2spark.softlayer(sc, configuration_name, auth_url, tenant, username, password) data = sc.textFile(slos.url(container_name, object_name))

(taken from https://github.com/ibm-cds-labs/ibmos2spark/tree/master/python#softlayer )

The next question is how to load .mat files - which seems like a detour through Read .mat files in Python and using "sc.binaryFiles()" to get them into memory first.

Upvotes: 1

charles gomes
charles gomes

Reputation: 2155

In addition to what @Sumit Goyal Answered. You need to download the file in local gpfs in order to use apis or libraries that don't support reading from swift object storage or in other words only support reading from local storage/file system.

objStorCred = { "auth_url": "https://identity.open.softlayer.com", "project": "object_storage_XXXXX", "projectId": "XXXXX5a3", "region": "dallas", "userId": "XXXXXX98a15e0", "username": "admin_fXXXXX9", "password": "XXXXX", "domainId": "aXXXX5a", "domainName": "XXXX", "role": "admin" }

from io import StringIO import requests import json import pandas as pd

# @hidden_cell

# This function accesses a file in your Object Storage. The definition contains your credentials.

# You might want to remove those credentials before you share your notebook.

def get_object_storage_file(container, filename):

"""This functions returns a StringIO object containing the file content from Bluemix Object Storage."""

url1 = ''.join(['https://identity.open.softlayer.com', '/v3/auth/tokens'])
data = {'auth': {'identity': {'methods': ['password'],
        'password': {'user': {'name': objStorCred['username'],'domain': {'id': objStorCred['domainId']},
        'password': objStorCred['password']}}}}}
headers1 = {'Content-Type': 'application/json'}
resp1 = requests.post(url=url1, data=json.dumps(data), headers=headers1)
resp1_body = resp1.json()
for e1 in resp1_body['token']['catalog']:
    if(e1['type']=='object-store'):
        for e2 in e1['endpoints']:
                    if(e2['interface']=='public'and e2['region']=='dallas'):
                        url2 = ''.join([e2['url'],'/', container, '/', filename])
s_subject_token = resp1.headers['x-subject-token']
headers2 = {'X-Auth-Token': s_subject_token, 'accept': 'application/json'}
resp2 = requests.get(url=url2, headers=headers2)
return resp2

Note the instead of getting a stringIO object, we get the response object.

Now you can use intermediate local storage to store the .mat file.

Then call this function.

r = get_object_storage_file("containerr1", "example.mat")

with open('example.mat', 'wb') as file:  
file.write(r.content)

Now read the file using h5py. You may need to install h5py using pip install h5py.

import h5py

f = h5py.File('example.mat') f.keys()

Thanks, Charles.

Upvotes: 2

Raphael K
Raphael K

Reputation: 2353

In R:

## Credentials and libraries to write to object storage

## Install necessary library
install_github('IBMDataScience/objectStoreR')
library('objectStoreR')  

## Provide Credentials (fill in with your details from Bluemix)
credentials <-list(auth_url = "https://identity.open.softlayer.com",
         project = "object_storage_d7a568f8_ac53_4bc4_8834_f0e9962068f9",
         project_id = "e0c826f12030487493z2df3957621744",
         region = "dallas",
         user_id = "694102a676ef4252u19492c45fbebc4b",
         domain_id = "47ea410d2b51478d9f119fade708fbefe4",
         domain_name =  "1004827",
         username = "admin_9c5c874ed726b5a41c7bb4f8b55f45e3e2c35778",
         password = "Tj^d9rZoDhy5eb]U",
         container = "mycontainer", 
         filename = "myfile.csv")

To write a file out to Object Storage:

## Status '201' is a successful signal
write.csv(outputDF,'myOutputFile.csv', row.names = F)
status <- objectStore.put(credentials,'myOutputFile.csv')
paste("Status for final output CSV:", status, sep = " ")

Similarly, to save a model object (note you have to change to file name in the list of credentials or create a second credentials variable):

saveRDS(object = finalMod, file = "myModel.rds")
status <- objectStore.put(credentials, "myModel.rds")
paste("Status for model object:", status, sep = " ")

Hope this helps!

Upvotes: 0

Sumit Goyal
Sumit Goyal

Reputation: 575

You can use the function generated by the insert to code feature and plug in the credentials from the other object storage. For example:

from io import StringIO
import requests
import json
import pandas as pd

# @hidden_cell
# This function accesses a file in your Object Storage. The definition contains your credentials.
# You might want to remove those credentials before you share your notebook.
def get_object_storage_file_with_credentials(container, filename):
"""This functions returns a StringIO object containing
the file content from Bluemix Object Storage."""

url1 = ''.join(['https://identity.open.softlayer.com', '/v3/auth/tokens'])
data = {'auth': {'identity': {'methods': ['password'],
        'password': {'user': {'name': 'admin_xxxx','domain': {'id': 'xxxxxxxxxxx'},
        'password': 'xxxxxxxxxx'}}}}}
headers1 = {'Content-Type': 'application/json'}
resp1 = requests.post(url=url1, data=json.dumps(data), headers=headers1)
resp1_body = resp1.json()
for e1 in resp1_body['token']['catalog']:
    if(e1['type']=='object-store'):
        for e2 in e1['endpoints']:
                    if(e2['interface']=='public'and e2['region']=='dallas'):
                        url2 = ''.join([e2['url'],'/', container, '/', filename])
s_subject_token = resp1.headers['x-subject-token']
headers2 = {'X-Auth-Token': s_subject_token, 'accept': 'application/json'}
resp2 = requests.get(url=url2, headers=headers2)
return StringIO(resp2.text)

Here, replace the values for user name, domain id and password from your next Bluemix Object Store Credentials. After that you can simply access the files from a container in that object storage by:

cars_df = pd.read_csv(get_object_storage_file_with_credentials('<containerName>', '<filename>.csv'))
cars_df.head()

Upvotes: 1

Related Questions