CODEWITHSUNDEEP
ibm-cloudobject-storage
arjan kroon
arjan kroon

Reputation: 875

How do I do bulk file storage with IBM Object Storage?

I'm using IBM Object Storage to store huge amounts of very small files, say more than 1500 small files in one hour. (Total size of the 1500 files is about 5 MB) I'm using the object store api to post the files, one file at a time.

The problem is that for storing 1500 small files it takes about 15 minutes in total. This is with setting up and closing the connection with the object store.

Is there a way to do a sort of bulk post, to send more than one file in one post?

Regards,

Upvotes: 1

Views: 727

Answers (1)

Sanjay.Joshi
Sanjay.Joshi

Reputation: 304

Look at the archive-auto-extract feature available within Openstack Swift (Bluemix Object Storage). I assume that you are familiar with obtaining the X-Auth-Token and Storage_URL from Bluemix object storage. If not, my post about large file manifests explains the process. From the doc, the constraints include:

  • You must use the tar utility to create the tar archive file.
  • You can upload regular files but you cannot upload other items (for example, empty directories or symbolic links).
  • You must UTF-8-encode the member names.

Basic steps would be:

  1. Confirm that IBM Bluemix supports this feature by viewing info details for the service @ https://dal.objectstorage.open.softlayer.com/info . You'll see a JSON section within the response similar to:
    "bulk_upload": {
          "max_failed_extractions": 1000,
          "max_containers_per_extraction": 10000
    }
  1. Create a tar archive of your desired file set. tar gzip is most common.

  2. Upload this tar archive to object storage with a special parameter that tells swift to auto-extract the contents into the container for you.

    PUT /v1/AUTH_myaccount/my_backups/?extract-archive=tar.gz

    From the docs: To upload an archive file, make a PUT request. Add the extract-archive=format query parameter to indicate that you are uploading a tar archive file instead of normal content. Include within the request body the contents of the local file backup.tar.gz.

    Something like:

    AUTH_myaccount/my_backups/etc/config1.conf
AUTH_myaccount/my_backups/etc/cool.jpg
AUTH_myaccount/my_backups/home/john/bluemix.conf
...

  3. Inspect the results. Any top-level directory in the archive should create a new container in your Swift object-storage account.

Voila! Bulk upload. Hope this helps.

Upvotes: 1

Related Questions