CODEWITHSUNDEEP
openstack-swift
carpogoryanin
carpogoryanin

Reputation: 21

OpenStack Swift client for the fastest synchronisation of huge amount of files?

I have folder with a lot of files (~50k, 3Gb). I need to sync this folder recursively to my container in OpenStak Swift-like storage.

I have tried to use cli duck (cyberduck), but it crashes on huge list of files in prepare process. I am trying to use supload utility, but it is so slow :(

May be somebody recommends me the best approach (some cli better) for this situation?

Upvotes: 0

Views: 1511

Answers (2)

Marcelo Amorim
Marcelo Amorim

Reputation: 1682

Try http://rclone.org/docs/

It has "sync" operation and "Bandwidth limit".

If this not solve your speed problem, the root cause is not on the clients.

Upvotes: 1

crunchengine
crunchengine

Reputation: 46

You should use official python-swiftclient package and simply : 

# load your openstack credentials
source openrc.sh
cd path_to_directory_you_want_to_sync
# upload all the files recursively keeping good paths
swift upload --changed your_container *

Swift does not support rsync-like synchronization, but I use this little script to delete in the container the files you deleted locally and to upload new files without asking swift to compare each files : 

#!/bin/bash

cd $2
diff <(find * -type f -print | sort) <(swift list $1 | sort) | while read x; do
if [[ $x == \>* ]]; then
        echo "Need to delete ${x:2}"
        swift delete $1 "${x:2}"
    elif [[ $x == \<* ]]; then
        echo "Need to upload ${x:2}"
        swift upload $1 "${x:2}"
    fi
done
cd -

Use with :

./swift_sync.sh your_container directory_to_sync

Upvotes: 3

Related Questions