CODEWITHSUNDEEP
pythoncharacter-encodinggsutil
Francesco Pedrotti
Francesco Pedrotti

Reputation: 51

google gsutil: ascii codec can't encode character

I'm trying to sync my local folder on google cloud nearline storage, but I get the following error:

"Caught non-retryable exception while listing file:///var/lib/..: 'ascii' codec can't encode character u'\xe9' in position 170: ordinal not in range(128)
CommandException: Caught non-retryable exception - aborting rsync
Caught ^C - exiting"

Is there any solution please? Thank you.

Upvotes: 3

Views: 2225

Answers (2)

Harshavardhana
Harshavardhana

Reputation: 1428

Alternatively you can use https://github.com/minio/mc which supports Google Cloud Storage in S3 compatibility mode.

mc is fully UTF-8 , can upload and download with all known UTF-8 characters.

Get your AccessKeyID and SecretAccessKey by following Google Credentials Guide. Once you have them update your ~/./mc/config.json configuration file.

$ mc config host add storage.googleapis.com <your_access_key> <your_secret_key> S3v2

NOTE: Google Cloud Storage only supports Legacy Signature Version 2 :(, so you have to pick - S3v2

Upvotes: 1

Jordan
Jordan

Reputation: 703

It looks like you have an accented e - é located in a filename that you are attempting to sync.

Using UTF-8 for all object names and filenames will ensure that gsutil doesn’t encounter character encoding errors while operating on the files. Unfortunately, it’s still possible that files uploaded / downloaded this way can have interoperability problems, for a number of reasons unrelated to gsutil; example: copying filenames across different platforms like Linux and Windows. If you are using the Windows console you can specify wildcards or folders (using the -R option) rather than explicitly named individual files in order to prevent this.

If you have too many files you can use a tool to convert the old character encoding to UTF-8. One such tool is native2ascii.

Upvotes: 1

Related Questions