CODEWITHSUNDEEP
google-cloud-storagegsutil
fredrik
fredrik

Reputation: 10281

How to properly run gsutil from crontab?

This is my entry in /etc/crontab, CentOS 6.6:

0 0 */1 * * fredrik /home/fredrik/google-cloud-sdk/bin/gsutil -d -m rsync -r -C [src] [dst] &> [log]

And I'm getting this error: OSError: [Errno 13] Permission denied: '/.config'

The command runs fine if executed in the shell. I've noticed I cannot run 0 0 */1 * * fredrik gsutil ... without the full path to gsutil, so I'm assuming I'm missing something in the environment in which cron is running...?

Here's the full traceback:

Traceback (most recent call last):
  File "/home/fredrik/google-cloud-sdk/bin/bootstrapping/gsutil.py", line 68, in <module>
    bootstrapping.PrerunChecks(can_be_gce=True)
  File "/home/fredrik/google-cloud-sdk/bin/bootstrapping/bootstrapping.py", line 279, in PrerunChecks
    CheckCredOrExit(can_be_gce=can_be_gce)
  File "/home/fredrik/google-cloud-sdk/bin/bootstrapping/bootstrapping.py", line 167, in CheckCredOrExit
    cred = c_store.Load()
  File "/home/fredrik/google-cloud-sdk/bin/bootstrapping/../../lib/googlecloudsdk/core/credentials/store.py", line 195, in Load
    account = properties.VALUES.core.account.Get()
  File "/home/fredrik/google-cloud-sdk/bin/bootstrapping/../../lib/googlecloudsdk/core/properties.py", line 393, in Get
    return _GetProperty(self, _PropertiesFile.Load(), required)
  File "/home/fredrik/google-cloud-sdk/bin/bootstrapping/../../lib/googlecloudsdk/core/properties.py", line 618, in _GetProperty
    value = callback()
  File "/home/fredrik/google-cloud-sdk/bin/bootstrapping/../../lib/googlecloudsdk/core/properties.py", line 286, in <lambda>
    'account', callbacks=[lambda: c_gce.Metadata().DefaultAccount()])
  File "/home/fredrik/google-cloud-sdk/bin/bootstrapping/../../lib/googlecloudsdk/core/credentials/gce.py", line 179, in Metadata
    _metadata_lock.lock(function=_CreateMetadata, argument=None)
  File "/usr/lib64/python2.6/mutex.py", line 44, in lock
    function(argument)
  File "/home/fredrik/google-cloud-sdk/bin/bootstrapping/../../lib/googlecloudsdk/core/credentials/gce.py", line 178, in _CreateMetadata
    _metadata = _GCEMetadata()
  File "/home/fredrik/google-cloud-sdk/bin/bootstrapping/../../lib/googlecloudsdk/core/credentials/gce.py", line 73, in __init__
    _CacheIsOnGCE(self.connected)
  File "/home/fredrik/google-cloud-sdk/bin/bootstrapping/../../lib/googlecloudsdk/core/credentials/gce.py", line 186, in _CacheIsOnGCE
    config.Paths().GCECachePath()) as gcecache_file:
  File "/home/fredrik/google-cloud-sdk/bin/bootstrapping/../../lib/googlecloudsdk/core/util/files.py", line 465, in OpenForWritingPrivate
    MakeDir(full_parent_dir_path, mode=0700)
  File "/home/fredrik/google-cloud-sdk/bin/bootstrapping/../../lib/googlecloudsdk/core/util/files.py", line 44, in MakeDir
    os.makedirs(path, mode=mode)
  File "/usr/lib64/python2.6/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.6/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/.config'

Upvotes: 4

Views: 7751

Answers (5)

niainaLens
niainaLens

Reputation: 811

because cron is ran in a very limited environment, you need to source your .bash_profile to get your environment config.

* * * * * source ~/.bash_profile && your_cmd_here

Upvotes: 0

4 hours later
4 hours later

Reputation: 1

For anyone trying to manage images with gsutil from PHP running Apache -

Made a new directory called apache-shared and chgrp/chown'd www-data (or whichever user your Apache runs on, run "top" to check). Copied the .boto file into the directory and ran the following without issue:

shell_exec('export BOTO_CONFIG=/apache-shared/.boto && export PATH=/sbin:/bin:/usr/sbin:/usr/bin:/home/user/google-cloud-sdk/bin && gsutil command image gs://bucket');

Upvotes: -1

fredrik
fredrik

Reputation: 10281

Thanks to Mike and jterrace for helping me getting this working. In the end, I had to revise these environment variables: PATH, HOME, BOTO_CONFIG (except for any other default ones).

PATH=/sbin:/bin:/usr/sbin:/usr/bin:/home/fredrik/google-cloud-sdk/bin
HOME=/home/fredrik
BOTO_CONFIG="/home/fredrik/.config/gcloud/legacy_credentials/[your-email-address]/.boto"

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name command to be executed

  0  0 */1 *  * fredrik   gsutil -d -m rsync -r -C /local-folder/ gs://my-bucket/my-folder/ > /logs/gsutil.log 2>&1

The > gsutil.log 2>&1 pipes both stdout and stderr to the same file. Also, it will overwrite the log file the next time gsutil runs. In order to make it append to the log file, use >> gsutil.log 2>&1. This should be safe on both Linux and OS X.

I'm noticing that the debug flag -d creates enormous log files on large data volumes, so I might opt out on that flag, personally.

Upvotes: 9

jterrace
jterrace

Reputation: 67063

I believe you're getting this error because the HOME environment variable is not set when running under cron. Try setting HOME=/home/fredrik.

Upvotes: 3

Mike Schwartz
Mike Schwartz

Reputation: 12145

You're probably getting a different boto config file when running from cron. Please try running the following both ways (as root, and then via cron), and see if you get different config file lists for the two cases:

gsutil -D ls 2>&1 | grep config_file_list

The reason this happens is that cron unsets most environment variables before running jobs, so you need to manually set the BOTO_CONFIG environment variable in your cron script before running gsutil, i.e.,:

BOTO_CONFIG="/root/.boto"
gsutil rsync ...

Upvotes: 4

Related Questions