CODEWITHSUNDEEP
amazon-web-servicesamazon-s3cron
Vassily
Vassily

Reputation: 5428

Why aws cli command has no effect while calling from cron?

Here is my cron jobs list:

root@b03fbed2b08d:~# crontab -l
*/3 * * * * S3_BACKUPS_BUCKET=rake-backups /root/dump_psql.sh > /root/logs/cron-2018-01-28T02:24.log

Content of dump_psql.sh

#!/bin/sh

BACKUP_PATH="/root/backups"
S3_URL="s3://${S3_BACKUPS_BUCKET}"

echo "$(date) Dumping ${POSTGRES_DB} database"
DB_NAME=rake
USER=dbroot
pg_dump -w -c -U $POSTGRES_USER $POSTGRES_DB > "${BACKUP_PATH}/$(date '+%Y-%m-%dT%H:%M').sql"
echo "$(date) Database dump created"

echo "$(date) Syncing ${BACKUP_PATH} folder with ${S3_URL} as $(whoami)"
/usr/local/bin/aws s3 sync $BACKUP_PATH $S3_URL
echo "$(date) Syncing completed"

When I call this script manually It works fine and gives this output:

Sun Jan 28 02:51:46 UTC 2018 Dumping rake database
Sun Jan 28 02:51:47 UTC 2018 Database dump created
Sun Jan 28 02:51:47 UTC 2018 Syncing /root/backups folder with s3://rake-backups as root
upload: backups/2018-01-28T02:48.sql to s3://rake-backups/2018-01-28T02:48.sql
upload: backups/2018-01-28T02:51.sql to s3://rake-backups/2018-01-28T02:51.sql
Sun Jan 28 02:51:48 UTC 2018 Syncing completed

But cron job output looks like this:

Sun Jan 28 02:48:01 UTC 2018 Dumping  database
Sun Jan 28 02:48:01 UTC 2018 Database dump created
Sun Jan 28 02:48:01 UTC 2018 Syncing /root/backups folder with s3://rake-backups as root
Sun Jan 28 02:48:05 UTC 2018 Syncing completed

e.g aws sync has no any output (environment variables are in place) and script has no effect - backups are not in the butcket. Where I am wrong?

Upvotes: 0

Views: 100

Answers (2)

akshay laxapati
akshay laxapati

Reputation: 11

i also tried running this from cron its not working but if i manually run it the output is successfully generated`enter code here`

 #!/bin/sh  
/bin/rm -rf /home/user/data.txt
HOME=/home/user/
HOME="/home/user/"
#source /home/user/sc.sh
export AWS_CONFIG_FILE="/root/.aws/config"
export AWS_ACCESS_KEY_ID=sbc
export AWS_SECRET_ACCESS_KEY=sbc
#export AWS_ACCESS_KEY=sbc
export AWS_SECRET_KEY=sbc
export AWS_DEFAULT_REGION=us-east-1
#####################################################################

STARTTIME=$(/bin/date --date='-59 seconds' +"%Y-%m-%dT%H:%M:%S")
ENDTIME=$(/bin/date +"%Y-%m-%dT%H:%M:%S")
/bin/echo $STARTTIME 
/bin/echo $ENDTIME  
aws --version 
###################################################################################

/bin/echo "Memory/RAM Usage of Production Database :" > /home/user/filename.txt 

T=$(aws cloudwatch get-metric-statistics --namespace AWS/RDS --metric-name FreeableMemory --dimensions Name=DBInstanceIdentifier,Value=NAMEOFDB --statistics Maximum --start-time "$STARTTIME" --end-time "$ENDTIME" --period 360)
/bin/echo $T 

TS=$(aws cloudwatch get-metric-statistics --namespace AWS/RDS --metric-name FreeableMemory --dimensions Name=DBInstanceIdentifier,Value=NAMEOFDB --statistics Maximum --start-time "$STARTTIME" --end-time "$ENDTIME" --period 360|/bin/grep Timestamp) >> /home/user/filename.txt
/bin/echo $TS

STARTTIME=$(/bin/date --date='-59 seconds' +"%Y-%m-%dT%H:%M:%S")
ENDTIME=$(/bin/date +"%Y-%m-%dT%H:%M:%S")

FREEM=$(aws cloudwatch get-metric-statistics --namespace AWS/RDS --metric-name FreeableMemory --dimensions Name=DBInstanceIdentifier,Value=NAMEOFDB --statistics Maximum --start-time "$STARTTIME" --end-time "$ENDTIME" --period 360|/bin/grep Maximum|/bin/cut -d: -f 2|/usr/bin/tr -d ','|cut -d. -f 1 ) >> /home/user/filename.txt


FREEMG=$((FREEM/1073741824))
/bin/echo "Memory Utilization = $FREEMG GB /244GB " >> /home/user/filename.txt


###################################################################################################################################################################################
STARTTIME=$(/bin/date --date='-59 seconds' +"%Y-%m-%dT%H:%M:%S")
ENDTIME=$(/bin/date +"%Y-%m-%dT%H:%M:%S")
/bin/echo ""
/bin/echo " Storage Utilization of DriverConnect Production Database Server :" >> /home/user/filename.txt

#echo ""
# Timestamp
TS=$(aws cloudwatch get-metric-statistics --namespace AWS/RDS --metric-name FreeStorageSpace --dimensions Name=DBInstanceIdentifier,Value=NAMEOFDB --statistics Maximum --start-time "$STARTTIME" --end-time "$ENDTIME" --period 360|/bin/grep Timestamp)
/bin/echo $TS  >> /home/user/filename.txt


STARTTIME=$(/bin/date --date='-59 seconds' +"%Y-%m-%dT%H:%M:%S")
ENDTIME=$(/bin/date +"%Y-%m-%dT%H:%M:%S")

# storage utilization in GB

FREEM=$(aws cloudwatch get-metric-statistics --namespace AWS/RDS --metric-name FreeStorageSpace --dimensions Name=DBInstanceIdentifier,Value=NAMEOFDB --statistics Maximum --start-time "$STARTTIME" --end-time "$ENDTIME" --period 360|grep Maximum|/bin/cut -d: -f 2 | /usr/bin/tr -d ','|/bin/cut -d. -f 1 ) >> /home/user/filename.txt
#echo " Occupied Storage of Driverconnect Production Database : " >> /home/user/filename.txt
FREEMG=$((FREEM/1073741824)) 
/bin/echo " Available Storage = $FREEMG Gb/200 Gb" >> /home/user/filename.txt

/bin/echo "################################################################################################################################################################################################"
/bin/echo "################################################################################################################################################################################################"
/bin/echo "################################################################################################################################################################################################"
/bin/echo "################################################################################################################################################################################################"
/bin/echo "################################################################################################################################################################################################"
/bin/echo "################################################################################################################################################################################################"

Upvotes: 0

John Hanley
John Hanley

Reputation: 81356

You are not checking that your commands are failing. You are not checking for errors. You are also not sending the error output to your log file.

This document describes the return value from AWS commands:

AWS CLI Return Codes

Modify both your commands to check for failure. Notice the change for the output of pg_dump and the addition of sending the error output to standard output.

pg_dump -w -c -U $POSTGRES_USER $POSTGRES_DB -f "${BACKUP_PATH}/$(date '+%Y-%m-%dT%H:%M').sql" 2>&1

if [ $? -eq 0 ]
then
    echo "$(date) Database dump created"
else
    echo "$(date) Database dump FAILED"
fi

/usr/local/bin/aws s3 sync $BACKUP_PATH $S3_URL 2>&1
if [ $? -eq 0 ]
then
    echo "$(date) Syncing completed"
else
    echo "$(date) Syncing FAILED"
fi

Now review the log file again for error messages and to determine which command failed.

Upvotes: 2

Related Questions