CODEWITHSUNDEEP
bashcurlqualtrics
D.sen
D.sen

Reputation: 922

Corrupted zipfile after curl request

I am using the Qualtrics API to retrieve survey data at scheduled intervals. Below is my shell script (bash) get_responses.sh which posts the export, measures the download completion rate, gets the export, and stores/unzips the file.

STARTDATE=$(date -v-7d "+%Y-%m-%d")
STARTDATESTRING=$STARTDATE"T00:00:00-07:00:00"
ENDDATE=$(date "+%Y-%m-%d")
ENDDATESTRING=$ENDDATE"T00:00:00-07:00:00"

result=$(curl -X POST -H 'X-API-TOKEN: MYAPITOKEN' -H 'Content-Type: application/json' -d '{
    "surveyId": "SV_000000000000",
    "startDate": "startDate": "'"$STARTDATESTRING"'",
    "endDate": "endDate": "'"$ENDDATESTRING"'",
    "format": "csv",
    "useLocalTime": true,
    "useLabels": true
}' "https://myorg.qualtrics.com/API/v3/responseexports")

es_id=$(echo "$result" | /usr/local/Cellar/jq/1.5_2/bin/jq --raw-output '.result.id')

curl -H "X-API-TOKEN: MYAPITOKEN" "https://myorg.qualtrics.com/API/v3/responseexports/${es_id}"

curl -X GET -H "Content-Type: application/json" -H "X-API-TOKEN: MYAPITOKEN" "https://myorg.qualtrics.com/API/v3/responseexports/${es_id}/file" -o "/Users/myname/Desktop/WEA/response.zip"

unzip "/Users/myname/Desktop/WEA/response.zip" -d "/Users/myname/Desktop/WEA/"

I intermittently get the following error-

% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 339 100 133 100 206 165 256 --:--:-- --:--:-- --:--:-- 256 {"result":{"percentComplete":0.0,"file":null,"status":"in progress"},"meta":{"httpStatus":"200 - OK","requestId":"2c55524c-03aa-495c-8de7-b54d5b441b34"}} % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 129 100 129 0 0 405 0 --:--:-- --:--:-- --:--:-- 405

End-of-central-directory signature not found. Either this file is not a zipfile, or it constitutes one disk of a multi-part archive. In the latter case the central directory and zipfile comment will be found on the last disk(s) of this archive.

unzip: cannot find zipfile directory in one of /Users/myname/Desktop/response.zip or /Users/myname/Desktop/WEA/response.zip.zip, and cannot find /Users/myname/Desktop/WEA/response.zip.ZIP, period.

The odd part is that this error usually only occurs when I first run the script. If I rerun it, without making any changes, it will go through with no errors. To my understanding this error message means a corrupted zip file. Am I corrupting the POST request by saving it into a variable? I need to capture the post request output somehow as it provides the es_id that I need for the subsequent GET request. I would hard code it but the es_id refreshes weekly.

Upvotes: 0

Views: 1669

Answers (1)

D.sen
D.sen

Reputation: 922

Using @T.Gibbons advice I was able to eliminate the error on the initial run by incorporating a while loop into the script.

STARTDATE=$(date -v-7d "+%Y-%m-%d")
STARTDATESTRING=$STARTDATE"T00:00:00-07:00:00"
ENDDATE=$(date "+%Y-%m-%d")
ENDDATESTRING=$ENDDATE"T00:00:00-07:00:00"
post_response=$(curl -X POST -H 'X-API-TOKEN: MYAPITOKEN' -H 'Content-Type: application/json' -d '{
    "surveyId": "SV_00000000000000",
    "startDate": "'"$STARTDATESTRING"'",
    "endDate": "'"$ENDDATESTRING"'",
    "format": "csv",
    "useLocalTime": true,
    "useLabels": true
}' "https://myorg.qualtrics.com/API/v3/responseexports")

es_id=$(echo "$post_response" | /usr/local/Cellar/jq/1.5_2/bin/jq --raw-output '.result.id')

percent_complete=0
while [ $percent_complete -ne 100 ]
do
    response=$(curl -H "X-API-TOKEN: MYAPITOKEN" "https://myorg.qualtrics.com/API/v3/responseexports/${es_id}")
    percent_complete=$(echo "$response" | /usr/local/Cellar/jq/1.5_2/bin/jq --raw-output '.result.percentComplete')
done

curl -X GET -H "Content-Type: application/json" -H "X-API-TOKEN: MYAPITOKEN" "https://myorg.qualtrics.com/API/v3/responseexports/${es_id}/file" -o "/Users/myname/Desktop/WEA/response.zip"

unzip "/Users/myname/Desktop/WEA/response.zip" -d "/Users/myname/Desktop/WEA"

rm "/Users/myname/Desktop/WEA/response.zip"

Upvotes: 1

Related Questions