Reputation: 1496
I am having a problem with validating the file existence in this script. The script fails when there is not a file in the ADDITION or the DELETIONS path.
How can I validate that there is a file in the ADDITION or DELETION path?
I'm doing my first steps with bash and Makefile so any help will be appreciated.
execute-s3-sync: $(foreach BUCKET,$(S3_BUCKETS),execute-$(BUCKET)-s3-sync) execute-large-files-s3-sync
execute-%-s3-sync: require-target-environment property-SOURCE_DATA_EXPORT_KEY property-TARGET_DATA_EXPORT_KEY
export AWS_DEFAULT_REGION=$(TARGET_AWS_REGION)
MANIFEST="$(if $(filter $*,large-files),large-files,$(call get-source-s3-bucket,$*)).gz"
ADDITION="target/migrations/$(TARGET_DATA_EXPORT_KEY)/diff/$(SOURCE_DATA_EXPORT_KEY)/additions/$$MANIFEST"
DELETION="target/migrations/$(TARGET_DATA_EXPORT_KEY)/diff/$(SOURCE_DATA_EXPORT_KEY)/deletions/$$MANIFEST"
[[ -f "$$ADDITION" ]] && \
pipenv run python scripts/large-files-copy.py \
$(foreach BUCKET,$(S3_BUCKETS),--bucket-equivalences $(call get-source-s3-bucket,$(BUCKET)) $(call get-target-s3-bucket,$*)) \
--manifest $$ADDITION \
submit-to-batch \
--job-queue "$(STACK_PREFIX)-batch-job-queue" \
--job-definition "$(STACK_PREFIX)-batch-job-definition"
[[ -f "$$DELETION" ]] && \
pipenv run python scripts/large-files-copy.py \
$(foreach BUCKET,$(S3_BUCKETS),--bucket-equivalences $(call get-source-s3-bucket,$(BUCKET)) $(call get-target-s3-bucket,$*)) \
--manifest $$DELETION \
local-run \
--operation remove;
Upvotes: 2
Views: 122
Reputation: 6994
you have to make this a huge, single bash command; make executes each line separately in the shell and variables are not available in later steps. To improve readability/maintainability, I would replace the test -f ... && xxx
statements here by if
blocks.
E.g.
execute-%-s3-sync: ...
MANIFEST="$(if $(filter $*,large-files),large-files,$(call get-source-s3-bucket,$*)).gz" && \
ADDITION="target/migrations/$(TARGET_DATA_EXPORT_KEY)/diff/$(SOURCE_DATA_EXPORT_KEY)/additions/$$MANIFEST" && \
DELETION="target/migrations/$(TARGET_DATA_EXPORT_KEY)/diff/$(SOURCE_DATA_EXPORT_KEY)/deletions/$$MANIFEST" && \
if test -f "$$ADDITION"; then \
...
fi && \
if test -f "$$DELETION"; then \
...
Regarding failing, you have to express things in a positive manner. E.g.
BAD:
test -f $FILE && do_something ## --> will fail when $FILE does not exists
GOOD:
! test -f $FILE || do_something ## --> fails only, when do_something fails
GOOD:
if test -f $FILE; then
do_something
fi
Upvotes: 1