CODEWITHSUNDEEP
bashgoogle-cloud-platformgoogle-cloud-storageastronomer
Mani Shankar.S
Mani Shankar.S

Reputation: 39

Bash Operator to access GCS bucket file in GCP Astronomer

I need to run some bash operation on a file that has been present in GCS Bucket.

bash_operator = BashOperator(
    task_id='mani_bash',
    bash_command="""if [ `awk -F: '/^[^HDR][^TRL]/ { print }' gs://<bucketname>/<location>/filename.txt | awk -F "|" '{print NF-1}' | uniq | wc -l` -eq 1 ];
 then
 if [ `awk -F: '/^[^HDR][^TRL]/ { print }' gs://<bucketname>/<location>/filename.txt | awk -F "|" '{print NF-1}' | uniq` -eq 9 ]; then 
echo 'rite'; 
fi;
 else
 echo 'not rite'; 
fi""",
)

I am getting this error.

awk: cannot open gs://bucketname/location/filename.txt (No such file or directory)

Can someone let me know, is it possible to access GCS bucket using bash operator. if yes, please let me know, how to access.

Upvotes: 0

Views: 571

Answers (1)

Mani Shankar.S
Mani Shankar.S

Reputation: 39

gsutil cat helped as solution

bash_operator = BashOperator(
    task_id='mani_bash',
    bash_command="""if [ `gsutil cat gs://<bucketname>/<location>/filename.txt | awk -F: '/^[^HDR][^TRL]/ { print }' | awk -F "|" '{print NF-1}' | uniq | wc -l` -eq 1 ];
 then
 if [ `gsutil cat gs://<bucketname>/<location>/filename.txt | awk -F: '/^[^HDR][^TRL]/ { print }' | awk -F "|" '{print NF-1}' | uniq` -eq 9 ]; then 
echo 'rite'; 
fi;
 else
 echo 'not rite'; 
fi""",
)

Upvotes: 0

Related Questions