CODEWITHSUNDEEP
linuxshellparallel-processing
lc wanigasinghe
lc wanigasinghe

Reputation: 19

linux parallel execution and get status error

The parallel execution is working properly as expected. But i need to get the states of each file. Here I need to check it by using the if condition. This is just a sample and i have many scenarios like this in my code.

I'm a beginner at #help shell scripts & linux but actually i love linux now. I'm expecting an alternative method or a help to resolve this. Thank you everyone.

( echo $FILE_PATH/import_csv_location_data_into_hive.sh; echo $FILE_PATH/import_csv_order_data_into_hive.sh; echo $FILE_PATH/import_csv_order_with_delivery_info_into_hive.sh; ) | parallel bash 

wait
IMPORTED_LOCATIONS_STATUS=$? IMPORTED_ORDERS_STATUS=$? IMPORTED_ORDERS_WITH_DELIVERY_INFO_STATUS=$?


if [[ "$IMPORTED_LOCATIONS_STATUS" == "0" && "$IMPORTED_ORDERS_STATUS" == "0" && "$IMPORTED_ORDERS_WITH_DELIVERY_INFO_STATUS" == "0" ]];
then
  echo  "COMPLETED."
  exit 0 
else
  STATUS="FAILED."
  exit 1
fi

Sample code

enter image description here

Upvotes: 1

Views: 266

Answers (2)

Ole Tange
Ole Tange

Reputation: 33705

If you only need to know if all finished with no errors:

(
 echo $FILE_PATH/import_csv_location_data_into_hive.sh;
 echo $FILE_PATH/import_csv_order_data_into_hive.sh;
 echo $FILE_PATH/import_csv_order_with_delivery_info_into_hive.sh;
) | parallel bash

if [[ "$?" == "0" ]]; then
  echo  "COMPLETED."
  exit 0 
else
  STATUS="FAILED."
  exit 1
fi

Shorter, but less readable IMO:

(
 echo $FILE_PATH/import_csv_location_data_into_hive.sh;
 echo $FILE_PATH/import_csv_order_data_into_hive.sh;
 echo $FILE_PATH/import_csv_order_with_delivery_info_into_hive.sh;
) | if parallel bash ; then
  echo  "COMPLETED."
  exit 0 
else
  STATUS="FAILED."
  exit 1
fi

Upvotes: 1

James Risner
James Risner

Reputation: 6085

(echo Your-Special-Header; \
    echo whoami; \
    echo cat nothere \
) | parallel --header : --results all.csv bash -c

The file nothere does not exist. Here is the csv output:

Seq,Host,Starttime,JobRuntime,Send,Receive,Exitval,Signal,Command,Your-Special-Header,Stdout,Stderr
1,:,1662823208.481,0.001,0,5,0,0,"bash -c whoami",whoami,"root
",
2,:,1662823208.481,0.003,0,0,1,0,"bash -c cat\ nothere","cat nothere",,"cat: nothere: No such file or directory
"

The 7th column is Exitval.

Upvotes: 1

Related Questions