Gitlab CI check if directory exists before pulling origin

Question

I'm trying to deploy my flask application to AWS EC2 instance using gitlab ci runner.

.gitlab.ci.yml

stages:
  - test
  - deploy

test_app:
  image: python:latest
  stage: test
  before_script:
  - python -V
  - pip install virtualenv
  - virtualenv env
  - source env/bin/activate
  - pip install flask
  script:
  - cd flask-ci-cd
  - python test.py

prod-deploy:
  stage: deploy
  only:
    - master # Run this job only on  changes for stage branch

  before_script:
    - mkdir -p ~/.ssh
    - echo -e "$RSA_KEY" > ~/.ssh/id_rsa
    - chmod 600 ~/.ssh/id_rsa
    - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'

  script:
    - bash .gitlab-deploy-prod.sh

  environment:
    name: deploy

.gitlab-deploy-prod.sh

#!/bin/bash

# Get servers list
set -f

# access server terminal
shell="ssh -o StrictHostKeyChecking=no ${SERVER_URL}"
git_token=$DEPLOY_TOKEN

echo "Deploy project on server ${SERVER_URL}"
if [ ${shell} -d "/flask-ci-cd" ] # check if directory exists
then
   eval "${shell} cd flask-ci-cd && git clone https://sbhusal123:${git_token}@gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd"
else
    eval "${shell} git pull https://sbhusal123:${git_token}@gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd && cd flask-ci-cd"
fi

Error: .gitlab-deploy-prod.sh: line 7: -o: command not found

How can i check if directory existing??

What i've tried.

#!/bin/bash

# Get servers list
set -f

# access server terminal
shell="ssh -o StrictHostKeyChecking=no ${SERVER_URL}"
git_token=$DEPLOY_TOKEN

eval "${shell}" # i thought gitlab would provide me with shell access
echo "Deploy project on server ${SERVER_URL}"

if [-d "/flask-ci-cd" ] # check if directory exists
then
   eval "cd flask-ci-cd && git clone https://sbhusal123:${git_token}@gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd"
else
    eval "git pull https://sbhusal123:${git_token}@gitlab.com/sbhusal123/flask-ci-cd.git master && cd flask-ci-cd && cd flask-ci-cd"
fi

I've tried to log into the ssh shell before executing the scripts inside if else. But it doesn't works the way intended.

Answer 1

1. Your script has some errors.
2. Do not use eval. No, eval does not work that way. eval is evil
3. When storing a command to a variable, do not use normal variables. Use bash arrays instead to preserve "words".
4. Commands passed via ssh are double escaped. I would advise to prefer to use here documents, they're simpler to get the quoting right. Note the difference in expansion when the here document delimiter is quoted or not.
5. i thought gitlab would provide me with shell access No, without open standard input the remote shell will just terminate, as it will read EOF from input. No, it doesn't work that way.
6. Instead of doing many remote connection, just transfer the execution to remote side once and do all the work there.
7. Take your time and research how quoting and word splitting works in shell.
8. git_token=$DEPLOY_TOKEN No, set variables are not exported to remote shell. Either pass them manually or expand them before calling the remote side. (Or you could also use ssh -o SendEnv=git_token and configure remote ssh with AcceptEnv=git_token I think, never tried it).
9. Read documentation for the utilities you use.
10. No, git clone doesn't take branch name after url. You can specify branch with --branch or -b option. After url it takes directory name. See git clone --help. Same for git pull.

How can i check if directory existing??

Use bash arrays to store the command. Check if the directory exists just by executing the test command on the remote side.

shell=(ssh -o StrictHostKeyChecking=no "${SERVER_URL}")
if "${shell[@]}" [ -d "/flask-ci-cd" ]; then
     ...

In case of directory name with spaces I would go with:

if "${shell[@]}" sh <<'EOF'
[ -d "/directory with spaces" ]
EOF
then

Pass set -x to sh to see what's happening also on the remote side.

For your script, try rather to move the execution to remote side - there is little logic in making 3 separate connections. I say just

echo "Deploy project on server ${SERVER_URL}"
ssh -o StrictHostKeyChecking=no "${SERVER_URL}" bash <<EOF
if [ ! -d /flask-ci-cd ]; then
     # Note: git_token is expanded on host side
     git clone https://sbhusal123:${git_token}@gitlab.com/sbhusal123/flask-ci-cd.git /flask-ci-cd
fi
cd /flask-ci-cd
git pull
EOF

But instead of getting the quoting in some cases right, use declare -p and declare -f to transfer properly quoted stuff to remote side. That way you do not need case about proper quoting - it will work naturally:

echo "Deploy project on server ${SERVER_URL}"
work() {
    if [ ! -d /flask-ci-cd ]; then
         # Note: git_token is expanded on host side
         git clone https://sbhusal123:"${git_token}"@gitlab.com/sbhusal123/flask-ci-cd.git /flask-ci-cd
    fi
    cd /flask-ci-cd
    git pull
{
ssh -o StrictHostKeyChecking=no "${SERVER_URL}" bash <<EOF
$(declare -p git_token)  # transfer variables you need
$(declare -f work)       # transfer function you need
work                     # call the function.
EOF

Answer 2

Updated answer for future reads.

.gitlab-ci.yml

stages:
  - test
  - deploy

test_app:
  image: python:latest
  stage: test
  before_script:
  - python -V
  - pip install virtualenv
  - virtualenv env
  - source env/bin/activate
  - pip install flask
  script:
  - cd flask-ci-cd
  - python test.py

prod-deploy:
  stage: deploy
  only:
    - master 

  before_script:
    - mkdir -p ~/.ssh
    - echo -e "$RSA_KEY" > ~/.ssh/id_rsa
    - chmod 600 ~/.ssh/id_rsa
    - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'

  script:
    - bash .gitlab-deploy-prod.sh

  environment:
    name: deploy

.gitlab-deploy-prod.sh

#!/bin/bash

# Get servers list
set -f
shell=(ssh -o StrictHostKeyChecking=no "${SERVER_URL}")
git_token=$DEPLOY_TOKEN

echo "Deploy project on server ${SERVER_URL}"
ssh -o StrictHostKeyChecking=no "${SERVER_URL}" bash <<EOF
if [ ! -d flask-ci-cd ]; then
    echo "\n Cloning into remote repo..."
    git clone https://sbhusal123:${git_token}@gitlab.com/sbhusal123/flask-ci-cd.git

    # Create and activate virtualenv
    echo "\n Creating virtual env"
    python3 -m venv env
else
    echo "Pulling remote repo origin..."
    cd flask-ci-cd
    git pull
    cd ..
fi

# Activate virtual env
echo "\n Activating virtual env..."
source env/bin/activate

# Install packages
cd flask-ci-cd/
echo "\n Installing dependencies..."
pip install -r requirements.txt
EOF

Answer 3

There is a test command which is explicit about checking files and directories:

test -d "/flask-ci-cd" && eval $then_commands || eval $else_commands  

Depending on the AWS instance I'd expect "test" to be available. I'd recommend putting the commands in variables. (e.g. eval $then_commands)