CODEWITHSUNDEEP
gitlab-ci
Mostafa Ghadimi
Mostafa Ghadimi

Reputation: 6736

How to run the Gitlab CI jobs with different runners independent from each other?

Description

As I have found from this Stackoverflow question and Gitlab-CI official documentation, by using tags keyword, different runners can be triggered for a single project. So I have registered different runners on my servers (one runner for each (staging and production) server) with dashboard_staging and dashboard_production tag names.

runners status

Everything works well and in order to start gitlab-runner properly, I have executed the following commands:

sudo gitlab-runner verify  # Everything was ok
sudo gitlab-runner start  # It was started successfully on both servers

Then I have committed the changes and push them on Gitlab and it was triggered successfully.

Problems

  1. I have executed the above commands, but one of the pipelines is still pending for a runner.

Pipeline flow

  1. Since build stage has not been done completely, it wouldn't progressed for the tag in which its job has been done.

Code

stages:
  - test
  - build
  - deploy

cache:
  untracked: true
  key:
    files:
      - yarn.lock
  paths:
    - node_modules/
    - .yarn

dashboard:test:staging:
  stage: test
  tags:
    - dashboard_staging
  when: manual
  before_script:
    - echo "do sth"
  only:
    - staging
  except:
    changes:
      - 'docker/**/*'
      - '*.md'
  script:
    - echo "do sth"

dashboard:test:production:
  stage: test
  tags:
    - dashboard_production
  when: manual
  before_script:
    - echo "do sth"
  only:
    - staging
  except:
    changes:
      - 'docker/**/*'
      - '*.md'
  script:
    - echo "do sth"

dashboard:build:staging:
  stage: build
  tags:
    - dashboard_staging
  only:
    - staging
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY  
  script:
    - echo "do sth"

dashboard:build:production:
  stage: build
  tags:
    - dashboard_production
  only:
    - staging
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY  
  script:
    - echo "do sth"


dashboard:deploy:staging:
  stage: deploy
  tags:
    - dashboard_staging
  except:
    changes:
      - 'docker/**/*'
      - '*.md'
  only:
    - staging
  before_script:
    - echo "do sth"
  script:
    - echo "do sth"


dashboard:deploy:production:
  stage: deploy
  tags:
    - dashboard_production
  except:
    changes:
      - 'docker/**/*'
      - '*.md'
  only:
    - staging
  before_script:
    - echo "do sth"
  script:
    - echo "do sth"

Questions

pending issue

Upvotes: 0

Views: 5639

Answers (2)

Mostafa Ghadimi
Mostafa Ghadimi

Reputation: 6736

I will post the details about how my problem and question was solved for future readers.

Question 1: How can I resolve this pending issue in Deploy stage?

Answer 1:

This problem was solved after executing the following three commands:

sudo gitlab-runner verify
sudo gitlab-runner start
sudo gitlab-runner run

P.S. If you are non-root and want to execute gitlab-runner without sudo (with your user privilege, you can easily add gitlab-runner to the sudoers by usermod -aG sudo gitlab-runner command.)

For more details, you can visit the following links:

Question 2: Is there any way to run the defined tags independently from each other?

Answer 2: As @Patrick has mentioned, there is a tag existing in gitlab-ci yaml file named needs that exactly works as expected.

stages:
  - test
  - build
  - deploy

cache:
  untracked: true
  key:
    files:
      - yarn.lock
  paths:
    - node_modules/
    - .yarn

dashboard:test:staging:
  stage: test
  tags:
    - dashboard_staging
  when: manual
  before_script:
    - echo "do sth"
  only:
    - staging
  except:
    changes:
      - 'docker/**/*'
      - '*.md'
  script:
    - echo "do sth"

dashboard:test:production:
  stage: test
  tags:
    - dashboard_production
  when: manual
  before_script:
    - echo "do sth"
  only:
    - staging
  except:
    changes:
      - 'docker/**/*'
      - '*.md'
  script:
    - echo "do sth"

dashboard:build:staging:
  stage: build
  tags:
    - dashboard_staging
  only:
    - staging
  needs: ["dashboard:test:staging"]
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY  
  script:
    - echo "do sth"

dashboard:build:production:
  stage: build
  tags:
    - dashboard_production
  only:
    - staging
  needs: ["dashboard:test:production"]
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY  
  script:
    - echo "do sth"


dashboard:deploy:staging:
  stage: deploy
  tags:
    - dashboard_staging
  except:
    changes:
      - 'docker/**/*'
      - '*.md'
  only:
    - staging
  needs: ["dashboard:build:staging"]
  before_script:
    - echo "do sth"
  script:
    - echo "do sth"


dashboard:deploy:production:
  stage: deploy
  tags:
    - dashboard_production
  except:
    changes:
      - 'docker/**/*'
      - '*.md'
  only:
    - staging
  needs: ["dashboard:build:production"]
  before_script:
    - echo "do sth"
  script:
    - echo "do sth"

For more information you can read the official documentation of Gitlab CI.

Upvotes: 1

Patrick
Patrick

Reputation: 3230

If I'm understanding what you'd like correctly, you want the "staging" jobs to run independently of the "production" jobs, and ignore the fact that the other job in the same stage may not have completed yet.

This is what the needs keyword is for (reference). Since you have not defined any needs keyword on your jobs, each job will wait for the whole previous stage to complete before it kicks off. The default behavior of GitLab CI is to run all jobs within a stage in parallel, then each stage in series. You can then override that with the needs keyword to start jobs independent of the stage they are in.

Try the following:

stages:
  - test
  - build
  - deploy

cache:
  untracked: true
  key:
    files:
      - yarn.lock
  paths:
    - node_modules/
    - .yarn

dashboard:test:staging:
  stage: test
  tags:
    - dashboard_staging
  when: manual
  before_script:
    - echo "do sth"
  only:
    - staging
  except:
    changes:
      - 'docker/**/*'
      - '*.md'
  script:
    - echo "do sth"

dashboard:test:production:
  stage: test
  tags:
    - dashboard_production
  when: manual
  before_script:
    - echo "do sth"
  only:
    - staging
  except:
    changes:
      - 'docker/**/*'
      - '*.md'
  script:
    - echo "do sth"

dashboard:build:staging:
  stage: build
  tags:
    - dashboard_staging
  only:
    - staging
  needs: ["dashboard:test:staging"]
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY  
  script:
    - echo "do sth"

dashboard:build:production:
  stage: build
  tags:
    - dashboard_production
  only:
    - staging
  needs: ["dashboard:test:production"]
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY  
  script:
    - echo "do sth"


dashboard:deploy:staging:
  stage: deploy
  tags:
    - dashboard_staging
  except:
    changes:
      - 'docker/**/*'
      - '*.md'
  only:
    - staging
  needs: ["dashboard:build:staging"]
  before_script:
    - echo "do sth"
  script:
    - echo "do sth"


dashboard:deploy:production:
  stage: deploy
  tags:
    - dashboard_production
  except:
    changes:
      - 'docker/**/*'
      - '*.md'
  only:
    - staging
  needs: ["dashboard:build:production"]
  before_script:
    - echo "do sth"
  script:
    - echo "do sth"

Upvotes: 1

Related Questions