I'm trying to generate a release with the executable through the ci/cd pipeline system that Gitlab uses. The aim is to use the executable as part of the release in another project to generate Visual Studio Code Snippets as part of that projects ci/cd pipeline system. This is a hobby project of mine that is aimed towards understanding ci/cd. Repository to generate an executable for: https://gitlab.com/supreme-commander-forged-alliance/other/snippet-generator And an example project to generate snippets for: https://gitlab.com/supreme-commander-forged-alliance/mods/dear-windowing This question is primarily about the former repository, but to prevent the xy problem I decided to add the overall scope of what I'm trying to do. This the .yml file as it stands writing this post: # This file is a template, and might need editing before it works on your project. # To contribute improvements to CI/CD templates, please follow the Development guide at: # https://docs.gitlab.com/ee/development/cicd/templates.html # This specific template is located at: # https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Bash.gitlab-ci.yml # See https://docs.gitlab.com/ee/ci/yaml/README.html for all available options # you can delete this line if you're not using Docker stages: - build - upload - release # determines where packages are installed, required for caching as that # can only happen for a folder inside the repository variables: STACK_ROOT: "${CI_PROJECT_DIR}/.stack-root" PACKAGE_REGISTRY_URL: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID} # cache everything stack related so that libraries do not need to be rebuild cache: paths: - .stack-work/ - .stack-root/ build: only: - master # refers back to the stages defined at the top of this file stage: build # allows us to have haskell build tools image: haskell:9 # allows us to keep track of the executable artifacts: expire_in: 30 days paths: - source/bin/ # things that need to be done during this job script: # build the project - echo ${PACKAGE_REGISTRY_URL} - cd source - stack update - stack build --copy-bins - stack clean upload: stage: upload image: curlimages/curl:latest # we don't need the cache with dependencies here cache: {} # upload the file to make it available as an asset later script: - echo "${PACKAGE_REGISTRY_URL}/snippet-generator" - | curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --upload-file "source/bin/snippet-generator" "${PACKAGE_REGISTRY_URL}/source/bin/snippet-generator" release: stage: release image: registry.gitlab.com/gitlab-org/release-cli # we don't need the cache with dependencies here cache: {} only: - master # release should be done manually when: manual # We recommend the use of `rules` to prevent these pipelines # from running. See the notes section below for details. except: - tags script: - echo "${PACKAGE_REGISTRY_URL}/snippet-generator" - > release-cli create --name release-branch-$CI_JOB_ID --description release-branch-$CI_COMMIT_REF_NAME-$CI_JOB_ID --tag-name job-$CI_JOB_ID --ref $CI_COMMIT_SHA --assets-link '{"name":"Snippet Generator","url":"${PACKAGE_REGISTRY_URL}/source/bin/snippet-generator" }' What is already working: The build stage: the source code is compiled and the executable is stored as an artifact. Dependencies are cached to speed up future runs. The upload stage: the artifact is available and I believe it to be uploaded. The job succeeds, regardless of the fact that an error is returned (1). The release stage: This is just not working. I keep getting errors (2). About (1), this is part of the log: $ echo "${PACKAGE_REGISTRY_URL}/snippet-generator" https://gitlab.com/api/v4/projects/24129638/snippet-generator $ curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --upload-file "source/bin/snippet-generator" "${PACKAGE_REGISTRY_URL}/source/bin/snippet-generator" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 11.7M 100 58 100 11.7M 36 7593k 0:00:01 0:00:01 --:--:-- 7590k {"error":"The provided content-type '' is not supported."} At first I thought this was an error from curl, but in retrospect the json is a response from the (gitlab) server. I can confirm it finds the right file due to the size of the file it is sending: it matches when I download the artifact manually. However, I did not manage to find documentation on the error that it describes and am at a loss whether it is a critical error causing the uploaded file to be rejected. About (2), this is part of the log: $ echo "${PACKAGE_REGISTRY_URL}/snippet-generator" https://gitlab.com/api/v4/projects/24129638/snippet-generator $ release-cli create --name release-branch-$CI_JOB_ID --description release-branch-$CI_COMMIT_REF_NAME-$CI_JOB_ID --tag-name job-$CI_JOB_ID --ref $CI_COMMIT_SHA --assets-link '{"name":"Snippet Generator","url":"${PACKAGE_REGISTRY_URL}/source/bin/snippet-generator" }' time="2021-07-19T15:27:18Z" level=info msg="Creating Release..." cli=release-cli command=create name=release-branch-1435508632 project-id=24129638 ref=e30341cbdecda9b171589aa1ba767ae5bd7583b3 server-url="https://gitlab.com" tag-name=job-1435508632 version=0.8.0 time="2021-07-19T15:27:18Z" level=warning msg="file does not exist, using string value for --description" cli=release-cli description=release-branch-master-1435508632 error="open /builds/supreme-commander-forged-alliance/other/snippet-generator/release-branch-master-1435508632: no such file or directory" version=0.8.0 time="2021-07-19T15:27:19Z" level=fatal msg="failed to create release: API Error Response status_code: 400 message: Validation failed: Links url is blocked: Only allowed schemes are http, https, ftp" cli=release-cli version=0.8.0 I do not understand how it constructs the URL 'open/builds/supreme-commander-forged-alliance/other/snippet-generator/release-branch-master-1435508632'. That URL is not provided by me. But where does this URL come from? And what URL does it require? In general I'm a bit at a loss, I've tried: Reading up on the official documentation Experimenting with this example project: https://gitlab.com/digitalcloudops/release-cli/-/tree/master/docs/examples/release-assets-as-generic-package Looking up videos on Youtube, such as this one: https://www.youtube.com/watch?v=EuwLdbCu3DE But I just don't know how to approach it further.

Reputation: 53

Gitlab CI/CD: generating a release with executable attached

I'm trying to generate a release with the executable through the ci/cd pipeline system that Gitlab uses. The aim is to use the executable as part of the release in another project to generate Visual Studio Code Snippets as part of that projects ci/cd pipeline system. This is a hobby project of mine that is aimed towards understanding ci/cd.

Repository to generate an executable for:

https://gitlab.com/supreme-commander-forged-alliance/other/snippet-generator

And an example project to generate snippets for:

https://gitlab.com/supreme-commander-forged-alliance/mods/dear-windowing

This question is primarily about the former repository, but to prevent the xy problem I decided to add the overall scope of what I'm trying to do.

This the .yml file as it stands writing this post:

# This file is a template, and might need editing before it works on your project.
# To contribute improvements to CI/CD templates, please follow the Development guide at:
# https://docs.gitlab.com/ee/development/cicd/templates.html
# This specific template is located at:
# https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Bash.gitlab-ci.yml

# See https://docs.gitlab.com/ee/ci/yaml/README.html for all available options

# you can delete this line if you're not using Docker

stages:
  - build
  - upload
  - release

# determines where packages are installed, required for caching as that 
# can only happen for a folder inside the repository
variables:
  STACK_ROOT: "${CI_PROJECT_DIR}/.stack-root"
  PACKAGE_REGISTRY_URL: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}
# cache everything stack related so that libraries do not need to be rebuild
cache:
  paths:
    - .stack-work/
    - .stack-root/

build:

  only:
    - master

  # refers back to the stages defined at the top of this file
  stage: build

  # allows us to have haskell build tools
  image: haskell:9

  # allows us to keep track of the executable
  artifacts:  
    expire_in: 30 days
    paths:
      - source/bin/

  # things that need to be done during this job
  script:
      # build the project
    - echo ${PACKAGE_REGISTRY_URL}
    - cd source
    - stack update
    - stack build --copy-bins
    - stack clean

upload:
  stage: upload
  image: curlimages/curl:latest

  # we don't need the cache with dependencies here
  cache: {}

  # upload the file to make it available as an asset later
  script:
    - echo "${PACKAGE_REGISTRY_URL}/snippet-generator"
    - |
      curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --upload-file "source/bin/snippet-generator" "${PACKAGE_REGISTRY_URL}/source/bin/snippet-generator"


release:
  stage: release
  image: registry.gitlab.com/gitlab-org/release-cli

  # we don't need the cache with dependencies here
  cache: {}

  only:
    - master

  # release should be done manually
  when: manual

  # We recommend the use of `rules` to prevent these pipelines
  # from running. See the notes section below for details.
  except:
    - tags
    
  script:
    - echo "${PACKAGE_REGISTRY_URL}/snippet-generator"
    - >
      release-cli create --name release-branch-$CI_JOB_ID --description release-branch-$CI_COMMIT_REF_NAME-$CI_JOB_ID
      --tag-name job-$CI_JOB_ID --ref $CI_COMMIT_SHA
      --assets-link '{"name":"Snippet Generator","url":"${PACKAGE_REGISTRY_URL}/source/bin/snippet-generator" }'

What is already working:

The build stage: the source code is compiled and the executable is stored as an artifact. Dependencies are cached to speed up future runs.
The upload stage: the artifact is available and I believe it to be uploaded. The job succeeds, regardless of the fact that an error is returned (1).
The release stage: This is just not working. I keep getting errors (2).

About (1), this is part of the log:

$ echo "${PACKAGE_REGISTRY_URL}/snippet-generator"
https://gitlab.com/api/v4/projects/24129638/snippet-generator
$ curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --upload-file "source/bin/snippet-generator" "${PACKAGE_REGISTRY_URL}/source/bin/snippet-generator"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11.7M  100    58  100 11.7M     36  7593k  0:00:01  0:00:01 --:--:-- 7590k
{"error":"The provided content-type '' is not supported."}

At first I thought this was an error from curl, but in retrospect the json is a response from the (gitlab) server. I can confirm it finds the right file due to the size of the file it is sending: it matches when I download the artifact manually. However, I did not manage to find documentation on the error that it describes and am at a loss whether it is a critical error causing the uploaded file to be rejected.

About (2), this is part of the log:

$ echo "${PACKAGE_REGISTRY_URL}/snippet-generator"
https://gitlab.com/api/v4/projects/24129638/snippet-generator
$ release-cli create --name release-branch-$CI_JOB_ID --description release-branch-$CI_COMMIT_REF_NAME-$CI_JOB_ID --tag-name job-$CI_JOB_ID --ref $CI_COMMIT_SHA --assets-link '{"name":"Snippet Generator","url":"${PACKAGE_REGISTRY_URL}/source/bin/snippet-generator" }'
time="2021-07-19T15:27:18Z" level=info msg="Creating Release..." cli=release-cli command=create name=release-branch-1435508632 project-id=24129638 ref=e30341cbdecda9b171589aa1ba767ae5bd7583b3 server-url="https://gitlab.com" tag-name=job-1435508632 version=0.8.0
time="2021-07-19T15:27:18Z" level=warning msg="file does not exist, using string value for --description" cli=release-cli description=release-branch-master-1435508632 error="open /builds/supreme-commander-forged-alliance/other/snippet-generator/release-branch-master-1435508632: no such file or directory" version=0.8.0
time="2021-07-19T15:27:19Z" level=fatal msg="failed to create release: API Error Response status_code: 400 message: Validation failed: Links url is blocked: Only allowed schemes are http, https, ftp" cli=release-cli version=0.8.0

I do not understand how it constructs the URL 'open/builds/supreme-commander-forged-alliance/other/snippet-generator/release-branch-master-1435508632'. That URL is not provided by me. But where does this URL come from? And what URL does it require?

In general I'm a bit at a loss, I've tried:

Reading up on the official documentation
Experimenting with this example project: https://gitlab.com/digitalcloudops/release-cli/-/tree/master/docs/examples/release-assets-as-generic-package
Looking up videos on Youtube, such as this one: https://www.youtube.com/watch?v=EuwLdbCu3DE

But I just don't know how to approach it further.

Upvotes: 4

Answers (2)

Vladimir Ch

Reputation: 547

set to app json

Content-Type: application/json

reformat content to json style:

{"body": "bla bla bla"}

works ok for me !

Upvotes: 2

slauth

Reputation: 3178

About (1), I think you are using the wrong URL here. It should look like $CI_API_V4_URL/projects/$CI_PROJECT_ID/packages/generic/$YOUR_PACKAGE_NAME/$YOUR_PACKAGE_VERSION. You could also add --fail to the curl command to have curl exit with an error if the server responds with an HTTP status code >= 400. Regarding the error message about the Content-Type, I guess curl does not imply one because your file does not have an extension. You can however manually specify it with (e.g.) --header "Content-Type: application/octet-stream". If this step succeeds you should be able to see the file in GitLab's Package Registry UI.

About (2), since GitLab 13.2 there is a dedicated release keyword which you could use instead. Since GitLab 13.12 there is also support for asset links.

Upvotes: 0

Gitlab CI/CD: generating a release with executable attached

Answers (2)

Related Questions