Why does GitLab Ci not find my cached folder?

Question

I have a list of CI jobs running in my GitLab and the Caching does not work as expected:

This is how my docu-generation job ends:

[09:19:33] Documentation generated in ./documentation/ in 4.397 seconds using gitbook theme
Creating cache angular...
00:02
WARNING: frontend/node_modules: no matching files  
frontend/documentation: found 136 matching files   
No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally. 
Created cache
Job succeeded

I then start a deployment Job (to GitLab Pages) but it fails because it doesn't find the documentation-folder:

$ cp -r frontend/documentation .public/frontend
cp: cannot stat 'frontend/documentation': No such file or directory

this is the cache config of the generation:

generate_docu_frontend:
  image: node:12.19.0
  stage: build
  cache:
    key: angular
    paths:
      - frontend/node_modules
      - frontend/documentation
  needs: ["download_angular"]

and this is for deployment:

deploy_documentation:
  stage: deploy
  cache:
    - key: angular
      paths:
        - frontend/node_modules
        - frontend/documentation
      policy: pull
    - key: laravel
      paths:
        - backend/vendor
        - backend/public/docs
      policy: pull

does anyone know why my documentation folder is missing?

Answer 1

The message in your job output No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally. just means that your runners are not using Amazon S3 to store your cache, or something similar like Minio.

Without S3/Minio, the cache only lives on the runner that first ran the job and cached the resources. This means that the next time the job runs and it happens to be picked up by a different runner, it won't have the cache. In that case, you'd run into an error like this.

There's a couple ways around this:

1. Configure your runners to use S3/Minio (Minio is an open source, free-to-use license if you're interested in hosting it yourself).
2. Only use one runner (not a great solution since generally more runners means faster pipelines and this would slow things down considerably, though it would solve the cache problem).
3. Use tags. Tags are used to ensure that a job runs on a specific runner(s). Let's say for example that 1 out of your 10 runners have access to your production servers, but all have access to your lower environment servers. Your lower-env jobs can run on any runner, but your Production Deployment job has to run on the one runner with prod access. You can do this by putting a Tag on the runner called let's say prod-access and putting the same tag on the prod deploy job. This will ensure that job will run on the runner with prod access. The same thing can be used here to ensure the cache is available.
4. Use artifacts instead of cache. I'll explain this option below as it's really what you should be using for this use case.

Let's briefly explain the difference between Cache and Artifacts:

• Cache is generally best used for dependency installation like npm or composer (for PHP projects). When you have a job that runs npm ci or composer install, you don't want it to run every since time your pipeline runs when you don't necessary change the dependencies as it wastes time. Use the cache keyword to cache the dependencies so that subsequent pipelines don't have to install the dependencies again.

• Artifacts are best used when you need to share files or directories between jobs in the same pipeline. For example, after installing npm dependencies, you might need to use the node_modules directory in another job in the pipeline. Artifacts are also uploaded to the GitLab server by the runner at the end of the job, opposed to being stored locally on the runner that ran the job. All previous artifacts will be downloaded for all subsequent jobs, unless controlled with either dependencies or needs.

Artifacts are the better choice for your use case.

Let's update your .gitlab-ci.yml file to use artifacts instead of cache:

stages:
  - build
  - deploy

generate_docu_frontend:
  image: node:12.19.0
  stage: build
  script:
    - ./generate_docs.sh # this is just a representation of whatever steps you run to generate the docs
  artifacts:
    paths:
      - frontend/node_modules
      - frontend/documentation
    expire_in: 6 hours # your GitLab instance will have a default, you can override it like this
    when: on_success # don't attempt to upload the docs if generating them failed

deploy_documentation:
  stage: deploy
  script:
    - ls # just an example showing that frontend/node_modules and frontend/documentation are present
    - deploy.sh # whatever else you need to run this job