CODEWITHSUNDEEP
gitlabgitlab-cigitlab-ci-runner
delanne
delanne

Reputation: 440

gitlab 8.2.1, How to use cache in .gitlab-ci.yml

I'm trying to use 'cache' in .gitlab-ci.yml (http://doc.gitlab.com/ce/ci/yaml/README.html#cache). My gitlab version is 8.2.1 and my Runner is: 

$ docker exec -it gitlab-runner gitlab-runner -v 
gitlab-runner version 0.7.2 (998cf5d)

So according to the doc, everything is up to date, but I'm unable to use the cache ;-(. All my files are always deleted. Am I doing something wrong?

A cache archive is created, but not passed to the next jobs.

Here is my .gitlab-ci.yml

$ cat .gitlab-ci.yml
    stages:
    - createcache
    - testcache

    createcache:
      type: createcache
      cache:
        untracked: true
        paths:
          - doc/
      script:
        - touch doc/cache.txt

    testcache:
      type: testcache
      cache:
        untracked: true
        paths:
          - doc/
      script:
        - find .
        - ls doc/cache.txt

Output of the job 'createcache'

Running on runner-141d90d4-project-2-concurrent-0 via 849d416b5994...
Fetching changes...
HEAD is now at 2ffbadb MUST BE REVERTED
[...]
$ touch doc/cache.txt
[...]
Archiving cache...
INFO[0000] Creating archive cache.tgz ...              
INFO[0000] Done!                                        

Build succeeded.

Output of the job 'testcache'

Running on runner-141d90d4-project-2-concurrent-0 via 849d416b5994...
Fetching changes...
Removing doc/cache.txt
[...]
$ ls doc/cache.txt
ls: cannot access doc/cache.txt: No such file or directory

ERROR: Build failed with: exit code 1

My workaround

My workaround is to manually untar what's in the /cache directory ... I'm pretty sure that's not the correct way to use cache ...

$ cat .gitlab-ci.yml
    stages:
    - build
    - test
    - deploy

    image: ubuntu:latest

    before_script:
      - export CACHE_FILE=`echo ${CI_PROJECT_DIR}/createcache/${CI_BUILD_REF_NAME}/cache.tgz | sed -e "s|/builds|/cache|"`

    createcache:
      type: build
      cache:
        untracked: true
        paths:
          - doc/
      script:
        - find . | grep -v ".git"
        - mkdir -p doc
        - touch doc/cache.txt

    testcache:
      type: test
      script:
        - env
        - find . | grep -v ".git"
        - tar xvzf ${CACHE_FILE}
        - ls doc/cache.txt

Upvotes: 30

Views: 43652

Answers (4)

akop
akop

Reputation: 7865

We found out, that is better to cache the whole .gradle-folder. This should be used only once per project. If k8s used for gitlab-runner, then you should turn the daemon off.

gitlab-ci.yaml:

variables:
  GRADLE_OPTS: "-Dorg.gradle.daemon=false" # no gradle-daemon on k8s

# ...

buildJar:
  stage: build
  image: gradle:5.6.4-jdk11
  before_script:
    - export GRADLE_USER_HOME=`pwd`/.gradle
  script:
    - gradle assemble
  cache:
    key: "gradleCache"
    paths:
      - .gradle

Upvotes: 0

Lance Chen
Lance Chen

Reputation: 596

8.2 only supports per-job cache, and 8.3 will introduce "group" cache that serves among jobs according to @ayufan's comment in Possibility to cache folders in build directory (#97).

However, although I cannot be 100% sure, by quick digging the source code of gitlab-ci-muti-runner, docker executor doesn't seems to work with the cache functionality. Since a whole new container is created and destroyed in every job, the cache.tgz archive would no longer exist in the next build.

Errata:

The above description is incorrect due to testing in a misconfigured environment. By default, gitlab-ci-multi-runner creates a dedicated data volume container as a cache storage for each concurrent build. The cache container is mounted to directory /cache in the application container and those cache.tgz tarballs are placed under /cache by default. So caches are actually reusable among independent builds.

Updates 2015/12/11:

Just found out that "group" cache has already been implemented in gitlab-runner@7dc9524f6ef0144b3797fc07c9035f38a8ad0512, maybe not yet released and documented. You can enable it with

cache:
  paths:
    - doc/
  group: sharedcache

The result is one cache tarball being placed under path <namespace>/<repo>/sharedcache/ instead of two cache tarballs being placed separately under path <namespace>/<repo>/createcache/ and <namespace>/<repo>/testcache/.

Updates 2017/12/04:

"group" cache has been replaced by cache:key. Use the key key to make the cache share between jobs or git references. By default, a cache is shared between all jobs. So, simply write the following would do the job

cache:
  paths:
    - doc/

Checkout GitLab CI cache:key and gitlab-runner@d34f76f86a7c2fc4d399e9922175bcc614434016 for more information.

Upvotes: 14

Patrick
Patrick

Reputation: 35234

It seems it is not possible to cache container local files with shared runners. You have to put your files in e.g. the cache folder:

before_script:
  - export GRADLE_USER_HOME=/cache/.gradle

...

cache:
  paths:
    - $GRADLE_USER_HOME/caches/
    - $GRADLE_USER_HOME/wrapper/
    - $GRADLE_USER_HOME/build-cache/

Strictly it is not required to put the /cache folder as files to cache, because this happens automatically, but I leave it for clarity (also when I want to move the gradle cache)

Upvotes: 4

wcc526
wcc526

Reputation: 4105

https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/issues/327

image: java:openjdk-8-jdk

before_script:
    - export GRADLE_USER_HOME=`pwd`/.gradle

cache:
  paths:
    - .gradle/wrapper
    - .gradle/caches

build:
  stage: build
  script:
     - ./gradlew assemble

test:
  stage: test
  script:
     - ./gradlew check

Upvotes: 25

Related Questions