Cache git repositories installed with Composer from source on Travis CI (and other CI tools)?

Question

I work on several Symfony bundles hosted on GitHub and tested automatically with Travis CI.

The longest part of the tests is the installation of the requirements by Composer.

I configured Travis CI to install packages with composer update --prefer-dist and cache the $HOME/.composer/cache directory. Tests took a total time of 18 minutes, thanks to caching:

Installing doctrine/lexer (v1.0.1)

Loading from cache

But a week ago I saw a message from Composer:

Installing doctrine/lexer (v1.0.1)

Downloading: Connecting... Failed to download doctrine/lexer from dist: Could not authenticate against github.com

I changed the configuration to composer update --prefer-source because of this. This seems to be a common practice across Symfony bundles. The tests suite took 28 minutes.

I know that I can register GitHub keys in Travis CI in order to avoid the API limit while using the Composer --prefer-dist option.

Are they some other ways to cache the dependencies? E.g by cloning the dependencies repositories in a cache?

Answer 1

GitHub has removed the API rate limits, composer can now be used with --prefer-dist and the zip files can be cached. Here is an example of configuration in .travis.yml:

cache:
  directories:
    - $HOME/.composer/cache

# …

install:
  - composer update --prefer-dist

---

Here is the announce:

Hi Niels and Jordi,

We've spent some time digging in, considering all the options to resolve your issues, weighing your needs against infrastructure strain and availability.

I'm happy to report that our Infrastructure team believes that due to their work on our Git backend since these APIs were introduced, we're now able to drop these rate limits on the API side. We deployed those changes a couple of hours ago. Getting an archive link 1 via the API will no longer count against your hourly rate limit (authenticated or not). This should make Composer installs happy.

Let us know if you see any funny business.

Cheers,

Wynn Netherland
Platform Engineering Manager, GitHub

Source.

Answer 2

I tested caching of the vendor/ directory and it worked. I used tar in order to create an uncompressed archive $HOME/vendor-cache/ and configured Travis CI for this directory.

Commands have two goals:

1. Extract the vendor/ from cache if it's available
2. Put the vendor/ in the cache after the tests

Here is an example .travis.yml file:

sudo: false

language: php

cache:
  directories:
    - $HOME/.composer/cache
    # This is where vendor/ backups will be stored
    - $HOME/vendor-cache/

php:
  - […]

env:
  - SYMFONY_VERSION="2.7.*"
  - SYMFONY_VERSION="2.8.*"
  - SYMFONY_VERSION="3.0.*"

before_install:
 # Create an hash corresponding to the PHP version and the dependencies
 - tohash="${SYMFONY_VERSION}"
 - cachefile="`echo -n "$tohash" | sha1sum | cut -d " " -f 1`.tar"
 # Extract cache archive if the file exists
 - if [[ -f $HOME/vendor-cache/$cachefile ]]; then tar -xf $HOME/vendor-cache/$cachefile ; fi

install:
  - composer self-update
  - composer update --profile --no-progress

script: php ./vendor/bin/phpunit

# Create cache archive from vendor/ directory
before_cache:
 - if [[ -f $HOME/vendor-cache/$cachefile ]]; rm -fv $HOME/vendor-cache/$cachefile ; fi
 - tar -cf $HOME/vendor-cache/$cachefile vendor/

Here is a fully annotated .travis.yml file with more verbose output:

sudo: false

language: php

cache:
  directories:
    - $HOME/.composer/cache
    # This is where vendor/ backups will be stored
    - $HOME/vendor-cache/

git:
  depth: 5

php:
  - […]

env:
  - SYMFONY_VERSION="2.7.*"
  - SYMFONY_VERSION="2.8.*"
  - SYMFONY_VERSION="3.0.*"

before_install:
 # Create an hash corresponding to the PHP version and the dependencies
 - echo "Vendor cache content:" ; ls -lh $HOME/vendor-cache/
 - echo "Values used for hash:"
 - tohash="${SYMFONY_VERSION}"
 - echo "$tohash"
 - cachefile="`echo -n "$tohash" | sha1sum | cut -d " " -f 1`.tar"
 - echo "cachefile = ${cachefile}"
 # Extract cache archive if the file exists
 - if [[ -f $HOME/vendor-cache/$cachefile ]]; then echo "Extract cache archive"; tar -xf $HOME/vendor-cache/$cachefile ; echo "Done" ; else echo "Cache archive does not exist" ; fi
 - if [[ -d vendor/ ]]; then echo "Size of vendor directory extracted from cache:" ; du -hs vendor/; else echo "vendor/ directory does not exist"; fi

install:
  - composer self-update
  - composer update --profile --no-progress

script: php ./vendor/bin/phpunit

# Create cache archive from vendor/ directory
before_cache:
 - if [[ -f $HOME/vendor-cache/$cachefile ]]; then echo "Delete previous cache archive"; rm -fv $HOME/vendor-cache/$cachefile ; echo "Done" ; else echo "No cache archive to delete" ; fi
 - echo "Create cache archive" ; tar -cf $HOME/vendor-cache/$cachefile vendor/ ; echo "Done"
 - echo "Size of cache archive:" ; ls -lh $HOME/vendor-cache/$cachefile

By using this method, composer update took 30 seconds, instead of about 2 minutes without cache (note that the comparison is not perfect, some other changes are applied but that's still a good estimate).

It's preferable to limit the number of parallel builds the first time you launch builds, so that caches won't suffer from a race condition.