Managing (older) CI artifacts in GitLab CE Omnibus

Question

Good day all,

I run a GitLab CE Omnibus (8.4.3) installation for my company. We recently started using CI, but still mainly for generating documentation. Binary builds is in the process of being added.

As part of the system, I run a daily backup task (using gitlab-rake gitlab:backup:create). Over the last couple of days, these artifacts have started becoming extremely large, even though it is just documentation at this stage (doxygen generated images seem to be the major source of the problem). Because artifacts are included in the backup, daily backups (current strategy keeps 2 weeks' worth) have become unwieldy in size.

I've searched a fair bit for some answers, but most of the hits I get seem to be related to feature requests at this stage. I'm aware that CI is still relatively new, but I would imagine artifact management is reasonably high on the requirements list.

I have three questions:

1. Is there a way to manage/delete older artifacts in GitLab, short of manually deleting them from the disk (and possibly breaking links in the process)? It would be ideal if this process could be automated.

2. Following on from 1, is it possible to target specific classes of artifacts in the cleaning strategy (ie, only delete older documentation targets, but not binaries, etc)?

3. Is it possible to either entirely, or based on CI target types, exclude artifacts from the recommended gitlab-rake gitlab:backup:create backup procedure?

Any links, tips or advice would be highly appreciated!

Kind regards,

[UPDATE] Some more reading, yielded the following:

1. As of Gitlab 8.5, I can manually delete single build artifacts. This helps, but is not scalable. Timeline for proper artifact management (including expiry dates, etc) seems to be Gitlab 8.7.

2. There seems to be no clear request for handling of different build target artifacts in different ways yet.

3. No information on removing artifacts from the backup task.

Answer 1

You can remove old artifacts from the file system. However this will result in 404 errors if you try to download them using the GitLab UI. In my case no one cares.

sudo du -hd 1 /var/opt/gitlab/gitlab-rails/shared/artifacts
sudo rm -rf /var/opt/gitlab/gitlab-rails/shared/artifacts/[whatever you can spare]

See also https://docs.gitlab.com/omnibus/settings/configuration.html#disable-storage-directories-management

Answer 2

As mentioned in my question, better support for managing artifacts is due probably sometime around Gitlab 8.7. See the earlier answer by Marco van Neerbos about removing artifacts from backups.

For the moment, these are the options for managing existing artifacts:

1. Upgrade to Gitlab 8.5, which will allow you to manually remove individual builds (you still need to track them down, and click the 'Erase' button).
2. Adapted from Gitlab CE Issue 5572 with fixes, use the following to do bulk erase (and database updates) if you know the group and project name:

$ sudo gitlab-rails console
> p = Project.find_with_namespace("group/project")
> p.builds.where.not(artifacts_file: nil).find_each(&:remove_artifacts_file!)
> p.builds.where.not(artifacts_metadata: nil).find_each(&:remove_artifacts_metadata!)

The remove_* commands will return nil, but if you check on disk, the files will be gone, and the links removed from the relevant builds pages.

3. Currently, I can find no information on restricting any artifact related functions to specific types/classes/targets of artifacts.

Answer 3

I would be very interested in finding out answers to 1 & 2 too.

For excluding items on the backup task try the SKIP argument:

gitlab-rake gitlab:backup:create SKIP=artifacts,builds

You can also SKIP repositories,lfs,uploads. Just supply them as a comma separated list. See also https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/raketasks/backup_restore.md