CODEWITHSUNDEEP
dockergitlabregistryadministration
Roger Lehmann
Roger Lehmann

Reputation: 1047

GitLab: How to list registry containers with size

I have a self-hosten GitLab CE Omnibus installation (version 11.5.2) running including the container registry. Now, the disk size needed to host all those containers increase quite fast. As an admin, I want to list all Docker images in this registry including their size, so I can maybe let those get deleted.

Maybe I haven't looked hard enough but currently, I couldn't find something in the Admin Panel of GitLab. Before I make myself the work of creating a script to compare that weird linking between repositories and blobs directories in /var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2 and then aggregating the sizes based on the repositories, i wanted to ask:

Is there some CLI command or even a curl call to the registry to get the information I want?

Upvotes: 5

Views: 9344

Answers (2)

Phil
Phil

Reputation: 151

There is a Gitlab Rails Console script that is documented here: https://docs.gitlab.com/ee/administration/packages/container_registry.html#registry-disk-space-usage-by-project

You can adapt that command to increase the number of projects it will find, as it's only looking at the 100 last projects.

For example, to list all projects and their sizes (may be slow for a large number of projects):

projects_and_size = [["project_id", "creator_id", "registry_size_bytes", "project path"]]
# You need to specify the projects that you want to look through. You can get these in any manner.
Project.find_each do |p|
   project_total_size = 0
   container_repositories = p.container_repositories

   container_repositories.each do |c|
       c.tags.each do |t|
          project_total_size = project_total_size + t.total_size unless t.total_size.nil?
       end
   end

   if project_total_size > 0
      projects_and_size << [p.project_id, p.creator&.id, project_total_size, p.full_path]
   end
end

# print it as comma separated output
projects_and_size.each do |ps|
   puts "%s,%s,%s,%s" % ps
end

Note that Roger Lehmann's answer is deprecated. If your repositories are nested, you will miss projects. Additionally, from experience, it seems to under-count the disk space used by the the repositories it finds. It will also skip repositories that are created with Gitlab 14 and up.

Upvotes: 11

Roger Lehmann
Roger Lehmann

Reputation: 1047

Update: This answer is deprecated by now. Please see the accepted answer for a solution built into GitLab's Rails console directly.

Original Post:
Thanks to great comment from @Rekovni, my problem is somehwat solved.

First: The huge amount of used disk space by Docker Images was due to a bug in Gitlab/Docker Registry. Follow the link from Rekovni's comment below my question.

Second: In his link, there's also an experimental tool which is being developed by GitLab. It lists and optionally deletes those old unused Docker layers (related to the bug).

Third: If anyone wants do his own thing, I hacked together a pretty ugly script which lists the image size for every repo:

#!/usr/bin/env python3
# coding: utf-8

import os
from os.path import join, getsize
import subprocess

def get_human_readable_size(size,precision=2):
    suffixes=['B','KB','MB','GB','TB']
    suffixIndex = 0
    while size > 1024 and suffixIndex < 4:
        suffixIndex += 1
        size = size/1024.0
    return "%.*f%s"%(precision,size,suffixes[suffixIndex])


registry_path = '/var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/'
repos = []

for repo in os.listdir(registry_path + 'repositories'):
    images = os.listdir(registry_path + 'repositories/' + repo)
    for image in images:
        try:
            layers = os.listdir(registry_path + 'repositories/{}/{}/_layers/sha256'.format(repo, image))
            imagesize = 0
            # get image size
            for layer in layers:
                # get size of layer
                for root, dirs, files in os.walk("{}/blobs/sha256/{}/{}".format(registry_path, layer[:2], layer)):
                     imagesize += (sum(getsize(join(root, name)) for name in files))
            repos.append({'group': repo, 'image': image, 'size': imagesize})
        # if folder doesn't exists, just skip it
        except FileNotFoundError:
            pass

repos.sort(key=lambda k: k['size'], reverse=True)
for repo in repos:
    print("{}/{}: {}".format(repo['group'], repo['image'], get_human_readable_size(repo['size'])))

But please do note, it's really static, doesn't list specific tags for an image, doesn't take into account that some layers might be used by other images as well. But it will give you a rough estimate in case you don't want to use Gitlab's tool written above. You might use the ugly script as you like, but I do not take any liability whatsoever.

Upvotes: 6

Related Questions