Installing gems in Dockerfile without Gemfile

Question

My team hosts our Rails application out of a Dockerfile. We have a few slow gems that are really slowing down our builds. (I'm looking at yougrpc. 😠)

Is it possible to install a few gems before copying the Gemfile into our Dockerfile? This would allow Docker to cache those build steps, so we don't have to reinstall the slow gems every time the Gemfile changes.

I tried this, but bundle install is still installing grpc, sassc and nokogiri.

RUN gem install grpc --version 1.28.0
RUN gem install sassc --version 2.2.1
RUN gem install nokogiri --version 1.10.9

WORKDIR /app

ADD Gemfile Gemfile.lock .ruby-version /app/
RUN bundle install

Answer 1

You could use the options bundle install --deployment or --path to specify a folder where you could install your gems (build a container only for this purpose and then copy the folder outside docker). Then map this directory as a volume in your container, running bundle install after the mapping is done... or simply copy the folder:

# outside actual image

bundle install --path=vendor/cache

# actual docker image
COPY . /app
COPY vendor/cache /app/vendor/cache
COPY .bundle /app/.bundle 

WORKDIR /app

RUN bundle install --path=vendor/cache

The command with --path=vendor/cache generates a bundle config just as this one:

---
BUNDLE_DEPLOYMENT: "true"
BUNDLE_PATH: "vendor/cache"

As long as your compiled-extension gems versions don't change on your Gemfile, they shouldn't be installed again.

In order to not commit the vendor/cache folder to source-files, you could host it somewhere and have developers download it. I think where you could leave this folder, and how to copy it to the image, is a rather open-ended matter and I'd need more information about your setup to recommend something. Where exactly are you deploying your app in production?

EXAMPLE:

AWS lambda gems with native extensions presents a problem - you might install them inside your OS system but they will not be compatible with the lambda environment.

The approach to make a gem bundle which works is to do the following:

1 - Use a container to install the gems with native extensions, in a way that is compatible with lambda:

docker run --rm -v "$PWD":/var/task lambci/lambda:build-ruby2.7 bundle install --deployment

2 - Then, on another container, you MAP THE GEMS as a volume or copy them, and run your application

docker run --rm --env-file=.env -v $PWD:/var/task:ro,delegated lambci/lambda:ruby2.7 send.lambda_handler

I suggest here that you do something similar for your application - use one image to build your platform gems, and then another to run the application, passing these gems to it. As long as those two images are the same OS, the gems will be compatible.

A setup similar to this one has worked for my team on a non-docker deployment pipeline, where we pre-install the gems in a folder before pushing a new version and setup bundler to get the gems from it.

Answer 2

Problem:

So let's understand the problem first. Whenever you try to spin up your application it takes so much time. I think I understand your problem very well.

1. download gem from internet
2. install and spinup application

Possible solutions:

This is a performance of docker files. we have 2 areas which are creating the problems, and 2 different solutions

Solutions:

• Download gem files locally put in a folder and in dockerfile copy those gem files inside the container and then install.

• Create a base image for your application. Create a base image from Ruby/any other you like and install your gems either using the above method:

  1. copy gems from the local folder and then paste inside the container or
  2. Use the RUN command to install the gems. Whatever you like.

This will be a one time process. With this, you can have your base image which already contains your time-consuming gems installed. Now inside your application dockerfile (which is a responsible for the startup the application), you just need to use your own created base image instead of Ruby or Linux from Docker Hub

We will see how to build your own base pre-configured image.
Let us see step by step.

Let's follow this GitHub repo: https://github.com/dupinder/docker-ruby-gem-game

Folder Structure (This will help to understand this article)

• application/dockerfile
• gems/download and place local gems
• dockerfile

---

• Create a base image and install local gems inside docker container.

dockerfile

    FROM ruby:latest
    RUN mkdir -p /gems 
    COPY /gems/grpc-1.28.0-universal-darwin.gem /gems/grpc-1.28.0-universal-darwin.gem
    COPY /gems/sassc-2.2.1.gem /gems/sassc-2.2.1.gem
    WORKDIR /gems
    RUN gem install --force --local *.gem

Build an image from this dockerfile using follwing command

docker build --rm -f "dockerfile" -t ruby-gem-base-image:latest "."

• Step 1: I am using the base image as ruby, you can your whatever you want if you are using Linux then in next step you need to install Ruby

• Step 2: Create a folder name gems in the container.

• Step 3 & 4: Copy the Gems you need to install inside the container from local directory to the directory inside docker container.
• Step 5 & 6: Change the working directory to gems, because we want to install gems place inside this directory, So next command is gem install --force --local *.gem which help to install gems inside a local directory.

With this, we solved 50% time-consumption. Now docker will never download gems from internet each time and install.

Now let us check is there any of our required gems installed our not. For that:

• Run command docker images we will have our newly build image ruby-gem-base-image

• Run container in detached mode, so that we can exec later on docker run -it -d ruby-gem-base-image

• Run docker ps to get container ID.

• Run exec as bash inside container docker exec -it d28234630343 bash.
• Run gem list This will print a list of gems installed, and you will see your required gems there.

See your gems are installed from local directory.

---

If you follow these steps your problem is resolved. But Now you need if before starting your app docker container already have installed gems.

For this problem, we can use ruby-gem-base-image image as our ruby application base image. If you remember GitHub repo we have an application directory which has one dockerfile if we see that.

FROM ruby-gem-base-image:latest
CMD ["gem", "list"]

This is your dockerfile which can be used when you want to deploy an application, Use prebuilt docker image which has your gems. I write the task gem list to check is this container have gems from parent image or not.

I think this is a bit clear. Your problem will be resolved with this.

If you need any other help or need help to understand this process, please ask.

---------Dupinder.

Answer 3

You can think about splitting your gemfile, think about the following files.

slow-gems

ruby File.read('.ruby-version').strip

gem 'rubocop'

gemfile

ruby File.read('.ruby-version').strip

# add the "slow" gems to the gem-bundle so we do not have to redefine them.
instance_eval File.read('slow-gems')

gem 'flay'

dockerfile

WORKDIR /app

ADD slow-gems slow-gems.lock .ruby-version /app/
RUN bundle install --gemfile=slow-gems

ADD Gemfile Gemfile.lock /app/
RUN bundle install

This also prevents you from redefining all the gems in the docker image and the gemfile. The only problem you might encounter that the version in both lock-files will drift apart. For now, I don't have a solution for that, but this can also happen while using your current method of adding them to the dockerfile.

The second bundle install will re-use the already installed gem from the slow-gems-file, this takes less than a second.

Adition: Don't forget to use docker's built-in caching, else this will not be faster and does not help you.