CODEWITHSUNDEEP
rubyaws-lambdapgaws-samaws-sam-cli
Valeria Cavallaro
Valeria Cavallaro

Reputation: 11

AWS SAM Include gem with native dependencies

I want to run my lambda function locally (I am currently using ruby 2.7.1), but when I require a gem that needs native dependencies it fails because it doesn't find them.

I tried to use the 'pg' gem to connect to a postgresql database. Then I proceeded to run sam build and sam local invoke HelloWorldFunction --event events/event.json, which failed with the next error:

Invoking app.lambda_handler (ruby2.7)
Failed to download a new amazon/aws-sam-cli-emulation-image-ruby2.7:rapid-1.0.0 image. 
Invoking with the already downloaded image.
Mounting /home/user/sam-app/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated 
inside runtime container
Init error when loading handler app.lambda_handler
{
  "errorMessage": "libpq.so.5: cannot open shared object file: No such file or directory - /var/task/vendor/bundle/ruby/2.7.0/gems/pg-1.2.3/lib/pg_ext.so",
  "errorType": "Init<LoadError>",
  "stackTrace": [
    "/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'",
    "/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'",
    "/var/task/vendor/bundle/ruby/2.7.0/gems/pg-1.2.3/lib/pg.rb:5:in `<top (required)>'",
    "/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'",
    "/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'",
    "/var/task/app.rb:3:in `<top (required)>'",
    "/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'",
    "/var/lang/lib/ruby/site_ruby/2.7.0/rubygems/core_ext/kernel_require.rb:92:in `require'"
  ]
}

Next thing I tried was to execute sam build --use-container and got a Gem::Ext::BuildError with the message ERROR: Failed to build gem native extension.

It seems that the external libraries that the gem needs are not being included.

My questions are:

I read something about using Layers for this but I don't fully understand, as it's my first time working with lambda functions.

It also seems that the sam proyect has some open issues about solving this, but it won't be in the near future.

Any help will be very much appreciated! Thank you!

Upvotes: 1

Views: 1370

Answers (1)

Dan Leyden
Dan Leyden

Reputation: 319

Layers in Lambda basically give you a way of installing dependencies that aren't related directly to your function but are used by it. It is a really great way to support re-use and to speed up build and deploy times. The idea here is that you would install common or shared dependencies like the postgresql libraries into the layer, then just reference them from your function.

In short, they give you a way to place things on the filesystem that the lambda can access at runtime.

However, there are a few interlinked issues in this case:

  • the pg gem requires packages to be installed on the OS and files to be available at build-time
  • the pg gem does not install the dependencies itself

This means that using layers is a little less simple than would be ideal.

For your issue, try the following:

  1. Create a new directory called pg_layer
  2. In that directory, create a file called Makefile (the case is important) with the following content:
LIB_DIR = $(ARTIFACTS_DIR)/lib
LIB_FILES = \
  /usr/lib64/libpq.so.* \
  /usr/lib64/libldap_r-2.4.so.2* \
  /usr/lib64/liblber-2.4.so.* \
  /usr/lib64/libsasl2.so.* \
  /usr/lib64/libssl3.so \
  /usr/lib64/libsmime3.so \
  /usr/lib64/libnss3.so


build-PGLayer:
  yum install -y amazon-linux-extras
  yum install -y postgresql-devel postgresql-libs
  mkdir -p $(ARTIFACTS_DIR)/vendor/bundle
  mkdir -p $(LIB_DIR)
  for f in $(LIB_FILES); do cp $$f $(LIB_DIR); done
  cp -r . $(ARTIFACTS_DIR)
  mkdir -p $(ARTIFACTS_DIR)/ruby/gems
  gem install pg --install-dir $(ARTIFACTS_DIR)/ruby/gems/2.7.0
  1. Alter your template.yaml file, and add the below to your Resources section:
  PGLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      CompatibleRuntimes:
        - ruby2.7
      ContentUri: 'pg_layer'
    Metadata:
      BuildMethod: makefile
  1. In the template.yaml file, alter your function definition to add the below to the Properties:
      Layers:
        - !Ref PGLayer
  1. Remove pg from your function's Gemfile (it will be available automatically because of the layer and leaving it there will prevent you from building the function)

When you're ready to build, use the command sam build --use-container. To run your lambda function locally with the layer, use sam local start-api. When you're ready to deploy sam deploy will push up your entire application, including the layer, and configure the function to use the layer.

Upvotes: 4

Related Questions