AWS Lambda with Nodejs native modules

Question

Recently I started with AWS Lambda functions, my Nodejs application was working well until I tried to use web3.js package. After I added the line

const Web3 = require('web3');

I got the error "Internal Server Error" for the HTTP endpoint, and the following in CloudWatch logs

module initialization error: Error
at Object.Module._extensions..node (module.js:681:18)
at Module.load (module.js:565:32)
at tryModuleLoad (module.js:505:12)
at Function.Module._load (module.js:497:3)
at Module.require (module.js:596:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/var/task/node_modules/scrypt/index.js:3:20)
at Module._compile (module.js:652:30)
at Object.Module._extensions..js (module.js:663:10)
at Module.load (module.js:565:32)

Locally I have no issues to use web3.js package. So I started to dig deeper to understand what's wrong here. There are some native modules among dependencies. Some googling ends up with the idea that these packages should be compiled on Amazon Linux platform, otherwise it will not work. I started to create docker image to reach this goal.

Dockerfile

FROM amazonlinux:latest

# Install development tools
RUN yum update -y \
  && yum install gcc gcc44 gcc-c++ libgcc44 make cmake tar gzip git -y

# Install nvm and nodejs
RUN touch ~/.bashrc \
  && curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.11/install.sh | bash \
  && source ~/.bashrc \
  && nvm install v8.10

CMD source ~/.bashrc && npm install

Now in the root directory of my app I run the following command to install npm packages and compile native modules using docker image with Amazon Linux

docker run -it --rm -v $(pwd):/app -w /app docker/awslinuximage

I use serverless framework for deployment. In theory after deploy Lambda function should work, but in practice it doesn't. I found similar issues on Stackoverflow, but none helpful.

Moreover, I think this is a common problem for cloud functions to support Nodejs native modules, which should be compiled for specific OS.

Any idea and help to solve this issue appreciated. Thank you.

Answer 1

The scrypt binaries used by web3 have to be compiled on Lambda's execution environment specified in the docs for the function to work. Detailed instructions can be found in this blog post towards the end. You can use the below Dockerfile to automate the process without creating an instance.

FROM amazonlinux:2017.03.1.20170812

SHELL ["/bin/bash", "-c"] 

RUN yum update -y && \
    yum groupinstall -y "Development Tools"

# Install node using nvm 
ENV NVM_VERSION v0.33.11
ENV NVM_DIR /root/.nvm
RUN mkdir -p ${NVM_DIR}
RUN touch ~/.bashrc && chmod +x ~/.bashrc

RUN curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.11/install.sh | bash

ENV NODE_VERSION v8.10.0
RUN source ${NVM_DIR}/nvm.sh && \
    nvm install ${NODE_VERSION} && \
    nvm alias default ${NODE_VERSION} && \
    nvm use default

ENV NODE_PATH ${NVM_DIR}/${NODE_VERSION}/lib/node_modules
ENV PATH ${NVM_DIR}/versions/node/${NODE_VERSION}/bin:${PATH}

# Install global dependencies
RUN npm install -g node-gyp && \
    npm i -g serverless@1.39.1

# Set aws credentials in the image for serverless to use
# Save your aws (credentials/config) files at ./secrets.aws.(credentials/config) 
COPY ./secrets.aws.credentials /root/.aws/credentials
COPY ./secrets.aws.config /root/.aws/config

ENV APP_PATH /usr/src/app
WORKDIR ${APP_PATH}

# Install app dependencies
COPY package*.json ./
RUN npm install

COPY . .

CMD [ "sls", "deploy" ]

Build and run the Dockerfile

docker run --rm -it $(docker build -q .)

Hope it helps

Answer 2

I don't want to answer with a link, but you can find some information about there about building .node files. Otherwise known as the node-addon-api. It allows you to compile "native extensions" (files that end in .node). Yes, you need to do this for your target platform and node version. However, you won't need to spin up your Docker image on each deploy/build. You can just copy along your .node file for the ride. You could even just run Docker locally and do this too. It should vastly simplify your process.

I use this in AWS Lambda with Node.js 8.10 for another proprietary driver module and it works just fine. So I can confirm these native modules do work in AWS Lambda. In fact, the company that compiled it did so against "linux" and not some specific AWS Lambda variant of Linux from what I gather. So it seems as if it's a more forgiving approach when you have the need to compile things.