CODEWITHSUNDEEP
pythonamazon-web-servicesaws-lambdadaskdask-dataframe
Raghavi Ravi
Raghavi Ravi

Reputation: 21

Dask[dataframe] "requirements are not installed" on AWS lambda

Trying to use dask.dataframe in AWS lambda. But in-spite of bundling dask.dataframe as part of bundle, I repeatedly see this error :

ImportError: Dask dataframe requirements are not installed.

Please either conda or pip install as follows:

conda install dask # either conda install
python -m pip install "dask[dataframe]" --upgrade # or python -m pip install

Tried the following:

  1. Bundling dask[complete] and uploading it as aws lambda layer
  2. Bundling dask[dataframe] and uploading it as aws lambda layer
  3. Bundling the code and dependencies and placing in s3 bucket

python version 3.8

enter image description here

Upvotes: 0

Views: 666

Answers (1)

SultanOrazbayev
SultanOrazbayev

Reputation: 16561

One potential error is that dask dataframe is not installed on worker instances. There are some possible solutions here.

I don't have much experience with running dask on Lambda with custom environments, but I used coiled to run dask-based code on Lambda. Using coiled-runtime gives a robust set of packages and there is an easy way to run dask using your own AWS account/credentials, see docs.

Upvotes: 1

Related Questions