Unable to import Pandas in AWS Lambda layer

Question

I uploaded python pandas to Lambda and when I run the Lambda, I see the following error:

"errorMessage": "Unable to import module 'lambda_function': C extension: No module named 'pandas._libs.interval' not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first."

I am really not sure what is supposed to be going on here. But for more context, I created a directory called "python" and then I run the following line on my terminal

python3.8 -m pip install pandas -t .

I then zip the "python" directory and then create a new layer and upload the zip file.

Struggling to see where I am following short.

Answer 1

If you are working with pandas on AWS, it is better to use the AWS SDK for pandas (AWS Datawrangler) instead of installing pandas or importing a third party layer.

You can learn more about installing by clicking on the following link:

https://aws-sdk-pandas.readthedocs.io/en/stable/install.html

Just add layer from the AWS layers:

Or via its ARN:

arn:aws:lambda:<region>:336392948345:layer:AWSSDKPandas-Python<python-version>:<layer-version>.

For example:

arn:aws:lambda:us-east-1:336392948345:layer:AWSSDKPandas-Python37:1.

The full list of ARNs is available here.

Answer 2

Instead of installing the pandas through pip, you could try with the wheel files as well. This worked for me. Just download the required wheel files, and unpack the files into the directory which you are planning to upload as a layer, zip the directory, and upload it to the lambda layers.

Refer to this for step by step guide...

Answer 3

Have you considered using publicly available layer for pandas, for example from this popular repository:keithrozario/Klayers?

For instance, the list of layers available for python 3.8 is here for us-east-1.

In this case, for us-east-1 you could add the pandas layer using:

arn:aws:lambda:us-east-1:770693421928:layer:Klayers-python38-pandas:16

Update, making custom layer

I just created the custom layer with pandas and xlrd and can confirm that it works.

The technique used includes docker tool described in the recent AWS blog:

• How do I create a Lambda layer using a simulated Lambda environment with Docker?

Thus for this question, I verified it as follows:

1. Create empty folder, e.g. mylayer.

2. Go to the folder and create requirements.txt file with the content of

pandas
xlrd

3. Run the following docker command:

docker run -v "$PWD":/var/task "lambci/lambda:build-python3.8" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.8/site-packages/; exit"

4. Create layer as zip:

zip -r mypandaslayer.zip python > /dev/null

5. Create lambda layer based on mypandaslayer.zip in the AWS Console. Don't forget to specify Compatible runtimes to python3.8.

6. Test the layer in lambda using the following lambda function:

import json

import pandas
import xlrd

def lambda_handler(event, context):
    
    print(dir(pandas))
    print(dir(xlrd))

The function executes correctly:

['BooleanDtype', 'Categorical', 'CategoricalDtype', 'CategoricalIndex', 'DataFrame', 'DateOffset', 'DatetimeIndex', 'DatetimeTZDtype', 'ExcelFile', 'ExcelWriter', 'Float64Index', 'Grouper', 'HDFStore', 'Index', 'IndexSlice', 'Int16Dtype', 'Int32Dtype', 'Int64Dtype', 'Int64Index', 'Int8Dtype', 'Interval', 'IntervalDtype', 'IntervalIndex', 'MultiIndex', 'NA', 'NaT', 'NamedAgg', 'Period', 'PeriodDtype', 'PeriodIndex', 'RangeIndex', 'Series', 'SparseDtype', 'StringDtype', 'Timedelta', 'TimedeltaIndex', 'Timestamp', 'UInt16Dtype', 'UInt32Dtype', 'UInt64Dtype', 'UInt64Index', 'UInt8Dtype', '__builtins__', '__cached__', '__doc__', '__docformat__', '__file__', '__getattr__', '__git_version__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_config', '_hashtable', '_is_numpy_dev', '_lib', '_libs', '_np_version_under1p16', '_np_version_under1p17', '_np_version_under1p18', '_testing', '_tslib', '_typing', '_version', 'api', 'array', 'arrays', 'bdate_range', 'compat', 'concat', 'core', 'crosstab', 'cut', 'date_range', 'describe_option', 'errors', 'eval', 'factorize', 'get_dummies', 'get_option', 'infer_freq', 'interval_range', 'io', 'isna', 'isnull', 'json_normalize', 'lreshape', 'melt', 'merge', 'merge_asof', 'merge_ordered', 'notna', 'notnull', 'offsets', 'option_context', 'options', 'pandas', 'period_range', 'pivot', 'pivot_table', 'plotting', 'qcut', 'read_clipboard', 'read_csv', 'read_excel', 'read_feather', 'read_fwf', 'read_gbq', 'read_hdf', 'read_html', 'read_json', 'read_orc', 'read_parquet', 'read_pickle', 'read_sas', 'read_spss', 'read_sql', 'read_sql_query', 'read_sql_table', 'read_stata', 'read_table', 'reset_option', 'set_eng_float_format', 'set_option', 'show_versions', 'test', 'testing', 'timedelta_range', 'to_datetime', 'to_numeric', 'to_pickle', 'to_timedelta', 'tseries', 'unique', 'util', 'value_counts', 'wide_to_long']
['Book', 'FMLA_TYPE_ARRAY', 'FMLA_TYPE_CELL', 'FMLA_TYPE_COND_FMT', 'FMLA_TYPE_DATA_VAL', 'FMLA_TYPE_NAME', 'FMLA_TYPE_SHARED', 'MMAP_AVAILABLE', 'Operand', 'Ref3D', 'USE_MMAP', 'X12Book', 'XLDateError', 'XLRDError', 'XL_CELL_BLANK', 'XL_CELL_BOOLEAN', 'XL_CELL_DATE', 'XL_CELL_EMPTY', 'XL_CELL_ERROR', 'XL_CELL_NUMBER', 'XL_CELL_TEXT', '__VERSION__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', 'biff_text_from_num', 'biffh', 'book', 'cellname', 'cellnameabs', 'colname', 'compdoc', 'count_records', 'decompile_formula', 'dump', 'dump_formula', 'empty_cell', 'error_text_from_code', 'evaluate_name_formula', 'formatting', 'formula', 'info', 'mmap', 'oBOOL', 'oERR', 'oNUM', 'oREF', 'oREL', 'oSTRG', 'oUNK', 'okind_dict', 'open_workbook', 'os', 'pprint', 'rangename3d', 'rangename3drel', 'sheet', 'sys', 'timemachine', 'xldate', 'xldate_as_datetime', 'xldate_as_tuple', 'xlsx', 'zipfile']