CODEWITHSUNDEEP
pythondockeramazon-s3boto3amazon-sagemaker
AMAN GUPTA
AMAN GUPTA

Reputation: 141

AWS sagemaker error - AttributeError: 'NoneType' object has no attribute 'startswith'

As per this - How to use a pretrained model from s3 to predict some data? , I was trying to use an existing model to create an endpoint, but I was facing the following error -

    Traceback (most recent call last):
  File "/miniconda3/lib/python3.7/site-packages/gunicorn/workers/base_async.py", line 55, in handle
    self.handle_request(listener_name, req, client, addr)
  File "/miniconda3/lib/python3.7/site-packages/gunicorn/workers/ggevent.py", line 143, in handle_request
    super().handle_request(listener_name, req, sock, addr)
  File "/miniconda3/lib/python3.7/site-packages/gunicorn/workers/base_async.py", line 106, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/miniconda3/lib/python3.7/site-packages/sagemaker_sklearn_container/serving.py", line 124, in main
    serving_env.module_dir)
  File "/miniconda3/lib/python3.7/site-packages/sagemaker_sklearn_container/serving.py", line 101, in import_module
    user_module = importlib.import_module(module_name)
  File "/miniconda3/lib/python3.7/importlib/__init__.py", line 118, in import_module
    if name.startswith('.'):

As per Problem deploying the best estimator gotten with sagemaker.estimator.Estimator (w/ sklearn custom image), https://forums.aws.amazon.com/thread.jspa?threadID=313838 , I am using the correct env variables (along with SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT, SAGEMAKER_PROGRAM, and SAGEMAKER_SUBMIT_DIRECTORY), but somehow the health checks are failing while creation of the endpoint.

I tried the similar thing via AWS console and it is working surprisingly. Is there a work around for this to do it via code?

My code snippet:

trainedmodel = sagemaker.model.Model(
model_data='s3://my-bucket/my-key/output/model.tar.gz',
image='my-image',
env={"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT": "text/csv", 
     "SAGEMAKER_USE_NGINX": "True", 
     "SAGEMAKER_WORKER_CLASS_TYPE": "gevent", 
     "SAGEMAKER_KEEP_ALIVE_SEC": "60", 
     "SAGEMAKER_CONTAINER_LOG_LEVEL": "20",
     "SAGEMAKER_ENABLE_CLOUDWATCH_METRICS": "false",
     "SAGEMAKER_PROGRAM": "my-script.py",
     "SAGEMAKER_REGION": "us-east-1",
     "SAGEMAKER_SUBMIT_DIRECTORY": "s3://my-bucket/my-key/source/sourcedir.tar.gz"
    },
role=role)

trainedmodel.deploy(initial_instance_count=1, instance_type='ml.c4.xlarge', endpoint_name = 'my-endpoint')

Upvotes: 7

Views: 8916

Answers (4)

Bruno
Bruno

Reputation: 1

What worked for me was creating the model from a training job and then using the env variables listed in the question:

training_src_file     = "s3://bucket_data_assets/training-src-files.tar.gz"
churn_model = sagemaker_session.create_model_from_job(
    training_job_name=completed_training_job_name, 
    name=None, 
    role=role, image_uri=None, 
    model_data_url=None, 
    env={"SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT": "text/csv", 
     "SAGEMAKER_USE_NGINX": "True", 
     "SAGEMAKER_WORKER_CLASS_TYPE": "gevent", 
     "SAGEMAKER_KEEP_ALIVE_SEC": "60", 
     "SAGEMAKER_CONTAINER_LOG_LEVEL": "20",
     "SAGEMAKER_ENABLE_CLOUDWATCH_METRICS": "false",
     "SAGEMAKER_PROGRAM": "train.py", #the entry point present in training-src-files.tar.gz
     "SAGEMAKER_REGION": "us-east-1",
     "SAGEMAKER_SUBMIT_DIRECTORY": training_src_file,
    }

Upvotes: 0

Yash Choubey
Yash Choubey

Reputation: 29

Try using: sagemaker.model.FrameworkModel to create model with required scripts

Upvotes: 0

Deng Chuyang
Deng Chuyang

Reputation: 332

Based on your stack trace, it looks like the container cannot find your entry_point module (my-script.py).

By default, the container will add 'opt/ml/code' to the Python path and modules under this directory can be imported.

You can modify this path to other values by providing values to SAGEMAKER_BASE_PATH (default to '/opt/ml') and put your script under '<SAGEMAKER_BASE_PATH>/code' and the container will import module 'SAGEMAKER_BASE_PATH>/code/SAGEMAKER_PROGRAM'

Upvotes: 0

Delta Charlie
Delta Charlie

Reputation: 111

For anyone else landing here, I had a similar issue when creating sklearn models from a ModelPackage.

Error message in endpoint logs when trying to create endpoint:

AttributeError: 'NoneType' object has no attribute 'startswith'

Solved with the following when defining the model package:

  • Env variable SAGEMAKER_SUBMIT_DIRECTORY should be set to the directory on the container, normally '/opt/ml/model/'
  • SAGEMAKER_PROGRAM should be set to the name of the serving script, e.g. 'sagemaker_serve.py'

These are specified under the 'Environment' section part of each entry in the 'Containers' section.

Upvotes: 6

Related Questions