CODEWITHSUNDEEP
amazon-web-servicesamazon-sagemaker
Uwais Iqbal
Uwais Iqbal

Reputation: 990

Sagemaker Pipelines - Unable to parse Pipeline Definition

I'm using Sagemaker Pipelines to chain together two consecutive ProcessingJobs. I'm getting a weird error when I call pipeline.upsert()

botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the CreatePipeline operation: Unable to parse pipeline definition. Property 'null' with value 'null' is not of expected type 'String'

This is what my pipeline looks like:

    step_process_data = ProcessingStep(
        name='ProcessDataStep',
        processor=script_processor,
        code=os.path.join(BASE_DIR, "scripts/preprocess.py"),
        job_arguments=job_arguments
    )
    
    step_split_data = ProcessingStep(
        name='SplitDataStep',
        processor=script_processor,
        code=os.path.join(BASE_DIR, "scripts/split_data.py"),
        job_arguments=job_arguments,
        depends_on=[step_process_data]
    )
    
    pipeline = Pipeline(
        name="DataPreperationPipeline",
        steps=[step_process_data, step_split_data],
        sagemaker_session=sagemaker_session
    )

Any thoughts on what I am doing wrong or missing?

Upvotes: 2

Views: 3742

Answers (2)

Kawin Nikomborirak
Kawin Nikomborirak

Reputation: 111

I ran into the same issue where my job_arguments were not all strings. I'd make sure all items in job_arguments are of the same type.

Upvotes: 11

Arun Lokanatha
Arun Lokanatha

Reputation: 510

Not sure if you have all the objects are set up correctly can you please follow the below example and verify.

https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker-pipelines/tabular/abalone_build_train_deploy/sagemaker-pipelines-preprocess-train-evaluate-batch-transform.ipynb

Upvotes: 1

Related Questions