Databricks API - Instance Pool - How to update an existing job to use instance pool instead?

Question

I am trying to update a batch of jobs to use some instance pools with the databricks api and when I try to use the update endpoint, the job just does not update. It says it executed without errors, but when I check the job, it was not updated.

What am I doing wrong?

What i used to update the job:

I used the get endpoint using the job_id to get my job settings and all I updated the resulting data with the values that i needed and executed the call to update the job.

'custom_tags': {'ResourceClass': 'Serverless'},
'driver_instance_pool_id': 'my-pool-id',
'driver_node_type_id': None,
'instance_pool_id': 'my-other-pool-id',
'node_type_id': None

I used this documentation, https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsUpdate

here is my payload

{
    "created_time": 1672165913242,
    "creator_user_name": "email@email.com",
    "job_id": 123123123123,
    "run_as_owner": true,
    "run_as_user_name": "email@email.com",
    "settings": {
        "email_notifications": {
            "no_alert_for_skipped_runs": false,
            "on_failure": [
                "email1@email.com",
                "email2@email.com"
            ]
        },
        "format": "MULTI_TASK",
        "job_clusters": [
            {
                "job_cluster_key": "the_cluster_key",
                "new_cluster": {
                    "autoscale": {
                        "max_workers": 4,
                        "min_workers": 2
                    },
                    "aws_attributes": {
                        "availability": "SPOT_WITH_FALLBACK",
                        "ebs_volume_count": 0,
                        "first_on_demand": 1,
                        "instance_profile_arn": "arn:aws:iam::XXXXXXXXXX:instance-profile/instance-profile",
                        "spot_bid_price_percent": 100,
                        "zone_id": "us-east-1a"
                    },
                    "cluster_log_conf": {
                        "s3": {
                            "canned_acl": "bucket-owner-full-control",
                            "destination": "s3://some-bucket/log/log_123123123/",
                            "enable_encryption": true,
                            "region": "us-east-1"
                        }
                    },
                    "cluster_name": "",
                    "custom_tags": {
                        "ResourceClass": "Serverless"
                    },
                    "data_security_mode": "SINGLE_USER",
                    "driver_instance_pool_id": "my-driver-pool-id",
                    "enable_elastic_disk": true,
                    "instance_pool_id": "my-worker-pool-id",
                    "runtime_engine": "PHOTON",
                    "spark_conf": {...},
                    "spark_env_vars": {...},
                    "spark_version": "..."
                }
            }
        ],
        "max_concurrent_runs": 1,
        "name": "my_job",
        "schedule": {...},
        "tags": {...},
        "tasks": [{...},{...},{...}],
        "timeout_seconds": 79200,
        "webhook_notifications": {}
    }
}

I tried to use the update endpoint and reading the docs for information but I found nothing related to the issue.

Answer 1

I finally got it

I was using partial update and found that this does not work for the whole job payload

So I changed the endpoint to use full update (reset) and it worked