CODEWITHSUNDEEP
amazon-s3apache-drill
pramo
pramo

Reputation: 21

Issues connecting apache drill with aws s3 region which only supports Signature Version 4

I'm having issues connecting to s3 using Apache Drill storage plugin

this is the error i receive

Error: SYSTEM ERROR: AmazonS3Exception: Status Code: 400, AWS Service: Amazon S3, AWS Request ID: 53660B0E11F34387, AWS Error Code: null, AWS Error Message: Bad Request

core-site.xml 

<property>
    <name>fs.s3a.access.key</name>
    <value>accesskey</value>
</property>

<property>
    <name>fs.s3a.secret.key</name>
    <value>secretkey</value>
</property>

<property>
    <name>fs.s3a.endpoint</name>
    <value>s3.ap-south-1.amazonaws.com</value>
</property>

storage plugin

{
  "type": "file",
  "connection": "s3a://bucket/",
  "config": null,
  "workspaces": {
    "tmp": {
      "location": "/tmp",
      "writable": true,
      "defaultInputFormat": null,
      "allowAccessOutsideWorkspace": false
    },
    "root": {
      "location": "/",
      "writable": false,
      "defaultInputFormat": null,
      "allowAccessOutsideWorkspace": false
    }
  },
  "formats": {
    "psv": {
      "type": "text",
      "extensions": [
        "tbl"
      ],
      "delimiter": "|"
    },
    "csv": {
      "type": "text",
      "extensions": [
        "csv"
      ],
      "delimiter": ","
    },
    "tsv": {
      "type": "text",
      "extensions": [
        "tsv"
      ],
      "delimiter": "\t"
    },
    "parquet": {
      "type": "parquet"
    },
    "json": {
      "type": "json",
      "extensions": [
        "json"
      ]
    },
    "avro": {
      "type": "avro"
    },
    "sequencefile": {
      "type": "sequencefile",
      "extensions": [
        "seq"
      ]
    },
    "csvh": {
      "type": "text",
      "extensions": [
        "csvh"
      ],
      "extractHeader": true,
      "delimiter": ","
    }
  },
  "enabled": true
}

this works fine with us-east-1 which supports both signature 2 & 4 versions, the problem is specific to new regions which only support signature version 4. any suggestions on how to fix this ?

Upvotes: 2

Views: 165

Answers (1)

abhilash
abhilash

Reputation: 21

I had the same error occurred while trying to connect Apache Drill version 1.16.0 to Signature Version 4 only S3 bucket in MEA region. Logs indicated illegal location constraint though the bucket url had location included.

Upgraded Drill to 1.20.0 and Zookeeper to 3.5.7 and able to authenticate successfully. Hope this helps!

Upvotes: 2

Related Questions