CODEWITHSUNDEEP
javaamazon-s3apache-camel
Daniel Carl
Daniel Carl

Reputation: 23

'AWS S3 Key header missing' error when trying to upload file to Amazon S3 Using Apache Camel

I am attempting to upload a csv file from a local directory to AWS S3 using Apache Camel.

Referencing the documentation found here (https://camel.apache.org/staging/components/latest/aws-s3-component.html), I tried to create a simple route like so (I have of course removed keys and other identifying information and replaced them with [FAKE_INFO]):

from("file:fileName=${in.headers[fileName]}")
  .to("aws-s3://[BUCKET]?accessKey=[ACCESS_KEY]&secretKey=RAW([SECRET_KEY])&region=US_EAST_2&prefix=TEST.csv");

This results in the following error:

error: java.lang.IllegalArgumentException: AWS S3 Key header missing apache camel

After searching a bit online I removed the prefix that is passed and instead inserted a .setHeader to route like so:

from("file:fileName=${in.headers[fileName]}")
  .setHeader(S3Constants.KEY, simple("TEST.csv"))
  .to("aws-s3://[BUCKET]?accessKey=[ACCESS_KEY]&secretKey=RAW([SECRET_KEY])&region=US_EAST_2");

This works fine, as long as I am willing to hard code everything after the setHeader. However, for my particular use case I need to pass items from the exchange headers to feed the keys, bucket name, and fileName (this route is used by multiple files that go to different buckets based on different criteria which is received in the exchange headers). For some reason as soon as use setHeader to set the S3Constants.KEY, I am no longer able to access any of the exchange headers - in fact, I can't even assign the S3Constants.KEY value from an exchange header. As you can see, the fileName in the from section is assigned via an exchange header and I don't run into any issues there, so I know they are being received into the route.

Any thoughts on how I can modify this route so that it will allow me to upload files without the S3Constants and using exchange headers where appropriate?

Upvotes: 2

Views: 2464

Answers (1)

burki
burki

Reputation: 7005

Not sure if I understand you correct, but it sounds to me that

  • The problem of the question subject is already solved
  • Your only problem is the static destination address you want to have dynamic

To define dynamic destination addresses, there is a "dynamic to"

.toD(...)

You can use for example simple expressions in such a dynamic destination address

.toD("aws-s3://${in.header.bucket}?region=${in.header.region}&...")

See the Camel Docs (section "Dynamic To") for more details.

By the way: you write about "exchange headers". Don't confuse Exchange properties with Message headers!

  • Exchange properties are on the Exchange wrapper only and therefore lost with the Exchange after the Camel route has finished processing.
  • Message headers are on the message itself and therefore they are kept on the message even after routing it to a queue or whatever endpoint. This also implies that headers must be serializable.
  • You must access these two types differently. For example in Simple you get a header from the inbound message with ${in.header.myHeader} while you get an Exchange property with ${exchangeProperty.myProperty}

Upvotes: 1

Related Questions