CODEWITHSUNDEEP
amazon-kinesis-firehoseamazon-kinesis-kpl
wiltonio
wiltonio

Reputation: 141

Getting "Shard map update for stream xxx" failed using Amazon KPL: (AWSErrorMarshaller)Encountered Unknown AWSError

I'm trying to write to an Amazon Kinesis Firehose using the KPL library. Despite my best efforts, every attempt to connect and write to the library results in the following log lines:

2018-09-11 15:23:12.369  INFO 79765 --- [kpl-daemon-0003] c.a.s.k.producer.LogInputStreamReader    : [2018-09-11 15:23:12.368769] [0x0001379a][0x00007fff8bdd9380] [info] [main.cc:382] Starting up main producer
2018-09-11 15:23:12.369  INFO 79765 --- [kpl-daemon-0003] c.a.s.k.producer.LogInputStreamReader    : [2018-09-11 15:23:12.368833] [0x0001379a][0x00007fff8bdd9380] [info] [kinesis_producer.cc:101] Using Region: us-west-2
2018-09-11 15:23:12.369  INFO 79765 --- [kpl-daemon-0003] c.a.s.k.producer.LogInputStreamReader    : [2018-09-11 15:23:12.368850] [0x0001379a][0x00007fff8bdd9380] [info] [kinesis_producer.cc:120] Using per request threading model.
2018-09-11 15:23:12.369  INFO 79765 --- [kpl-daemon-0003] c.a.s.k.producer.LogInputStreamReader    : [2018-09-11 15:23:12.368862] [0x0001379a][0x00007fff8bdd9380] [info] [kinesis_producer.cc:171] Using Kinesis endpoint firehose.us-west-2.amazonaws.com:443
2018-09-11 15:23:12.369  INFO 79765 --- [kpl-daemon-0003] c.a.s.k.producer.LogInputStreamReader    : [2018-09-11 15:23:12.368910] [0x0001379a][0x00007fff8bdd9380] [info] [kinesis_producer.cc:101] Using Region: us-west-2
2018-09-11 15:23:12.369  INFO 79765 --- [kpl-daemon-0003] c.a.s.k.producer.LogInputStreamReader    : [2018-09-11 15:23:12.368926] [0x0001379a][0x00007fff8bdd9380] [info] [kinesis_producer.cc:120] Using per request threading model.
2018-09-11 15:23:12.369  INFO 79765 --- [kpl-daemon-0003] c.a.s.k.producer.LogInputStreamReader    : [2018-09-11 15:23:12.368940] [0x0001379a][0x00007fff8bdd9380] [info] [kinesis_producer.cc:52] Using default CloudWatch endpoint
2018-09-11 15:23:12.369  INFO 79765 --- [kpl-daemon-0003] c.a.s.k.producer.LogInputStreamReader    : [2018-09-11 15:23:12.369002] [0x0001379a][0x00007fff8bdd9380] [info] [main.cc:393] Entering join
2018-09-11 15:23:12.373  INFO 79765 --- [kpl-daemon-0003] c.a.s.k.producer.LogInputStreamReader    : [2018-09-11 15:23:12.372933] [0x0001379a][0x000070000c36a000] [info] [kinesis_producer.cc:197] Created pipeline for stream "redshift-guid-test"
2018-09-11 15:23:12.373  INFO 79765 --- [kpl-daemon-0003] c.a.s.k.producer.LogInputStreamReader    : [2018-09-11 15:23:12.373005] [0x0001379a][0x000070000c36a000] [info] [shard_map.cc:79] Updating shard map for stream "redshift-guid-test"
2018-09-11 15:23:13.102  WARN 79765 --- [kpl-daemon-0003] c.a.s.k.producer.LogInputStreamReader    : [2018-09-11 15:23:13.102494] [0x0001379a][0x000070000c805000] [warning] [AWS Log: WARN](AWSErrorMarshaller)Encountered Unknown AWSError
UnknownOperationException
:

2018-09-11 15:23:13.103 ERROR 79765 --- [kpl-daemon-0003] c.a.s.k.producer.LogInputStreamReader    : [2018-09-11 15:23:13.102572] [0x0001379a][0x000070000c805000] [error] [shard_map.cc:150] Shard map update for stream "redshift-guid-test" failed. Code: UnknownOperationException Message: Unable to parse ExceptionName: UnknownOperationException Message: ; retrying in 1000 ms
2018-09-11 15:23:14.002  WARN 79765 --- [kpl-daemon-0003] c.a.s.k.producer.LogInputStreamReader    : [2018-09-11 15:23:14.002252] [0x0001379a][0x000070000c90b000] [warning] [AWS Log: WARN](AWSErrorMarshaller)Encountered Unknown AWSError
UnknownOperationException

I am able to get a client connected and sending data using the standard AWS Client.

Here is how I'm creating the client.

       KinesisProducerConfiguration config = new KinesisProducerConfiguration()
            .setCredentialsProvider(credentials.getTemporaryCredentialsProvider())
            .setKinesisEndpoint("firehose.us-west-2.amazonaws.com")
            .setRegion(awsConfig.getRegion());

    KinesisProducer kinesisProducer = new KinesisProducer(config);

Upvotes: 2

Views: 744

Answers (1)

wiltonio
wiltonio

Reputation: 141

I was attempting to connect KPL to a firehose. I have just discovered that this is not supported. Lord knows why not.

Upvotes: 1

Related Questions