CODEWITHSUNDEEP
javagoogle-cloud-dataflowapache-beamgoogle-cloud-pubsub
Csaba Kassai
Csaba Kassai

Reputation: 183

Get Pub/Sub message's messageId field from PubSubIO with Apache Beam

Is it possible the get somehow the messageId field of a Pub/Sub message in a DoFn after using the PubSubIO Beam source to read the messages?

I need the default id which was assigned by the Pub/Sub service. I want to log it for debugging purposes.

Using a custom attribute for the unique id and the withIdAttribute() method is not possible for me, because I have no influence on the publisher in this case.

I use the 2.2.0 version of the Dataflow Java SDK.

Upvotes: 4

Views: 2486

Answers (2)

Andrew Pilloud
Andrew Pilloud

Reputation: 478

Support for reading the Pubsub message id was added starting with Beam v2.16.0. To turn it on, replace .readMessages() with .readMessagesWithMessageId() in your pipeline setup then it is as easy as message.getMessageId() after that change.

Upvotes: 3

Nathan Nasser
Nathan Nasser

Reputation: 1004

For debugging purposes you can use the seek option.

It creates a snapshot of the messages which you can replay when needed.

Upvotes: 0

Related Questions