CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-dataflowapache-beamgoogle-cloud-pubsub
vdolez
vdolez

Reputation: 1118

Pubsub message is missing a timestamp attribute

I'm trying to read the timestamp value of a Pubsub message from Apache Beam.

p.apply("Read PubSub messages", PubsubIO.readMessagesWithAttributes()
    .withIdAttribute("msg_id")
    .withTimestampAttribute("timestamp")
    .fromSubscription(options.getPubsubSubscription()))

But unfortunately, I got the following error which really surprises me as I thought every messages had a default timestamp.

An exception occured while executing the Java class. 
PubSub message is missing a value for timestamp attribute timestamp

Why is my message not timestamped ? Is it because I published it via the Pubsub UI ?

Upvotes: 0

Views: 2042

Answers (1)

Guillem Xercavins
Guillem Xercavins

Reputation: 7058

Every Pub/Sub message will have default timestamps assigned if you omit .withTimestampAttribute(). When you add .withTimestampAttribute("timestamp") it implies that you will be providing the timestamps in the timestamp attribute of each message. For example, using the UI:

enter image description here

Then, windowing will be relative to these timestamps and, if you need to access it from within the pipeline, you can use ProcessContext.timestamp() (more details here).

Upvotes: 5

Related Questions