CODEWITHSUNDEEP
google-cloud-platformpublish-subscribegoogle-cloud-pubsubgoogle-cloud-data-fusioncdap
Luca Natali
Luca Natali

Reputation: 414

CDAP ingestion from PubSub

I'm trying to load data from PubSub messages to GCS files. Simple pipeline: PubSub source -> JSON Parser -> GCS sink.

Since PubSub only accept the data argument as utf-8, how can I decode it in CDAP? Should I build a custom plugin implementing a decode function or is it better to pass my data as string using attributes in the PuSub message instead of 'data'?

Upvotes: 0

Views: 489

Answers (1)

Luca Natali
Luca Natali

Reputation: 414

I solved the issue using a Projector plugin instead of the JSON Parser between PubSub source and GCS sink. The Projector casts the byte message attribute of the PubSub source to a string (plain text).

Upvotes: 1

Related Questions