CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-dataflow
Prany
Prany

Reputation: 2133

Json exception on dataflow job

I am trying to load data from json file to Datastore using Dataflow job but getting error as

Error message from worker: com.google.protobuf.InvalidProtocolBufferException: com.google.gson.stream.MalformedJsonException: Unexpected value at line 1 column 2 path $ com.google.protobuf.util.JsonFormat$ParserImpl.merge(JsonFormat.java:1347) com.google.protobuf.util.JsonFormat$Parser.merge(JsonFormat.java:477) com.google.cloud.teleport.templates.common.DatastoreConverters$EntityJsonParser.merge(DatastoreConverters.java:497) com.google.cloud.teleport.templates.common.DatastoreConverters$JsonToEntity.processElement(DatastoreConverters.java

It looked like error is due to malformed json but my json is like below and is validated from jsonlint

[{
    "name1": "11",
    "name2": "haha"
}, {
    "name3": "value"
}]

Does anybody share some light whats wrong here. PS i am using template Datastore to Text Files on Cloud Storage on dataflow

Upvotes: 0

Views: 130

Answers (1)

danielm
danielm

Reputation: 3010

This template requires the input to be json-formatted Datastore Entities, not just any json objects, as mentioned here: https://cloud.google.com/dataflow/docs/guides/templates/provided-batch#cloud-storage-text-to-datastore

The format you need to adhere to is here: https://cloud.google.com/datastore/docs/reference/data/rest/v1/Entity

Upvotes: 1

Related Questions