CODEWITHSUNDEEP
google-cloud-datastoregoogle-cloud-dataflow
Malte
Malte

Reputation: 604

Google Dataflow writing insufficient data to datastore

One of my Batch-Jobs tonight failed with a Runtime-Exception. It writes Data to Datastore like 200 other jobs that were running tonight. This one failed with a very long list auf causes, the root of it should be this:

Caused by: com.google.datastore.v1.client.DatastoreException: I/O error, code=UNAVAILABLE
    at com.google.datastore.v1.client.RemoteRpc.makeException(RemoteRpc.java:126)
    at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:95)
    at com.google.datastore.v1.client.Datastore.commit(Datastore.java:84)
    at com.google.cloud.dataflow.sdk.io.datastore.DatastoreV1$DatastoreWriterFn.flushBatch(DatastoreV1.java:925)
    at com.google.cloud.dataflow.sdk.io.datastore.DatastoreV1$DatastoreWriterFn.processElement(DatastoreV1.java:892)
Caused by: java.io.IOException: insufficient data written
    at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.close(HttpURLConnection.java:3501)
    at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:81)
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
    at com.google.datastore.v1.client.RemoteRpc.call(RemoteRpc.java:87)
    at com.google.datastore.v1.client.Datastore.commit(Datastore.java:84)
    at com.google.cloud.dataflow.sdk.io.datastore.DatastoreV1$DatastoreWriterFn.flushBatch(DatastoreV1.java:925)
    at com.google.cloud.dataflow.sdk.io.datastore.DatastoreV1$DatastoreWriterFn.processElement(DatastoreV1.java:892)
    at com.google.cloud.dataflow.sdk.util.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:49)
    at com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.processElement(DoFnRunnerBase.java:139)
    at com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:188)
    at com.google.cloud.dataflow.sdk.runners.worker.ForwardingParDoFn.processElement(ForwardingParDoFn.java:42)
    at com.google.cloud.dataflow.sdk.runners.

How can this happen? It's very similar to all the other jobs I run. I am using the Dataflow-Version 1.9.0 and the standard DatastoreIO.v1().write....

The jobIds with this error message:

2017-08-29_17_05_19-6961364220840664744

2017-08-29_16_40_46-15665765683196208095

Is it possible to retrieve the errors/logs of a job from an outside application (Not cloud console) to automatically being able to restart jobs, if they would usually succeed and fail because of quota-issues or other reasons that are temporary? Thanks in advance

Upvotes: 0

Views: 298

Answers (1)

jkff
jkff

Reputation: 17913

This is most likely because DatastoreIO is trying to write more mutations in one RPC call than the Datastore RPC size limit allows. This is data-dependent - suppose the data for this job differs somewhat from data for other jobs. In any case: this issue was fixed in 2.1.0 - updating your SDK should help.

Upvotes: 0

Related Questions