CODEWITHSUNDEEP
javajsongoogle-bigquery
isaac9A
isaac9A

Reputation: 903

Streaming JSON into BigQuery using Java

I am trying to stream data into BigQuery using the Java driver similar to the tutorial on this page which inserts data into a BigQuery table from a map. The v2 of the streaming rest API supports specifying rows as JSON when inserting so I was wondering if I can stream JSON to bigquery using the Java driver rather than having to use a map like the below example.

Map<String, Object> rowContent = new HashMap<>();
rowContent.put("booleanField", true);
// Bytes are passed in base64
rowContent.put("bytesField", "Cg0NDg0="); // 0xA, 0xD, 0xD, 0xE, 0xD in base64
// Records are passed as a map
Map<String, Object> recordsContent = new HashMap<>();
recordsContent.put("stringField", "Hello, World!");
rowContent.put("recordField", recordsContent);
InsertAllResponse response = bigquery.insertAll(InsertAllRequest.newBuilder(tableId)
    .addRow("rowId", rowContent)
    // More rows can be added in the same RPC by invoking .addRow() on the builder
    .build());

i.e. is there some way to run bigquery.insertAll but pass in a json string rather than a Map?

Upvotes: 0

Views: 2118

Answers (1)

isaac9A
isaac9A

Reputation: 903

Ended up using Jackson to convert JSON string to Map using ObjectMapper class and then uploading using the same way as the example on Google's site for streaming to BigQuery in Java.

BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService();
TableId tableId = TableId.of("dataset_name", "table_name");
try {
  HashMap<String,Object> mapResult = new ObjectMapper().readValue(json_string, HashMap.class);
  InsertAllResponse response = bigquery.insertAll(InsertAllRequest.newBuilder(tableId)
    .addRow(UUID.randomUUID().toString(), mapResult)
    .build());
  if (response.hasErrors()) {
    // If any of the insertions failed, this lets you inspect the errors
    for (Entry<Long, List<BigQueryError>> entry : response.getInsertErrors().entrySet()) {
      System.out.println(entry);
    }
  }
} catch (IOException e) {
  // Failed to Map JSON String
  System.out.println(e);
}

Upvotes: 3

Related Questions