CODEWITHSUNDEEP
google-bigquerygoogle-cloud-dataflow
HaimB
HaimB

Reputation: 21

Load DATE type to Google BigQuery with Dataflow

Context

I'm fairly new to the Google Cloud Platform and I'm trying out Google Dataflow. I read a CSV file and simulate streaming data. CSV row publishes into topic (PubSub) and Dataflow reads it and inserts the data into a BigQuery table.

Problem

When my file contains types STRING, FLOAT, INTEGER my process finished successfully and data is loaded to BigQuery. But if I add to my process one of those types: DATETIME, TIME, DATE it always fails. There are many code examples but I didn't find code example that show how to handle it.

Data Examples

2017-01-23 - load it into DATE type

14:10:12 - load it into TIME type

I hope you guys can help me with this ...

Upvotes: 1

Views: 1239

Answers (1)

HaimB
HaimB

Reputation: 21

I tried few things:

//1
SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
  String dateInString = info.getEventDate();
  Date date = sdf.parse(dateInString);

row.set("EventDate", date);


//2
row.set("EventDate", new DateTime("2017-01-23")); 


//3
public String getEventDate() {
return get(Field.EventDate);  }
...
...
...
row.set("EventDate", info.getEventDate());


//4
private static DateTimeFormatter dmt = DateTimeFormat.forPattern("yyyy-MM-dd");
....
....
....
DateTime ds = dmt.parseDateTime(info.getEventDate());

Upvotes: 1

Related Questions