CODEWITHSUNDEEP
gogoogle-bigquery
TimC
TimC

Reputation: 1061

Inserting into bigquery using go updates / overwrites instead of inserts

I have an aws lambda written in Go that should insert into bigquery. It reference the cloud.google.com/go/bigquery package.

client, err := bigquery.NewClient(ctx, projectID, gcpOption)
if err != nil {
    println(fmt.Sprintf("error creating new client, %v", err))
    return fmt.Errorf("bigquery.NewClient: %v", err)
}
defer client.Close()

inserter := client.Dataset(datasetID).Table(tableID).Inserter()
if err := inserter.Put(ctx, items); err != nil {
    println(fmt.Sprintf("error inserting, %v", err))
    if multiError, ok := err.(bigquery.PutMultiError); ok {
        for _, err1 := range multiError {
            for _, err2 := range err1.Errors {
                fmt.Println(err2)
            }
        }
    } else {
        fmt.Println(err)
    }
    return err
} else {
    println("Inserted record")
}

When run, a record will be inserted, but running again will result in the previously inserted row being updated. This is not the behaviour I was expecting. I am relatively new to Golang and GCP, so perhaps I have the wrong expectations?

The table in big query is not partitioned. Items is an array of structs.

Upvotes: 0

Views: 1104

Answers (1)

shollyman
shollyman

Reputation: 4384

The Inserter can be used to achieve at-least-once data insertion semantics. The insert mechanism is not capable of upsert behavior, which is what you appear to be describing.

It's unclear to me how you're validating this behavior, but I'd take another look at that as a starting point.

More information about the tabledata.insertAll streaming API which underlies the go Inserter can be found here: https://cloud.google.com/bigquery/streaming-data-into-bigquery

Upvotes: 1

Related Questions