CODEWITHSUNDEEP
amazon-web-servicesgoamazon-kinesis-firehose
Sixsmith
Sixsmith

Reputation: 109

Inserting into firehose with Go

I have the following JSON file

{
    "@timestamp": "2021-11-19T21:32:55.196Z",
    "@version": "1",
    "message": "Manual test to firehose."
}

And I have the following Go function

func insertIntoFireHose(sess *session.Session, hoseName string) {
    svc := firehose.New(sess, aws.NewConfig().WithRegion("us-east-1"))
    //firehoseData := getObjectFromS3Bucket(sess)
    firehoseData, _ := os.ReadFile("/temp/test.json")

    var rec firehose.Record

    var recInput firehose.PutRecordInput

    dataJson, _ := json.Marshal(firehoseData)
    rec.SetData(dataJson)
    recInput.SetDeliveryStreamName(hoseName)
    recInput.SetRecord(&rec)

    res, err1 := svc.PutRecord(&recInput)

    if err1 != nil {
        log.Fatal(err1)
    }
    fmt.Println(res)

}

What I want to do is get a file and insert it into the firehose, but I get this error message:

{"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":
{"type":"not_x_content_exception",
"reason":"not_x_content_exception: Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"}}

And I'm not quite sure what I'm doing wrong.

Changing the record to get data directly from the file returns this error:

One or more records are malformed. Please ensure that each record is single valid JSON object and that it does not contain newlines.

Upvotes: 2

Views: 884

Answers (1)

David Yappeter
David Yappeter

Reputation: 1612

{
    "@timestamp": "2021-11-19T21:32:55.196Z",
    "@version": "1",
    "message": "Manual test to firehose.",
}

I don't think this is a valid JSON, it has a trailing comma in the 3rd line.

this is the valid one

{
    "@timestamp": "2021-11-19T21:32:55.196Z",
    "@version": "1",
    "message": "Manual test to firehose."
}

and firehoseData is already []byte, so I think you don't need to json.Marshal it again.

This is the result of the marshal

code:

package main

import (
    "encoding/json"
    "fmt"
    "os"
)

func main() {
    firehoseData, _ := os.ReadFile("./file.json") // same value

    fmt.Printf("%+v\n", string(firehoseData))

    test, err := json.Marshal(firehoseData)

    fmt.Printf("%+v\n", string(test))
    fmt.Printf("%+v\n", err)
}

output:

{
    "@timestamp": "2021-11-19T21:32:55.196Z",
    "@version": "1",
    "message": "Manual test to firehose.",   
}
"ew0KICAgICJAdGltZXN0YW1wIjogIjIwMjEtMTEtMTlUMjE6MzI6NTUuMTk2WiIsDQogICAgIkB2ZXJzaW9uIjogIjEiLA0KICAgICJtZXNzYWdlIjogIk1hbnVhbCB0ZXN0IHRvIGZpcmVob3NlLiIsDQp9"
<nil>

Upvotes: 2

Related Questions