CODEWITHSUNDEEP
jsoncsvparsinggo
Alechko
Alechko

Reputation: 1598

Parsing nested JSON objects in a CSV file with golang

I'm trying to parse a CSV file which contains a JSON object in the last column.
Here is an example with two rows from the input CSV file:

'id','value','createddate','attributes'
524256,CAFE,2018-04-06 16:41:01,{"Att1Numeric": 6, "Att2String": "abc"}
524257,BEBE,2018-04-06 17:00:00,{}

I tried using the parser from csv package:

func processFileAsCSV(f *multipart.Part) (int, error) {
  reader := csv.NewReader(f)
  reader.LazyQuotes = true
  reader.Comma = ','
  lineCount := 0
  for {
    line, err := reader.Read()
    if err == io.EOF {
        break
    } else if err != nil {
        fmt.Println("Error:", err)
        return 0, err
    }

    if lineCount%100000 == 0 {
        fmt.Println(lineCount)
    }
    lineCount++
    fmt.Println(lineCount, line)
    processLine(line) // do something with the line
  }

  fmt.Println("done!", lineCount)
  return lineCount, nil
}

But I got an error:

Error: line 2, column 0: wrong number of fields in line,

probably because the parser ignores the JSON scope which starts with {.

Should I be writing my own CSV parser, or is there a library that can handle this?

Upvotes: 0

Views: 853

Answers (1)

Jonathan Hall
Jonathan Hall

Reputation: 79604

Your CSV input doesn't follow normal CSV convention, by using unquoted fields (for JSON).

I think the best approach would be to pre-process your input, either in your Go program, or in an external script.

If your CSV input is predictable (as indicated in your question), it should be easy to properly quote last element, using a simple strings.Split call, for instance, before passing it to the CSV parser.

Upvotes: 3

Related Questions