CODEWITHSUNDEEP
jsongo
max pleaner
max pleaner

Reputation: 26758

Need to parse integers in JSON as integers, not floats

First of all let me explain the problem.

I have a stream of JSON records coming into my Golang app. It basically forwards these to a data store (InfluxDB). There are some integer values in the JSON, and also some float values. It is essential that these get forwarded to the data store with the original data type. If they don't, there will be type conflicts and the write operation will fail.

The Ruby JSON parser has no trouble doing this:

require 'json'
obj = { "a" => 123, "b" => 12.3 }
parsed = JSON.parse(obj.to_json)

print parsed["a"].class # => Integer
print parsed["b"].class # => Float

The encoding/json package in Golang, does have some trouble (all numbers are parsed as floats):

package main

import "encoding/json"
import "fmt"

func main () {
  str := "{\"a\":123,\"b\":12.3}"
  var parsed map[string]interface{}
  json.Unmarshal([]byte(str), &parsed)
  for key, val := range parsed {
    switch val.(type) {
    case int:
      fmt.Println("int type: ", key)
    case float64:
      fmt.Println("float type: ", key)
    default:
      fmt.Println("unknown type: ", key)
    }
  }
}

Which prints:

float type:  a
float type:  b

I need a way to parse ints as ints, and floats as floats, in the way the Ruby JSON parser does.

It is not feasible in this case to parse everything as strings and check whether or not there is a decimal. Certain values come in as strings such as "123" and I need to push those as strings.

I do not have structs for the parsed objects, nor is that an option. The golang app doesn't actually care about the schema and just forwards the input as it receives it.

I tried the approach outlined here: Other ways of verifying reflect.Type for int and float64 (using reflect) but it did not accurately identify the int:

t := reflect.TypeOf(parsed["a"])
k := t.Kind()
k == reflect.Int // false
k == reflect.Float64 // true

Upvotes: 3

Views: 9357

Answers (2)

peterSO
peterSO

Reputation: 166529

For example, Ruby JSON number types using the general Go mechanism for custom JSON values,

package main

import (
    "encoding/json"
    "fmt"
    "strconv"
)

func main() {
    str := `{"a":123,"b":12.3,"c":"123","d":"12.3","e":true}`
    var raw map[string]json.RawMessage
    err := json.Unmarshal([]byte(str), &raw)
    if err != nil {
        panic(err)
    }
    parsed := make(map[string]interface{}, len(raw))
    for key, val := range raw {
        s := string(val)
        i, err := strconv.ParseInt(s, 10, 64)
        if err == nil {
            parsed[key] = i
            continue
        }
        f, err := strconv.ParseFloat(s, 64)
        if err == nil {
            parsed[key] = f
            continue
        }
        var v interface{}
        err = json.Unmarshal(val, &v)
        if err == nil {
            parsed[key] = v
            continue
        }
        parsed[key] = val
    }
    for key, val := range parsed {
        fmt.Printf("%T: %v %v\n", val, key, val)
    }
}

Playground: https://play.golang.org/p/VmG8IZV4CG_Y

Output:

int64: a 123 
float64: b 12.3 
string: c 123 
string: d 12.3 
bool: e true

Another example, Ruby JSON number types using the Go json.Number type,

package main

import (
    "encoding/json"
    "fmt"
    "strings"
)

func main() {
    str := `{"a":123,"b":12.3,"c":"123","d":"12.3","e":true}`
    var parsed map[string]interface{}
    d := json.NewDecoder(strings.NewReader(str))
    d.UseNumber()
    err := d.Decode(&parsed)
    if err != nil {
        panic(err)
    }
    for key, val := range parsed {
        n, ok := val.(json.Number)
        if !ok {
            continue
        }
        if i, err := n.Int64(); err == nil {
            parsed[key] = i
            continue
        }
        if f, err := n.Float64(); err == nil {
            parsed[key] = f
            continue
        }
    }
    for key, val := range parsed {
        fmt.Printf("%T: %v %v\n", val, key, val)
    }
}

Playground: https://play.golang.org/p/Hk_Wb0EM-aY

Output:

int64: a 123
float64: b 12.3
string: c 123
string: d 12.3
bool: e true

A working version of @ShudiptaSharma's suggestion.

Upvotes: 5

Shudipta Sharma
Shudipta Sharma

Reputation: 5842

There exists a type json.Number in json package which has 3 format functions String() string, Float64() (float64, error) and Int64() (int64, error). We can use them to parse integer and float type.

So, I handle json integer parsing in this way:

package main

import "encoding/json"
import "fmt"
import "strings"
import (
    "reflect"
)

func main () {
    str := "{\"a\":123,\"b\":12.3}"

    var parsed map[string]interface{}
    d := json.NewDecoder(strings.NewReader(str))
    d.UseNumber()
    fmt.Println(d.Decode(&parsed))

    for key, val := range parsed {
        fmt.Println(reflect.TypeOf(val))
        fmt.Printf("decoded to %#v\n", val)
        switch val.(type) {
        case json.Number:
            if n, err := val.(json.Number).Int64(); err == nil {
                fmt.Println("int64 type: ", key, n)
            } else if f, err := val.(json.Number).Float64(); err == nil {
                fmt.Println("float64 type: ", key, f)
            } else {
                fmt.Println("string type: ", key, val)
            }
        default:
            fmt.Println("unknown type: ", key, val)
        }
        fmt.Println("===============")
    }
}

Upvotes: 2

Related Questions