CODEWITHSUNDEEP
jsontypesf#f#-data
primfaktor
primfaktor

Reputation: 2997

JsonProvider gives me different types that I cannot unify

I'm using the JSON type provider to load a JSON file that I created. The minimal input for the type provider looks like this:

{
  "conv1": {
    "weight": {
      "shape": [ 64, 3, 7, 7 ],
      "data": [ 1e-30, -0.01077061053365469 ]
    }
  },
  "bn1": {
    "eps": 1e-05,
    "weight": {
      "shape": [ 64 ],
      "data": [ 1e-30, 0.2651672959327698 ]
    },
    "bias": {
      "shape": [ 64 ],
      "data": [ 1e-30, 0.24643374979496002 ]
    }
  }
}

Although both weight parts have the same shape and type, the type provider gives me two different but equivalent types:

type Weight =
    inherit IJsonDocument
    new : shape: int [] * data: float [] -> Weight
    member Data : float []
    member JsonValue: JsonValue
    member Shape: int []

and

type Weight2 =
    inherit IJsonDocument
    new : shape: int [] * data: float [] -> Weight2
    member Data : float []
    member JsonValue: JsonValue
    member Shape: int []

First, that's not nice, but maybe it cannot figure out that they mean the same. So I sat down and tried to write a function that unifies both so that I can continue from there—I failed.

My first approach was to use overloading:

type Tensor = {
    Data:single[]
    Shape:int list
} with
    static member Unify1 (w:NN.Weight) = { Data = w.Data |> Array.map single; Shape = w.Shape |> Array.toList }
    static member Unify1 (w:NN.Weight2) = { Data = w.Data |> Array.map single; Shape = w.Shape |> Array.toList }

Error FS0438 Duplicate method. The method Unify1 has the same name and signature as another method in type Tensor once tuples, functions, units of measure and/or provided types are erased.

Then I tried manual type testing like this:

let unify2 (o:obj) =
    match o with
    | :? NN.Weight as w -> { Data = w.Data |> Array.map single; Shape = w.Shape |> Array.toList }
    | :? NN.Weight2 as w -> { Data = w.Data |> Array.map single; Shape = w.Shape |> Array.toList }
    | _ -> failwith "pattern oops"

This variant does not compile because

Error FS3062 This type test with a provided type JsonProvider<...>.Weight is not allowed because this provided type will be erased to Runtime.BaseTypes.IJsonDocument at runtime.

How can I get the type provider to produce a unified type? Alternatively, how would I go about unifying them myself while making the compiler happy?

Upvotes: 2

Views: 161

Answers (1)

Tomas Petricek
Tomas Petricek

Reputation: 243051

I agree that the inferred type is not as nice as it could be. The XML type provider has a static parameter Global which unifies XML elements across the whole document based on their name - so presumably, the JSON provider could do something similar (but it is more tricky, because we would have to identify that two records are "the same" based on either just their fields or the label used in the parent element...). If you are interested in contributing to F# Data, please open an issue to discuss this!

In the meantime, I think a reasonable workaround is to get the underlying JsonValue and then wrap it back into the provided Weight type. This will work for both of the weight records, because they have the same fields:

type NN = JsonProvider<"""{ ... }""">

let processWeight (js:JsonValue) = 
  let w = NN.Weight(js)
  w.Data, w.Shape

let nn = NN.GetSample()
nn.Conv1.Weight.JsonValue |> processWeight
nn.Bn1.Weight.JsonValue |> processWeight

A bit fancier version would be to use static member constraints to access the JsonValue property in the processWeight function (so that you can just call nn.Conv1.Weight |> processWeight), but I did not want to make the sample too complicated.

Upvotes: 1

Related Questions