Why does powershell give different result in one-liner than two-liner when converting JSON?

Question

Overview

From a powershell 3 prompt,, I want to call a RESTful service, get some JSON, and pretty-print it. I discovered that if I convert the data to a powershell object, and then convert the powershell object back to json, I get back a nice pretty-printed string. However, if I combine the two conversions into a one-liner with a pipe I will get a different result.

TL;DR: this:

PS> $psobj = $orig | ConvertFrom-JSON
PS> $psobj | ConvertTo-JSON

... gives me different result than this:

PS> $orig | ConvertFrom-JSON | ConvertTo-JSON

Original data

[
  {
    "Type": "1",
    "Name": "QA"
  },
  {
    "Type": "2",
    "Name": "whatver"
  }
]

Doing the conversion in two steps

I'm going to remove the whitespace (so it fits on one line...), convert it to a powershell object, and then convert it back to JSON. This works well, and gives me back the correct data:

PS> $orig = '[{"Type": "1","Name": "QA"},{"Type": "2","Name": "DEV"}]'
PS> $psobj = $orig | ConvertFrom-JSON
PS> $psobj | ConvertTo-JSON
[
    {
        "Type":  "1",
        "Name":  "QA"
    },
    {
        "Type":  "2",
        "Name":  "DEV"
    }
]

Combining the two steps with a pipe

However, if I combine those last two statements into a one-liner, I get a different result:

PS> $orig | ConvertFrom-JSON | ConvertTo-JSON
{
    "value":  [
                  {
                      "Type":  "1",
                      "Name":  "QA"
                  },
                  {
                      "Type":  "2",
                      "Name":  "DEV"
                  }
              ],
    "Count":  2
}

Notice the addition of the keys "value" and "Count". Why is there a difference? I'm sure it has something to do with the desire to return JSON object rather than a JSON array, but I don't understand why the way I do the conversion affects the end result.

Answer 1

Note: The problem will continue to plague Windows PowerShell (whose latest and last version is v5.1), but PowerShell (Core) 7+ is not affected.

The existing answers provide an effective workaround - enclosing $orig | ConvertFrom-JSON in (...) - but do not explain the problem correctly; also, the workaround cannot be used in all situations.

---

As for why use of an intermediate variable did not exhibit the problem:

• The in-pipeline distinction between emitting an array's elements one by one vs. the array as a whole (as a single object) is nullified if you collect the output in a variable; e.g., $a = Write-Output 1, 2 is effectively equivalent to $a = Write-Output -NoEnumerate 1, 2: while the latter emits array 1, 2 as a single object, in both cases $a ends up containing a 2-element array.

• Then using array $a as pipeline input causes it to be enumerated, i.e. sent though the pipeline element by element.

• By contrast, direct use of Write-Output -NoEnumerate 1, 2 as pipeline input would send the array as a whole through the pipeline, which is what ConvertFrom-Json effectively does.

---

The problematic behavior is a combination of two factors:

• ConvertFrom-Json deviates from normal output behavior by sending arrays as single objects through the pipeline. That is, with a JSON string representing an array, ConvertFrom-Json sends the resulting array of objects as a single object through the pipeline.

  • You can verify ConvertFrom-Json's surprising behavior as follows:

      PS> '[ "one", "two" ]' | ConvertFrom-Json | Get-Member
    
      TypeName: System.Object[]  # !! should be: System.String
      ...
    

  • If ConvertFrom-Json passed its output through the pipeline one by one - as cmdlets normally do - Get-Member would instead return the (distinct) types of the items in the collection, which is [System.String] in this case.

    • Enclosing a command in (...) forces enumeration of its output, which is why ($orig | ConvertFrom-Json) | ConvertTo-Json is an effective workaround.

  • This behavior was eventually changed in PowerShell (Core) 7+, which now emits the elements of an array one by one, as expected, unless you use the -NoEnumerate switch.

• The System.Array type - the base type for all arrays - has a .Count property defined for it via PowerShell's ETS (Extended Type System - see Get-Help about_Types.ps1xml) in Windows PowerShell, which causes ConvertTo-Json to include that property in the JSON string it creates, with the array elements included in a sibling value property.

  • This happens only when ConvertTo-Json sees an array as a whole as an input object,^[1] as produced by ConvertFrom-Json in this case; e.g., , (1, 2) | ConvertTo-Json surfaces the problem (a nested array whose inner array is sent as a single object), but
    1, 2 | ConvertTo-Json does not (the array elements are sent individually).

  • This ETS-supplied .Count property was effectively obsoleted in PSv3, when arrays implicitly gained a .Count property due to PowerShell now surfacing explicitly implemented interface members as well, which surfaced the ICollection.Count property (additionally, all objects were given an implicit .Count property in an effort to unify the handling of scalars and collections).

  • Sensibly, this ETS property has therefore been removed in PowerShell (Core) 7+; see below for a workaround in Windows PowerShell v3+.

---

Workaround (not needed in PowerShell (Core) 7+)

^{Tip of the hat, as many times before, to PetSerAl.}

Note: This workaround is PSv3+ by definition, because the Convert*-Json cmdlets were only introduced in v3.

Given that the ETS-supplied .Count property is (a) the cause of the problem and (b) effectively obsolete in PSv3+, the solution is to simply remove it before calling ConvertTo-Json - it is sufficient to do this once in a session; it should not affect other commands:

 # Remove the redundant ETS-supplied .Count property
 Remove-TypeData -ErrorAction Ignore System.Array 
 

With that, the extraneous .Count and .value properties should have disappeared:

 PS> '[ "one", "two" ]' | ConvertFrom-Json | ConvertTo-Json
 [
   "one",
   "two"
 ]

The above workaround also fixes the problem for array-valued properties; e.g.:

PS> '' | Select-Object @{ n='prop'; e={ @( 1, 2 ) } } | ConvertTo-Json
{
    "prop":  [
                 1,
                 2
             ]
}

Without the workaround, the value of "prop" would include the extraneous .Count and .value properties as well.

---

^{[1] Strictly speaking, the problem surfaces only if an array instance is (invisibly) wrapped in a [psobject] instance, which, however, is invariably the case when a whole array is passed via the pipeline, as all objects passing into and out of the pipeline invariably receive a [psobject] wrapper. Passing a non-wrapped array as an argument therefore does not exhibit the symptom, e.g.: ConvertTo-Json @(1, 2); you can provoke the symptom with a [psobject] cast: ConvertTo-Json ([psobject] @(1, 2))}

Answer 2

I've encountered the same problem here. I was able to resolve it by using a ForEach-Object and a PSCustomObject. Hope it helps you.

'' | ForEach-Object { [PSCustomObject]@{ prop= @( 1, 2 ) }} | ConvertTo-Json

It gives this, which is the expected behavior and using cleaner approach:

{
  "prop": [
        1,
        2
  ]
}

Answer 3

First off, why is this happening?

PowerShell automatically wraps multiple objects into a collection called a PSMemberSet that has a Count property on it. It's basically how PowerShell manages arbitrary arrays of objects. What's happening is that the Count property is getting added to the resulting JSON, yielding the undesirable results that you're seeing.

We can prove what I just stated above by doing the following:

$Json = @"
[
    {
        "Type":  "1",
        "Name":  "QA"
    },
    {
        "Type":  "2",
        "Name":  "DEV"
    }
]
"@;

# Deserialize the JSON into an array of "PSCustomObject" objects
$Deserialized = ConvertFrom-Json -InputObject $Json;
# Examine the PSBase property of the PowerShell array
# Note the .NET object type name: System.Management.Automation.PSMemberSet
$Deserialized.psbase | Get-Member;

Here is the output from the above

   TypeName: System.Management.Automation.PSMemberSet

Name           MemberType            Definition                                                                                                                                                                      
----           ----------            ----------                                                                                                                                                                      
Add            Method                int IList.Add(System.Object value)                                                                                                                                              
Address        Method                System.Object&, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 Address(int )                                                                       
Clear          Method                void IList.Clear()                      
......
......
Count          Property              int Count {get;}  

You can work around this behavior by referencing the SyncRoot property of the PSMemberSet (which implements the ICollection .NET interface), and passing the value of that property to ConvertTo-Json.

Here is a complete, working example:

$Json = @"
[
    {
        "Type":  "1",
        "Name":  "QA"
    },
    {
        "Type":  "2",
        "Name":  "DEV"
    }
]
"@;

($Json | ConvertFrom-Json) | ConvertTo-Json;

The correct (expected) output will be displayed, similar to the following:

[
    {
        "Type":  "1",
        "Name":  "QA"
    },
    {
        "Type":  "2",
        "Name":  "DEV"
    }
]
[
    {
        "Type":  "1",
        "Name":  "QA"
    },
    {
        "Type":  "2",
        "Name":  "DEV"
    }
]

Answer 4

The solution is to wrap the first two operations with parenthesis:

PS C:\> ($orig | ConvertFrom-JSON) | ConvertTo-JSON
[
    {
        "Type":  "1",
        "Name":  "QA"
    },
    {
        "Type":  "2",
        "Name":  "DEV"
    }
]

The parenthesis allow you to grab the output of the first two operations all at once. Without them, powershell will attempt to parse any objects its gets separately. The collection of PSCustomObject resulting from $orig | ConvertFrom-JSON contains two PSCustomObjects for the 1/QA and 2/DEV pairs, so by piping the output of that collection powershell attempts to handle the key/value pairs one-at-a-time.

Using parenthesis is a shorter way of "grouping" that output and allows you to operate on it without making a variable.