drill down to lowest JSON element using PowerShell

Question

I need to extract some information from a JSON file using ps. The JSON has nested components arrays which can also contain a components array.

{
"name": "app",
"components": [
    {
        "component_name": "comp1",
        "component_packages": [
            "comp1_package1",
            "comp1_package2"
        ],
        "project_id": "1234",
        "file_path": "requirements_file",
        "ref": "%%VERSION%%",
        "components": [
            {
                "component_name": "comp1.1",
                "component_packages": [
                    "comp1.1_package1"
                ],
                "project_id": "2345",
                "file_path": "requirements_file",
                "ref": "%%VERSION%%",
                "components": [
                    {
                        "component_name": "comp1.1.1",
                        "component_packages": [
                            "comp1.1.1_package1"
                        ],
                        "project_id": "3456",
                        "file_path": "requirements_file",
                        "ref": "%%VERSION%%",
                        "components": []
                    }]
            },
            {
                "component_name": "comp1.2",
                "component_packages": [
                    "comp1.2_package1"
                ],
                "project_id": "4567",
                "file_path": "requirements_file",
                "ref": "%%VERSION%%",
                "components": []
            }
        ]
    },
    {
        "component_name": "comp2",
        "component_packages": [
            "comp2_package1",
            "comp2_package2"
        ],
        "project_id": "5678",
        "file_path": "requirements_file",
        "ref": "%%VERSION%%",
        "components": [
            {
                "component_name": "comp2.1",
                "component_packages": [
                    "comp2.1_package1"
                ],
                "project_id": "6789",
                "file_path": "requirements_file",
                "ref": "%%VERSION%%",
                "components": []
            }
        ]
    }
]   
}

for each component inside components I need to execute a script to gather more information but I struggle with iterating to all the elements one by one.

I started to convert the JSON to a psobject (Get-Content -Raw "$json_path" | ConvertFrom-Json)

I don't want to fix the depth of the JSON. So the script should be adaptable.

I tried using a while loop

$comp = $object.components
while ( $comp -ne "" ) { 
   $comp | ForEach-Object {
       # to something
   }
}

but like this it is not suitable, because even if I overwrite $comp, the script will forget some entries.

Answer 1

Providing a different way to do it as recursive function / scriptblock calls in PowerShell is really not recommended, with a Json too big you would for example find stack overflow issues. See both answers here.

Recommended approach would be to traverse your Json using a Stack or Queue (or their generic counterparts Stack<T> and Queue<T>).

$json = Get-Content path\to\json.json -Raw | ConvertFrom-Json
$stack = [System.Collections.Stack]::new()
$stack.Push($json)

while ($stack.Count) {
    $next = $stack.Pop()
    foreach ($item in $next) {
        [pscustomobject]@{
            ComponentName = $item.component_name
            Item          = $item
        }

        # do stuff with `$item` here

        if ($item.components) {
            $stack.Push($item.components)
        }
    }
}

Answer 2

Handling object-graphs (independent of the source, like Json, Yaml or PowerShell itself) might indeed become pretty complex if it concerns several levels and a mixture of arrays and hash tables. That's why I started to write the ObjectGraphTools module which might help you to explore the concerned object, iterate through the nodes, or simply pinpoint a specific node (and change its value).

Install-Module -Name ObjectGraphTools

$Object = $Json | ConvertFrom-Json # Where $Json holds the json string of your question

Examples

List all the nodes under the $Object.components node:

$Object | Get-Node Components | Get-ChildNode

Path          Name Depth Value
----          ---- ----- -----
components[0]    0     2 @{component_name=comp1; component_packages=System.Object…
components[1]    1     2 @{component_name=comp2; component_packages=System.Object…

Recursively search for all Components nodes (your specific question):

$Object | Get-Node | Get-ChildNode -Recurse Components

Path                                                 Name       Depth Value
----                                                 ----       ----- -----
components                                           components     1 {@{component_…
components[0].components                             components     3 {@{component_…
components[0].components[0].components               components     5 {@{component_…
components[0].components[0].components[0].components components     7 {}
components[0].components[1].components               components     5 {}
components[1].components                             components     3 {@{component_…
components[1].components[0].components               components     5 {}

But it probably doesn't end here.
To get a full list of leaf nodes under the$Object.components node:

$Object | Get-Node Components | Get-ChildNode -Leaf -Recurse

Path                                                            Name           Depth Value
----                                                            ----           ----- -----
...
components[0].components[0].ref                                 ref                5 %%VERSION%%
components[0].components[0].components[0].component_name        component_name     7 comp1.1.1
components[0].components[0].components[0].component_packages[0] 0                  8 comp1.1.1_package1
components[0].components[0].components[0].project_id            project_id         7 3456
components[0].components[0].components[0].file_path             file_path          7 requirements_file
components[0].components[0].components[0].ref                   ref                7 %%VERSION%%
components[0].components[1].component_name                      component_name     5 comp1.2
components[0].components[1].component_packages[0]               0                  6 comp1.2_package1
components[0].components[1].project_id                          project_id         5 4567
components[0].components[1].file_path                           file_path          5 requirements_file
components[0].components[1].ref                                 ref                5 %%VERSION%%
components[1].component_name                                    component_name
...

You might target any of the specific nodes by using a specific path property, e.g.:

$Object | Get-Node components[0].components[0].components[0].project_id

Path                                                 Name       Depth Value
----                                                 ----       ----- -----
components[0].components[0].components[0].project_id project_id     7 3456

Or using Member-Access enumeration:

$Object | Get-Node components.components.components.project_id

Path                                                 Name       Depth Value
----                                                 ----       ----- -----
components[0].components[0].components[0].project_id project_id     7 3456

Where you probably heading to is to get one of the components with a specific child node and change the value of another child node.
The syntax also supports wildcards and has some Extend Dot Notation (Xdn) operators which lets you freely target a deep node. E.g.:

$Object | Get-Node ~project_id=3456

Path                                                 Name       Depth Value
----                                                 ----       ----- -----
components[0].components[0].components[0].project_id project_id     7 3456

E.g. to change the ref value of the component that has a product_id of 3456, e.g.:

($Object | Get-Node ~project_id=3456..ref).Value = '1.2.3.4'

(Confirm the results with: $Object | ConvertTo-Json -Depth 9)

Explanation:

• ~project_id finds any descendant node named project_id
• =3456 filters the resulted node where the value equals 3456
• .. selects the parent
• ref selects the node named ref (the sibling of the project_id node)
• The Value property of the node is a reference to the related value in the object-graph and might therefore might be used modify the original object.

Answer 3

Just use a Function and call it recursively

(note: this might not be the most efficient way to do this)

$JsonObj = $Json | ConvertFrom-Json


function Drill-Json {
    [CmdletBinding()]
    param (
        [Parameter(
            Mandatory,
            Position = 0,
            ValueFromPipeline,
            ValueFromPipelineByPropertyName,
            ValueFromRemainingArguments
        )]
        [array]$JsonComponent 
    )
        
    
    
    $JsonComponent | ForEach-Object {
        <# 

            DO STUFF

        #>
        # $_.component_name
        if (($_.components.count)) {
            Drill-Json -JsonComponent $_.components
        }
    
    }
}

Drill-Json -JsonComponent $JsonObj.components