Supporting lexical-scope ScriptBlock parameters (e.g. Where-Object)

Question

Consider the following arbitrary function and test cases:

Function Foo-MyBar {
    Param(
        [Parameter(Mandatory=$false)]
        [ScriptBlock] $Filter
    )

    if (!$Filter) { 
        $Filter = { $true } 
    }

    #$Filter = $Filter.GetNewClosure()

    Get-ChildItem "$env:SYSTEMROOT" | Where-Object $Filter   
}

##################################

$private:pattern = 'T*'

Get-Help Foo-MyBar -Detailed

Write-Host "`n`nUnfiltered..."
Foo-MyBar

Write-Host "`n`nTest 1:. Piped through Where-Object..."
Foo-MyBar | Where-Object { $_.Name -ilike $private:pattern  }

Write-Host "`n`nTest 2:. Supplied a naiive -Filter parameter"
Foo-MyBar -Filter { $_.Name -ilike $private:pattern }

In Test 1, we pipe the results of Foo-MyBar through a Where-Object filter, which compares the objects returned to a pattern contained in a private-scoped variable $private:pattern. In this case, this correctly returns all the files/folders in C:\ which start with the letter T.

In Test 2, we pass the same filtering script directly as a parameter to Foo-MyBar. However, by the time Foo-MyBar gets to running the filter, $private:pattern is not in scope, and so this returns no items.

I understand why this is the case -- because the ScriptBlock passed to Foo-MyBar is not a closure, so does not close over the $private:pattern variable and that variable is lost.

I note from comments that I previously had a flawed third test, which tried to pass {...}.GetNewClosure(), but this does not close over private-scoped variables -- thanks @PetSerAl for helping me clarify that.

The question is, how does Where-Object capture the value of $private:pattern in Test 1, and how do we achieve the same behaviour in our own functions/cmdlets?

(Preferably without requiring the caller to have to know about closures, or know to pass their filter script as a closure.)

I note that, if I uncomment the $Filter = $Filter.GetNewClosure() line inside Foo-MyBar, then it never returns any results, because $private:pattern is lost.

(As I said at the top, the function and parameter are arbitrary here, as a shortest-form reproduction of my real problem!)

Answer 1

The example given does not work because calling a function will enter a new scope by default. Where-Object will still invoke the filter script without entering one, but the scope of the function does not have the private variable.

There's three ways around this.

Put the function in a different module than the caller

Every module has a SessionState which has its own stack of SessionStateScopes. Every ScriptBlock is tied to the SessionState is was parsed in.

If you call a function defined in a module, a new scope is created within that module's SessionState, but not within the top level SessionState. Therefore when Where-Object invokes the filter script without entering a new scope, it does so on the current scope for the SessionState to which that ScriptBlock is tied.

This is a bit fragile, because if you want to call that function from your module, well you can't. It'll have the same issue.

Call the function with the dot source operator

You most likely already know the dot-source operator (.) for invoking script files without creating a new scope. That also works with command names and ScriptBlock objects.

. { 'same scope' }
. Foo-MyBar

Note, however, that this will invoke the function within the current scope of the SessionState that the function is from, so you cannot rely on . to always execute in the caller's current scope. Therefore, if you invoke functions associated with a different SessionState with the dot-source operator - such as functions defined in a (different) module - it may have unintended effects. Variables created will persist to future function invocations and any helper functions defined within the function itself will also persist.

Write a Cmdlet

Compiled commands (cmdlets) do not create a new scope when invoked. You can also use similar API's to what Where-Object use (though not the exact same ones)

Here's a rough implementation of how you could implement Where-Object using public API's

using System.Management.Automation;

namespace MyModule
{
    [Cmdlet(VerbsLifecycle.Invoke, "FooMyBar")]
    public class InvokeFooMyBarCommand : PSCmdlet
    {
        [Parameter(ValueFromPipeline = true)]
        public PSObject InputObject { get; set; }

        [Parameter(Position = 0)]
        public ScriptBlock FilterScript { get; set; }

        protected override void ProcessRecord()
        {
            var filterResult = InvokeCommand.InvokeScript(
                useLocalScope: false,
                scriptBlock: FilterScript,
                input: null,
                args: new[] { InputObject });

            if (LanguagePrimitives.IsTrue(filterResult))
            {
                WriteObject(filterResult, enumerateCollection: true);
            }
        }
    }
}

Answer 2

how does Where-Object capture the value of $private:pattern in Test 1

As can be seen in the source code for Where-Object in PowerShell Core, PowerShell internally invokes the filter script without confining it to its own local scope (_script is the private backing field for the FilterScript parameter, notice the useLocalScope: false argument passed to DoInvokeReturnAsIs()):

protected override void ProcessRecord()
{
    if (_inputObject == AutomationNull.Value)
        return;

    if (_script != null)
    {
        object result = _script.DoInvokeReturnAsIs(
            useLocalScope: false, // <-- notice this named argument right here
            errorHandlingBehavior: ScriptBlock.ErrorHandlingBehavior.WriteToCurrentErrorPipe,
            dollarUnder: InputObject,
            input: new object[] { _inputObject },
            scriptThis: AutomationNull.Value,
            args: Utils.EmptyArray<object>());

        if (_toBoolSite.Target.Invoke(_toBoolSite, result))
        {
            WriteObject(InputObject);
        }
    }
    // ...
}

how do we achieve the same behaviour in our own functions/cmdlets?

We don't - DoInvokeReturnAsIs() (and similar scriptblock invocation facilities) are marked internal and can therefore only be invoked by types contained in the System.Management.Automation assembly