Is there a PowerShell equivalent for Python's doctest module?

Question

I've just come across the doctest module in Python, which helps you perform automated testing against example code that's embedded within Python doc-strings. This ultimately helps ensure consistency between the documentation for a Python module, and the module's actual behavior.

Is there an equivalent capability in PowerShell, so I can test examples in the .EXAMPLE sections of PowerShell's built-in help?

This is an example of what I would be trying to do:

function MyFunction ($x, $y) {
    <#
    .EXAMPLE

    > MyFunction -x 2 -y 2
    4
    #>
    return $x + $y
}

MyFunction -x 2 -y 2

Answer 1

You could do this, although I'm not aware of any all-in-one built-in way to do it.

Method 1 - Create a scriptblock and execute it

Help documentation is an object, so can be leveraged to index into examples and their code. Below is the simplest example I could think of which executes your example code.

I'm not sure if this is what doctest does - it seems a bit dangerous to me but it might be what you're after! It's the simplest solution and I think will give you the most accurate results.

Function Test-Example {
    param (
        $Module
    )
    # Get the examples
    $examples = Get-Help $Module -Examples

    # Loop over the code of each example
    foreach ($exampleCode in $examples.examples.example.code) {
        # create a scriptblock of your code
        $scriptBlock = [scriptblock]::Create($exampleCode)

        # execute the scriptblock
        $scriptBlock.Invoke()
    }
}

---

Method 2 - Parse the example/function and make manual assertions

I think a potentially better way to this would be to parse your example and parse the function to make sure it's valid. The downside is this can get quite complex, especially if you're writing complex functions.

Here's some code that checks the example has the correct function name, parameters and valid values. It could probably be refactored (first time dealing with [System.Management.Automation.Language.Parser]) and doesn't deal with advanced functions at all.

If you care about things like Mandatory, ParameterSetName, ValidatePattern etc this probably isn't a good solution as it will require a lot of extension.

Function Check-Example {
    param (
        $Function
    )

    # we'll use this to get the example command later
    # source: https://vexx32.github.io/2018/12/20/Searching-PowerShell-Abstract-Syntax-Tree/
    $commandAstPredicate = {
        param([System.Management.Automation.Language.Ast]$AstObject)
        return ($AstObject -is [System.Management.Automation.Language.CommandAst])
    }

    # Get the examples
    $examples = Get-Help $Function -Examples

    # Parse the function
    $parsedFunction = [System.Management.Automation.Language.Parser]::ParseInput((Get-Content Function:$Function), [ref]$null, [ref]$null)

    # Loop over the code of each example
    foreach ($exampleCode in $examples.examples.example.code) {

        # parse the example code
        $parsedExample = [System.Management.Automation.Language.Parser]::ParseInput($exampleCode, [ref]$null, [ref]$null)

        # get the command, which gives us useful properties we can use
        $parsedExampleCommand = $parsedExample.Find($commandAstPredicate,$true).CommandElements

        # check the command name is correct
        "Function is correctly named: $($parsedExampleCommand[0].Value -eq $Function)"

        # loop over the command elements. skip the first one, which we assume is the function name
        foreach ($element in ($parsedExampleCommand | select -Skip 1)) {
            "" # new line

            # check parameter in example exists in function definition
            if ($element.ParameterName) {
                "Checking parameter $($element.ParameterName)"
                $parameterDefinition = $parsedFunction.ParamBlock.Parameters | where {$_.Name.VariablePath.Userpath -eq $element.ParameterName}

                if ($parameterDefinition) {
                    "Parameter $($element.ParameterName) exists"
                    # store the parameter name so we can use it to check the value, which we should find in the next loop
                    # this falls apart for switches, which have no value so they'll need some additional logic
                    $previousParameterName = $element.ParameterName
                }
            }
            # check the value has the same type as defined in the function, or can at least be cast to it.
            elseif ($element.Value) {
                "Checking value $($element.Value) of parameter $previousParameterName"
                $parameterDefinition = $parsedFunction.ParamBlock.Parameters | where {$_.Name.VariablePath.Userpath -eq $previousParameterName}
                "Parameter $previousParameterName has the same type: $($element.StaticType.Name -eq $parameterDefinition.StaticType.Name)"
                "Parameter $previousParameterName can be cast to correct type: $(-not [string]::IsNullOrEmpty($element.Value -as $parameterDefinition.StaticType))"
            }
            else {
                "Unexpected command element:"
                $element
            }
        }
    }
}

---

Method 3 - Use Pester (maybe out of scope)

I think this one is a bit off topic, but worth mentioning. Pester is the test framework for PowerShell and has features that could be helpful here. You could have a generic test that takes a script/function as argument and runs tests against the parsed examples/functions.

This is could involve executing the script like in method 1 or checking the parameters like in method 2. Pester has a HaveParameter assertion that allows you to check certain things about your function.

HaveParameter documenation, copied from link above:

Get-Command "Invoke-WebRequest" | Should -HaveParameter Uri -Mandatory
function f ([String] $Value = 8) { }
Get-Command f | Should -HaveParameter Value -Type String
Get-Command f | Should -Not -HaveParameter Name
Get-Command f | Should -HaveParameter Value -DefaultValue 8
Get-Command f | Should -HaveParameter Value -Not -Mandatory