CODEWITHSUNDEEP
.netpowershellazure
MSC
MSC

Reputation: 2041

Powershell Runspaces Parallel script execution: Is Set-AzureRmSqlDatabaseTransparentDataEncryption commandlet threadsafe?

When running the powershell commandlet: Set-AzureRmSqlDatabaseTransparentDataEncryption

in parallel and separate runspaces, we are seeing transient/intermittent failures with stacktrace seen below. However, when we run this same commandlet serially, I've not seen this issue happen.

Possible Reasons I've come up with so far:

Stack Trace for one of the scripts running this command:

[Exception:System.NullReferenceException: Object reference not set to an instance of an object.\r\n
at System.Collections.Specialized.OrderedDictionary.OrderedDictionaryEnumerator.get_Value()\r\n
at Microsoft.Azure.Commands.Common.Authentication.Factories.ClientFactory.GetCustomHandlers()\r\n
at Microsoft.Azure.Commands.Common.Authentication.Factories.ClientFactory.CreateClient[TClient](AzureContext context, Endpoint endpoint)\r\n
at Microsoft.Azure.Commands.Sql.TransparentDataEncryption.Services.AzureSqlDatabaseTransparentDataEncryptionCommunicator.GetCurrentSqlClient(String clientRequestId)\r\n
at Microsoft.Azure.Commands.Sql.TransparentDataEncryption.Adapter.AzureSqlDatabaseTransparentDataEncryptionAdapter.GetTransparentDataEncryption(String resourceGroupName, String serverName, String databaseName)\r\n
at Microsoft.Azure.Commands.Sql.TransparentDataEncryption.Cmdlet.SetAzureSqlDatabaseTransparentDataEncryption.GetEntity()\r\n
at Microsoft.Azure.Commands.Sql.Common.AzureSqlCmdletBase'2.ExecuteCmdlet()\r\n
at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()]\

Stack Trace For a second thread running the same commandlet:

Message: "Object reference not set to an instance of an object.", Source: "System", StackTrace: " at System.Collections.Specialized.OrderedDictionary.IndexOfKey(Object key)\r\n at System.Collections.Specialized.OrderedDictionary.set_Item(Object key, Object value)\r\n at Microsoft.Azure.Commands.Common.Authentication.Factories.ClientFactory.AddHandler[T](T handler)\r\n at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.BeginProcessing()\r\n at System.Management.Automation.Cmdlet.DoBeginProcessing()\r\n at System.Management.Automation.CommandProcessorBase.DoBegin()"`

Suspicious: both commands blow up when enumerating/accessing System.Collections.Specialized.OrderedDictionary according to the stack-traces.

Significance: Are the two instances of the command accessing the SAME dictionary?

Upvotes: 3

Views: 355

Answers (2)

MSC
MSC

Reputation: 2041

Will's answer seems to support our suspicion that this (and other) commandlets are not thread-safe. In order to deal with this, we "lock" around calling such commandlets using a mutex as the coordination mechanism.

Using this thread as inspiration, built a simple implementation of a process locking/coordination mechanism (using Mutexes) in powershell:

function Wait-OnMutex
{
    param(
    [parameter(Mandatory = $true)][string] $MutexId
    )

    try
    {
        $MutexInstance = New-Object System.Threading.Mutex -ArgumentList 'false', $MutexId

        while (-not $MutexInstance.WaitOne(1000))
        {
            Start-Sleep -m 500;
        }

        return $MutexInstance
    } 
    catch [System.Threading.AbandonedMutexException] 
    {
        $MutexInstance = New-Object System.Threading.Mutex -ArgumentList 'false', $MutexId
        return Wait-OnMutex -MutexId $MutexId
    }

}

##
## example script calling unsafe commandlet
##

$MutexInstance = Wait-OnMutex -MutexId 'SomeMutexId12345'

## this is where you do work inside the "lock"
## call the commandlet needing to be single-threaded
Set-AzureRmSqlDatabaseTransparentDataEncryption -arg1 value

$MutexInstance.ReleaseMutex()

Hope This helps somebody.

Upvotes: 3

Will Shao - MSFT
Will Shao - MSFT

Reputation: 1207

@MSC,

Is it known that this commandlet is not thread-safe? (on purpose or bug?)

Reason: As far as I known, It depended on the Class Library whether is thread safe. From the source code(http://referencesource.microsoft.com/#System/compmod/system/collections/specialized/ordereddictionary.cs,d02ab0d292f01b57 ), we can see the function (IndexOfKey(key)). It is more related to objectsArray. We can see this object code as following:

   private ArrayList objectsArray {
            get {
                if (_objectsArray == null) {
                    _objectsArray = new ArrayList(_initialCapacity);
                }
                return _objectsArray;
            }
        }

It appeared that if we used the parallel threads to call this function, it will throw the exception. It wasn't locked.

Howto:

  1. Use the lock mechanism to locked the 'objectArray' in powershell

  2. Use the single thread to call this function.

Hope that helps.

Upvotes: 2

Related Questions