Repository navigation
Is there a way to use RunspacePools such that module loading uses more than one processor? #7035
Description
I'm experimenting with running PowerShell unit tests in parallel. It seems like one open runspace per thread is needed to achieve speedup. I'm seeing good speedup after the runspaces are open (see #6965(comment). I haven't, however, found a reliable way to parallelize module importing that occurs when opening the runspaces. (Runspaces.OpenAsync() looks promising, but it seems to suffer from #7034.)
Calling PowerShell.BeginInvoke() on a several of PowerShell instances sharing a runspace pool seems to block while module loading completes. The result is that despite that there might be many processors available and one runspace for each, only one processor is utilized to perform module importing for all of the runspaces.
Steps to reproduce
$processorCount = [System.Environment]::ProcessorCount
Write-Host "Processor Count: $processorCount"
$moduleContent = {
function fibonacci {
param([int]$n)
[bigint]$a=0
[bigint]$b=1
foreach ($x in 0..$n)
{
$a,$b = $b,($a+$b)
}
$b
}
0..0 | % { fibonacci 100000 }
}
$modulePath = "$([System.IO.Path]::GetTempPath())slowLoading.psm1"
$moduleContent | Set-Content $modulePath
$t_import = Measure-Command {
Import-Module $modulePath -Force
}
$initialSessionState = [initialsessionstate]::CreateDefault()
$initialSessionState.ImportPSModule($modulePath)
$rsp = [runspacefactory]::CreateRunspacePool($initialSessionState)
$rsp.SetMaxRunspaces($processorCount) | Out-Null
$rsp.Open()
$ps = 1..$processorCount | % { [powershell]::Create().AddScript({'done'}) }
$ps | % { $_.RunspacePool = $rsp }
$t_begin = Measure-Command {
$invocation = $ps.BeginInvoke()
}
$t_wait = Measure-Command {
while ( $invocation.IsCompleted -contains $false )
{
sleep 0.1
}
}
[pscustomobject]@{
'name' = 'Import-Module slowLoading.psm1'
'time(ms)' = [int]$t_import.TotalMilliseconds
}
[pscustomobject]@{
'name' = 'BeginInvoke()'
'time(ms)' = [int]$t_begin.TotalMilliseconds
}
[pscustomobject]@{
'name' = 'Wait'
'time(ms)' = [int]$t_wait.TotalMilliseconds
}
Expected behavior
Processor Count: 16
name time(ms)
---- --------
Import-Module slowLoading.psm1 5122
BeginInvoke() 100 (approximately)
Wait 6000 (approximately)
Actual behavior
Processor Count: 16
name time(ms)
---- --------
Import-Module slowLoading.psm1 5122
BeginInvoke() 19452
Wait 13
Here is the CPU utilization graph for the above test run:
Environment data
> $PSVersionTable
Name Value
---- -----
PSVersion 6.1.0-preview.691
PSEdition Core
GitCommitId v6.1.0-preview.691
OS Microsoft Windows 6.3.9600
Platform Win32NT
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0
Activity
alx9r commentedon Jun 15, 2018
It looks like the following might be relevant to answering this question:
slowLoading.psm1is mostly spent executing the scriptblock.)Below are the results of some tests related to these aspects.
.OpenAsync() with slowLoading.psm1
It looks like
.OpenAsync()parallelizes importing ofslowLoading.psm1nicely. parallelLoadModule.ps1 is a demonstration. Here is the output of two test runs on a 16-core processor:Opening 16x runspaces took 1.75x as long. The CPU utilization reflects that all processors are used in parallel:
This seems like good news for parallelized importing of script modules. However,
slowLoading.psm1isn't a very good stand-in for real-world modules..OpenAsync() w/ Pester
Pester is a real-world script module that takes some time to import.
Changing the call in parallelModuleLoad.ps1 to
Import-Module $modulePathand.ImportPSModule($modulePath)toImport-Module Pesterand.ImportPSModule('Pester'), respectively, causes Pester to be imported into each of the runspaces when.OpenAsync()is called. Here are the corresponding tests runs:Opening 16x runspaces that have Pester imported took 7.69x as long. The CPU utilization reflects some parallelization occuring:
.OpenAsync() w/ Pester while Running Pester
It's likely that code in a script module will be executed while another runspace is importing that module.
This test profiles importing Pester into a number of runspaces while the first runspace performs CPU-bound execution using Pester. The results from a typical test run are as follows:
The time taken to import Pester into the 16 runspaces took about the same time as when Pester code was not simultaneously being executed (7229ms vs. 7627ms). On the other hand, the single-threaded throughput of Pester code while other runspaces were importing Pester was 0.58x the throughput of that code when executed solo.
This seems to point to the following:
alx9r commentedon Jun 23, 2018
Per #7153, sharing an
AuthorizationManagerobject amongstRunspaces being opened causes contention. I wonder ifRunspacePooluses a singleAuthorizationManagerwhen opening runspaces.microsoft-github-policy-service commentedon Nov 16, 2023
This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.
microsoft-github-policy-service commentedon Nov 16, 2023
This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.
microsoft-github-policy-service commentedon Nov 16, 2023
This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.
microsoft-github-policy-service commentedon Nov 23, 2023
This issue has been marked as "No Activity" as there has been no activity for 6 months. It has been closed for housekeeping purposes.