Feature Request - Where-Object speed improvements

Summary of the new feature/enhancement

As it stands I've stopped using Where-Object in performance-related scenarios. I used it a lot when creating different scripts but then I found out it's actually 7-10x slower than simple foreach loop.

$UsersAll = Get-ADUser -Properties Manager, DisplayName, EmailAddress -Filter '*'
$UsersWithManagers = foreach ($User in $UsersAll) {
    $Manager = ($UsersAll | Where-Object { $User.Manager -eq $_.DistinguishedName })
    [PSCustomobject] @{
        SamAccountName = $User.SamAccountName
        Manager        = $User.Manager
        ManagerDisplay = $Manager.DisplayName
        ManagerEmail   = $Manager.EmailAddress
    }
}
$UsersWithManagers | Format-Table -AutoSize

This takes 7 minutes 54 seconds on a domain of size 4412 users. With foreach approach like this:

$UsersAll = Get-ADUser -Properties Manager, DisplayName, EmailAddress -Filter '*'
$UsersWithManagers = foreach ($User in $UsersAll) {
    $Manager = foreach ($_ in $UsersAll) {
        if ($User.Manager -eq $_.DistinguishedName) {
            $_
            break
        }
    }
    [PSCustomobject] @{
        SamAccountName = $User.SamAccountName
        Manager        = $User.Manager
        ManagerDisplay = $Manager.DisplayName
        ManagerEmail   = $Manager.EmailAddress
    }
}
$UsersWithManagers | Format-Table -AutoSize

It takes just 59 seconds. I ended up using hashtable for that particular problem (https://evotec.xyz/how-i-didnt-know-how-powerful-and-fast-hashtables-are/) but the main idea here is that Where-Object is just slow for larger loops.

It would be really cool to "fix" this in PowerShell 7 so Where-Object can once again be used as GO TO to solution for filtering stuff.

Related article https://learn-powershell.net/2015/02/20/a-look-at-filtering-performance-in-powershell/

I think there is 2 slow thinks:

pipeline itself
where-object internally

I guess we could transform Where-Object to Filter function and get up to 20x improvements.
The code references:

PowerShell/src/System.Management.Automation/engine/InternalCommands.cs

Lines 1632 to 1643 in 94b6b6f

    
           object result = _script.DoInvokeReturnAsIs( 
        
               useLocalScope: false, 
        
               errorHandlingBehavior: ScriptBlock.ErrorHandlingBehavior.WriteToCurrentErrorPipe, 
        
               dollarUnder: InputObject, 
        
               input: new object[] { _inputObject }, 
        
               scriptThis: AutomationNull.Value, 
        
               args: Array.Empty<object>()); 
        
           if (_toBoolSite.Target.Invoke(_toBoolSite, result)) 
        
           { 
        
               WriteObject(InputObject); 
        
           }

PowerShell/src/System.Management.Automation/engine/parser/ast.cs

Lines 1031 to 1036 in 94b6b6f

    
           if (isFilter) 
        
           { 
        
               this.ProcessBlock = new NamedBlockAst(statements.Extent, TokenKind.Process, statements, true); 
        
               SetParent(ProcessBlock); 
        
           } 
        
           else

I have also an example for this problem. If you use there Where-Object function of PowerShell it is slow. If you use the Linq function of the object you get twice the perfomance, with more or less the same code..

Point of clarity, that .Where() method has absolutely nothing to do with Linq; Linq extension methods aren't supported in PS and can't be used without accessing them directly as static methods (i.e., [System.Linq.Enumerable]::Where()). The method you're using is specific to PS's method binding, it's documented in the about_arrays help topic :)

But yeah, pipeline binding and processing has some overhead, for sure. 🙂

This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.

This issue has been marked as "No Activity" as there has been no activity for 6 months. It has been closed for housekeeping purposes.

Navigation Menu

Repository navigation

Feature Request - Where-Object speed improvements #9941

Summary of the new feature/enhancement

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	object result = _script.DoInvokeReturnAsIs(
	useLocalScope: false,
	errorHandlingBehavior: ScriptBlock.ErrorHandlingBehavior.WriteToCurrentErrorPipe,
	dollarUnder: InputObject,
	input: new object[] { _inputObject },
	scriptThis: AutomationNull.Value,
	args: Array.Empty<object>());

	if (_toBoolSite.Target.Invoke(_toBoolSite, result))
	{
	WriteObject(InputObject);
	}

	if (isFilter)
	{
	this.ProcessBlock = new NamedBlockAst(statements.Extent, TokenKind.Process, statements, true);
	SetParent(ProcessBlock);
	}
	else

Feature Request - Where-Object speed improvements #9941

Description

Summary of the new feature/enhancement

Activity

iSazonov commented on Jun 19, 2019

paule96 commented on Aug 14, 2020

vexx32 commented on Aug 15, 2020

microsoft-github-policy-service commented on Nov 16, 2023

microsoft-github-policy-service commented on Nov 16, 2023

microsoft-github-policy-service commented on Nov 16, 2023

microsoft-github-policy-service commented on Nov 23, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions