[System.Convert]::FromBase64String causes memory leak with large strings

Prerequisites

Write a descriptive title.
Make sure you are able to repro it on the latest released version
Search the existing issues.
Refer to the FAQ.
Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce

This was tested on PowerShell 7.4.2

Download the .NET 8.0 installer for Windows x64 to use as the test file, the direct link to this is here: https://dotnet.microsoft.com/en-us/download/dotnet/thank-you/sdk-8.0.204-windows-x64-installer
Save this file into a folder on your computer somewhere
Create a PowerShell script with the following contents in the same folder as the .NET 8.0 installer:

Get-Date
Write-Host "Creating Path to dotnet.exe test file"
$file = Join-Path -Path $PSScriptRoot -ChildPath "dotnet.exe"
Write-Host "Reading all file bytes into memory"
$bytes = [System.IO.File]::ReadAllBytes( $file )
Write-Host "Converting file bytes to base64 string"
$base64 = [System.Convert]::ToBase64String( $bytes )
Write-Host "Converting base64 string back to file bytes"
$bytes = [System.Convert]::FromBase64String( $base64 )
Write-Host "Test complete"
Get-Date

NOTE: That if you test this with a newer version of the .NET 8.0 installer, you may have to modify the test script to pick the correct file for the test since the filename is hard coded on line 3.

Open a PowerShell window in the folder with the script and test file
Run the PowerShell script
Open Task Manager and observe the memory usage of PowerShell
Note the time required to complete the conversion from Base64 and the usage of upwards of 3.5 GB of RAM to do so

The .NET 8.0 installer for Windows x64 is approximately 222 MB in size. Reading into memory and converting to base64 then converting back should require about 790 MB of RAM with all variables remaining in scope during the process and no garbage collection happening or object disposal happening. The observed behavior appears to be memory-leak related as the amount of memory used once the conversion eventually completes is about 3.4 GB of RAM. These data points can be see in the attached screen shots.

Expected behavior

When you run the same script in PowerShell 7 and Windows PowerShell 5.1, you see two very different behaviors:

In PowerShell 7.4.2, the time to complete is 82 seconds and memory used is 3.4 GB
In PowerShell 5.1, the time to complete is 7 seconds and memory used is 1.0 GB

This suggests there is an error in the PowerShell 7.4.2 / .NET 8.0 implementation.

Actual behavior

In PowerShell 7.4.2, the time to complete is 82 seconds and memory used is 3.4 GB.

Error details

No response

Environment data

Name                           Value
----                           -----
PSVersion                      7.4.2
PSEdition                      Core
GitCommitId                    7.4.2
OS                             Microsoft Windows 10.0.22631
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Visuals

Testing in PowerShell 7.4.2

Testing in PowerShell 5.1

PowerShell 7.4.2 Memory Usage

PowerShell 5.1 Memory Usage

I cannot replicate the slowness you see as 7.4.2 takes less than 5 seconds for me and is in fact faster than WinPS but I do notice the large memory usage. My guess is that it is not only are now storing the 2 byte arrays (raw file data and the decoded base64 string) but also the base64 string is all allocated on the heap as part of the operation. Potentially WinPS/.NET Framework is more aggressive in reusing the array values but as per the above the CLR could be allocating the memory and just never freeing it so it can more efficient reuse the memory in the future.

Putting aside the above comment you can more efficiently base64 encode bytes by streaming it rather than reading all the input bytes into memory.

Function ConvertTo-Base64String {
    [OutputType([string])]
    [CmdletBinding()]
    param (
        [Parameter(Mandatory)]
        [string]$Path
    )

    $fs = $cryptoStream = $sr = $null
    try {
        $fs = [System.IO.File]::OpenRead($Path)
        $cryptoStream = [System.Security.CryptoGraphy.CryptoStream]::new(
            $fs,
            [System.Security.Cryptography.ToBase64Transform]::new(),
            [System.Security.Cryptography.CryptoStreamMode]::Read)
        $sr = [System.IO.StreamReader]::new($cryptoStream, [System.Text.Encoding]::ASCII)
        $sr.ReadToEnd()
    }
    finally {
        ${sr}?.Dispose()
        ${cryptoStream}?.Dispose()
        ${fs}?.Dispose()
    }
}

This will stream the raw bytes from the source file stream and produce the final output string. If you are storing this string into a file then you could optimize it further by streaming the output base64 CryptoStream to a file avoiding having to store all the data in PowerShell.

If you do need to store the base64 string as an object in PowerShell keep in mind this means you not only have to store the inflated size that base64 uses (($length / 3) * 4) but each char of the string takes two bytes so at a minimum you are looking at around 600MB for the dotnet installer. Using the above function I see the memory usage sits around 1.2GB which is less than WinPS (about 1.4GB). While this is still more than the ~600MB there could be other factors in place here like the CLR allocating more memory than strictly needs in anticipation of future needs or some other reason.

Thank you both for your responses and suggestions for optimization. I did reply to the Issue cross-posted in the dotnet/runtime project which you can see here dotnet/runtime#101061 (comment)

I ran [GC]::Collect() after running the test script in both WinPS and PS7 and noticed that in PS7 almost none of the used memory was released back to the operating system, which seemed suspicious as in WinPS almost all memory is released after the test.

UPDATE: Confirmed below that the FromBase64String delay and excessive memory usage only occurs through PowerShell 7, but not through a .NET console app. This definitely appears to be a .NET run-time issue and not a PowerShell issue, however I wanted to post it here as well as over there for visibility in case others encounter this oddity while building PowerShell modules (which is how I stumbled upon it).

I am aware that storing a 200 MB file in memory as base64 text is wildly inefficient, and is not the intention of how my PowerPass module should be used (which is how I discovered this in the first place), but since I stumbled upon this unexpected behavior I thought it prudent to at least report it.

But again, I appreciate all of the comments and feedback here, especially the suggestions for optimization techniques.

In PowerShell 7.4.2, the time to complete is 82 seconds and memory used is 3.4 GB.

Yes, the same memory usage, but time to complete is about 1.2 seconds

I retested the following updated script on my desktop PC. A Ryzen 5800X with 128 GB of RAM and PCIe Gen4 NVMe storage. The test ran much faster as expected, but the memory usage still remains high.

Even after invoking [GC]::Collect() around 3.2 GB of RAM still remains utilized by the pwsh process.

Reading the 222 MB file into memory takes 0.06 seconds and converting it to base64 takes 0.22 seconds and uses 845 MB of RAM across both operations as expected. The last operation [System.Convert]::FromBase64String uses 2.6 GB of RAM alone and takes 17 seconds.

I'll cross-post this in the dotnet/runtime issue. Thank you all for the feedback.

Updated test script:

$name = "random.bin"

$start = Get-Date

Write-Host "Creating Path to $name test file: " -NoNewline
$now = Get-Date
$file = Join-Path -Path $PSScriptRoot -ChildPath $name
Write-Host " $(((Get-Date) - $now).TotalMilliseconds) ms"

Write-Host "Reading all file bytes into memory: " -NoNewline
$now = Get-Date
$bytes = [System.IO.File]::ReadAllBytes( $file )
Write-Host " $(((Get-Date) - $now).TotalMilliseconds) ms"

Write-Host "Converting file bytes to base64 string: " -NoNewline
$now = Get-Date
$base64 = [System.Convert]::ToBase64String( $bytes )
Write-Host " $(((Get-Date) - $now).TotalMilliseconds) ms"

Write-Host "Converting base64 string back to file bytes: " -NoNewline
$now = Get-Date
$bytes = [System.Convert]::FromBase64String( $base64 )
Write-Host " $(((Get-Date) - $now).TotalMilliseconds) ms"

Write-Host "Test complete"

Write-Host "Total duration: $(((Get-Date) - $start).TotalMilliseconds) ms"

I just tested this same implementation using a C# console application for the dotnet/runtime team and the issues does NOT occur when running in a console application against .NET 8 on the latest SDK on Windows 11 Professional. My test results and C# code are here: dotnet/runtime#101061 (comment)

It seems that this is actually a memory leak in the PowerShell runtime for some reason. The dotnet/runtime crew was asking if the [System.Convert]::FromBase64String function was being replaced with something else by PowerShell, but since I haven't tweaked my PowerShell installation, I can't imagine what could be doing that. Plus, this issue happens on multiple PCs, my desktop and my laptop, and only within PowerShell 7.4.2.

I'm assuming PowerShell 7.4.2 is using .NET 8.0 under the hood. Does it ship with its own .NET run-time or does it rely on the run-time installed on the system?

For reference, the random.bin test file I am using is exactly 233,420,544 bytes in length. I generated it with this script:

param(
	[int]
	$Size
)
$blockSize = 256
$rand = [System.Random]::new()
$total = 0
[byte[]]$data = [System.Array]::CreateInstance( [byte[]], $blockSize )
$path = Join-Path -Path $PSScriptRoot -ChildPath "random.bin"
if( Test-Path $path ) {
	Remove-Item -Path $path -Force
}
$file = [System.IO.File]::OpenWrite( $path )
while( $total -lt $Size ) {
	$rand.NextBytes( $data )
	$file.Write( $data, 0, $data.Length )
	$total += $blockSize
}
$file.Flush()
$file.Close()

I'm assuming PowerShell 7.4.2 is using .NET 8.0 under the hood. Does it ship with its own .NET run-time or does it rely on the run-time installed on the system?

With PowerShell 7.4.2 installed from Microsoft Store, Get-ChildItem $PSHOME shows files like coreclr.dll; I think that means PowerShell has its own copy of the .NET Runtime.

[System.Runtime.InteropServices.RuntimeInformation]::FrameworkDescription is ".NET 8.0.4".

A couple more data points regarding the unexpected slowdown:

I see the slowdown only on Windows (neither on macOS nor on Linux), both in in 7.4.2 (.NET 8.0.4) and v7.5.0-preview.2 (.NET 9.0.0-preview.1.24080.9).
On Windows, I see the slowdown in 7.3.10 (.NET 7.0.14) too, albeit in less severe form (about twice as fast as 7.4.2 / v7.5.0-preview.2, but still way too slow); WinPS is fine.

In src/System.Management.Automation/engine/runtime/CompiledScriptBlock.cs, there is class SuspiciousContentChecker, which attempts to detect "suspicious strings" such as FromBase64String. That then apparently causes PowerShell to log some ETW event. I wonder if Windows Defender monitors those events and then spends time investigating the process.

I would like to remind that earlier we observed slow pwsh operations with files due to antivirus.

I'm assuming PowerShell 7.4.2 is using .NET 8.0 under the hood. Does it ship with its own .NET run-time or does it rely on the run-time installed on the system?

With PowerShell 7.4.2 installed from Microsoft Store, Get-ChildItem $PSHOME shows files like coreclr.dll; I think that means PowerShell has its own copy of the .NET Runtime.

[System.Runtime.InteropServices.RuntimeInformation]::FrameworkDescription is ".NET 8.0.4".

I am also seeing .NET 8.0.4 for the FrameworkDescription on my PowerShell 7 install which I setup from the MSI downloaded from Github. Assembly file versions are 8.0.424.16909.

I would like to remind that earlier we observed slow pwsh operations with files due to antivirus.

The slow operation is the call to [System.Convert]::FromBase64String which occurs after the file is loaded from disk into memory. On my desktop, this operation takes 17 seconds. The file load operation from disk takes 0.06 seconds.

Also, this only happens when doing this in PowerShell 7.4.2. Running this same test in a C# console application takes under 1 second.

In src/System.Management.Automation/engine/runtime/CompiledScriptBlock.cs, there is class SuspiciousContentChecker, which attempts to detect "suspicious strings" such as FromBase64String. That then apparently causes PowerShell to log some ETW event. I wonder if Windows Defender monitors those events and then spends time investigating the process.

I added a Program Setting for pwsh.exe into Exploit Protection under App & Browser Control and disabled all protections. Rerunning the test resulted in the same duration. 17 seconds and 3.4 GB of RAM usage. Is there another place I can check in Windows Security to prevent Defender from watching pwsh.exe?

Per your own comment, reflection can be used to bypass the problem.

I'm half expecting you to make PowerShell recognise MethodInfo.Invoke calls and log each element of object?[]? parameters to AMSI as if the method had been called directly.

Kalle Olavi Niemitalo (@KalleOlaviNiemitalo)

Kalle Olavi Niemitalo (@KalleOlaviNiemitalo), fair point: Both of the aforementioned workarounds amount to bypassing the intended AMSI calls - I merely summarized them, speaking as someone who's neither a security expert nor speaking in any official capacity.

Yes, prior to PS 7.4 raw byte handling in pipelines wasn't supported, but in 7.4+ it now is, between external (native) programs, so the following works as intended from PowerShell (also on Windows, if you install openssl.exe there); note the use of -in to specify the input file:

# OK in PS 7.4+
openssl base64 -in file.in | openssl base64 -d > file.out

I haven't looked into the implementation, but I assume (hope) that on Unix-like platforms the usual system-level data buffering applies, which is 64KB these days.

While < isn't available in PowerShell to byte-stream data to an external program, in v7.4+ you can feed [byte] or - much more efficiently - [byte[]] data output from PowerShell commands to external programs:

The - slow - solution is therefore (byte-by-byte processing on the PowerShell side):

Get-Content file.in -AsByteStream | openssl base64 | openssl base64 -d > file.out

The - much faster - solution, which, however, reads the input file in full, due to -Raw:

Get-Content file.in -Raw -AsByteStream |  openssl base64 | openssl base64 -d > file.out

The - more memory-efficient - solution that emulates Unix pipeline buffering is:

Get-Content file.in -ReadCount 64kb -AsByteStream | 
  % { , [byte[]] $_ } |
  openssl base64 | openssl base64 -d > file.out

Note the - unfortunate in terms of both verbosity and performance - need for an intermediate % (ForEach-Object) call that strongly types the [object[]]-typed arrays that -ReadCount produces as [byte[]], as that is the prerequisite for sending raw byte data to an external program.

Arguably, Get-Content's -ReadCount parameter should instead output:

[string[]] arrays by default
[byte[]] arrays in combination with -AsByteStream

This would obviate the need for the inefficient and awkward ForEach-Object helper call.

because it has to fill the entire 64KB first before it passes onto the pipeline

Yes, it's an imperfect emulation of the native Unix pipeline, but with file input (where there's no "dribbling"), it works well.

That said, it's rare for Unix-heritage utilities to accept input via stdin (the pipeline) only and not also via file-path operands; thus, with a file as the data source, passing the file's path as an argument to an external program is the simpler and better solution (such as in the openssl case, using the - syntactically unusual - -in parameter).

It would certainly be better if Get-Content always wrote AsByteStream as a byte array but I think it is too late to change that.

Hopefully not: Let's see what becomes of the feature request you've since created:

Get-Content with -AsByteStream -ReadCount X -Raw should return byte arrays #21510

rhubarb-geek-nz

Split-Content -LiteralPath $file -AsByteStream | ConvertTo-Base64 | ConvertFrom-Base64 | Set-Content -LiteralPath $copy -AsByteStream

Thank you for showing me this technique. So what I understand is happening with the PowerShell pipeline is Split-Content instantiates a small memory buffer, fills it with bytes from the file, then the pipeline hands it to ConvertTo-Base64 and so forth down the pipeline such that the same small memory buffer is reused for each read operation on the $file and pass through the pipeline with each read iteration.

rhubarb-geek-nz do you have your cmdlet source on Github?

Yes, but a new byte array is written to the output pipeline. So the same total amount of memory is allocated, just not all at the same time.

Looking at lines 170, 172, and 174 in https://github.com/rhubarb-geek-nz/SplitContent/blob/main/SplitContent/SplitContent.cs you copy the read buffer record into a new byte[] to pass into the WriteObject method handing it the pipeline. What is the consequence of passing the same byte[] instance for each block of data read from the file?

For example, let's say you:

$b64 = Split-Content -Path 'C:\temp\dotnet.exe' -AsByteStream | ConvertTo-Base64

In your converter, you pass the incoming byte[] into the Stream.Write method to send the bytes into the base64 stream. It would seem that with your Split-Content cmdlet you could optimize it by reusing the same in-memory buffer for WriteObject. Are there situations where this is unsafe and the Buffer.BlockCopy becomes necessary?

GitHub
SplitContent/SplitContent/SplitContent.cs at main · rhubarb-geek-nz/SplitContent
Split-Content tool for PowerShell. Contribute to rhubarb-geek-nz/SplitContent development by creating an account on GitHub.

The pipelines between cmdlets act like a queue and the cmdlets can work at different speeds. So the writer may add five records to the pipeline before the next stage starts consuming them.

Interesting. And I understand why this is done, you can take advantage of parallel processing on multi-core CPUs to improve the performance of a pipeline of commands rather than executing them all in sequence.

Does the PSCmdlet class expose access to the pipeline status, for example, can it check to see if n amount of objects it has placed into the pipeline have been consumed yet or not? This would help with throttling memory usage on large operations, like streaming a file into the pipeline. Or, does the PSCustomObject class or PSObject class have the capability to notify callers that is has been consumed?

I ran a memory profiler on the scenario

$fileBytes = [io.file]::ReadAllBytes("C:\Users\staff\Downloads\dotnet-sdk-8.0.204-win-x64.exe");
$base64String =[System.Convert]::ToBase64String($fileBytes);
# $base64String  is 620 Mb
# But as the image shows, the call to FromBase64String causes AMSI logging of method invocation 
# which allocates 2.7 GB of memory to do the logging.
$roundTripBytes =[System.Convert]::FromBase64String($base64String)

The code responsible for getting the param values for the logging looks like this.

 internal static void LogMemberInvocation(string targetName, string name, object[] args)
        {
                var contentName = "PowerShellMemberInvocation";
                var argsBuilder = new Text.StringBuilder();

                for (int i = 0; i < args.Length; i++)
                {
                    string value = ArgumentToString(args[i]);

                    if (i > 0) { argsBuilder.Append(", "); }
                    argsBuilder.Append($"<{value}>");
                }
                string content = $"<{targetName}>.{name}({argsBuilder})";
            var success = AmsiUtils.ReportContent(contentName, content);
    }


     private static string ArgumentToString(object arg)
    {
      object obj = PSObject.Base(arg);
      if (obj == null)
        return "null";
      if (obj is string str)
        return str; // perhaps limit to 256 bytes or so
      Type type = obj.GetType();
      return type.IsEnum || type.IsPrimitive || type == typeof (Guid) || type == typeof (Uri) || type == typeof (Version) || type == typeof (SemanticVersion) || type == typeof (BigInteger) || type == typeof (Decimal) ? obj.ToString() : type.FullName;
    }

Maybe it would make sense to limit the size of str to some sensible max-length?

I tagged WG-Security to have them look at this to see if it's viable.

Paul Higinbotham (@PaulHigin)

Paul Higinbotham (@PaulHigin) Can you advise here?

This is starting to cause problems on CI farms where lots of JSON is being generated. In some cases, there have been several orders of magnitude slowdowns, ranging from seconds locally (on folders excluded from AV) to 20 minutes on the farms.

Staffan Gustafsson (@powercode)

Staffan Gustafsson (@powercode), note that Paul Higinbotham (@PaulHigin) is retired now.
The current members of the Security WG are listed here.

Travis Plunk (@TravisEz13)

Travis Plunk (@TravisEz13), Sydney Smith (@SydneyhSmith) - Can you have a look? This is a performance issue with potential security implications.

Perhaps #24853 resolve the issue.

Are there plans to add AMSI logging to method invocation via reflection? I'm able to claw back some of the catastrophic performance loss that AMSI incurred using this trick, but I'm guessing it's only a matter of time before this gets caught up in AMSI's web too.

Navigation Menu

Repository navigation

[System.Convert]::FromBase64String causes memory leak with large strings #21473

Prerequisites

Steps to reproduce

Expected behavior

Actual behavior

Error details

Environment data

Visuals

Testing in PowerShell 7.4.2

Testing in PowerShell 5.1

PowerShell 7.4.2 Memory Usage

PowerShell 5.1 Memory Usage

23 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[System.Convert]::FromBase64String causes memory leak with large strings #21473

Description

Prerequisites

Steps to reproduce

Expected behavior

Actual behavior

Error details

Environment data

Visuals

Testing in PowerShell 7.4.2

Testing in PowerShell 5.1

PowerShell 7.4.2 Memory Usage

PowerShell 5.1 Memory Usage

Activity

jborean93 commented on Apr 15, 2024

chopinrlz commented on Apr 16, 2024

237dmitry commented on Apr 16, 2024

chopinrlz commented on Apr 16, 2024

chopinrlz commented on Apr 16, 2024

chopinrlz commented on Apr 16, 2024

KalleOlaviNiemitalo commented on Apr 16, 2024

mklement0 commented on Apr 16, 2024

KalleOlaviNiemitalo commented on Apr 16, 2024

iSazonov commented on Apr 16, 2024

chopinrlz commented on Apr 16, 2024

chopinrlz commented on Apr 16, 2024

chopinrlz commented on Apr 16, 2024

23 remaining items

KalleOlaviNiemitalo commented on Apr 19, 2024

mklement0 commented on Apr 20, 2024

mklement0 commented on Apr 20, 2024

mklement0 commented on Apr 21, 2024

chopinrlz commented on May 2, 2024

chopinrlz commented on May 5, 2024

chopinrlz commented on May 12, 2024

powercode commented on Sep 16, 2024

powercode commented on Nov 15, 2024

mklement0 commented on Nov 15, 2024

powercode commented on Feb 13, 2025

iSazonov commented on Feb 14, 2025

User1785604260 commented on Feb 14, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions