System.Byte error occurs when data size large

Prerequisites

Write a descriptive title.
Make sure you are able to repro it on the latest released version
Search the existing issues.
Refer to the FAQ.
Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce

I'm making an API request using Invoke-WebRequest and a large amount of data is returned.
$post.Content.length: 17689864
Below error message is returned.
Issue does not occur with earler versions of PS 7 or PS 5.1

$contentTypeValue = "application/json"
$authorization = "Bearer $accessToken"
$methodType = "GET"
$url = "API_ENDPOINT_RETURNING_a_LOT_OF_DATA"

$requestHeaders = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$requestHeaders.Add("Authorization", $authorization)
$requestHeaders.Add("Content-Type", $contentTypeValue)

$post = Invoke-WebRequest -Uri $url -Method $methodType -Headers $requestHeaders

Response body contains non-ascii characters.

If $($post.Content).length is too large than error returned on next line.

$utfPost = [System.Text.Encoding]::UTF8.GetString($([String]::new($post.Content)).ToCharArray())
$utfJsonOblect = $utfPost | ConvertFrom-Json
$return.utfJsonOblect = $utfJsonOblect

Expected behavior

$utfJsonOblect returns data without error.

Actual behavior

$utfJsonOblect returns below error message when data returned ($post.Content.length) is 17903138

Error details

Cannot convert argument "bytes", with value: "System.Char[]", for "GetString" to type "System.Byte[]": "Cannot convert value "" to type "System.Byte". Error: "Value was either too large or too small for an unsigned byte.

Environment data

PowerShell 7.3.8

Visuals

I don't think there's a PowerShell problem here:

[System.Text.Encoding]::UTF8.GetString() expects a [byte[]] array, yet you're passing it a [char[]] array.

Only [char] instances whose numeric value is between 0x0 and 0xFF (255) can be converted to [byte], any others cause the problem you're seeing.

A simple repro:

# '€' has a code point of 0x0x20ac (8364), which doesn't fit into a byte, which causes an error:
[System.Text.Encoding]::UTF8.GetString('h€'.ToCharArray())

In PowerShell Core, you shouldn't need to reinterpret a response as UTF-8-encoded; unless the web service misrepresents the encoding returned, UTF-8 is now the default.

In Windows PowerShell, the default encoding is ISO-8859-1; to force reinterpretation as UTF-8, use the following:

 [Text.Encoding]::UTF8.GetString($post.RawContentStream.ToArray())

Thank you for your response. A couple of things.
1 - I need to use the above $utfPost code because the Invoke-WebRequest response body returns ascii and non-ascii from an API endpoint.

2 - Here is snipet of what is returned from Invoke-WebRequest response body without using the $utfPost code.
{"schemas":["urn:ietf:params:scim:schemas:core:2.0:User"],"name":{"formatted":"KiÃªn fguyá»�n KiÃªn fguyá»�n","familyName" :"KiÃªn fguyá»�n","givenName":"KiÃªn fguyá»�n"},"id":"47500169ID","userName":"5977579"}

3 - The above $utfPost code works with this PS version. The issue is when the response body size reaches a certain size and then returns the error.
4 - The bug described above occurs in this PS 7 version but not PS 7 versions. I have no issues with earlier versions of PS 7.
5 - If you have a suggestion of a better way to handle this issue, please let me know.

As demonstrated in my previous comment, your use of [Text.Encoding]::UTF8.GetString() is conceptually flawed.

The comment also offers solutions.

Separately, as for whether there is a bug / change in PowerShell behavior:

Based on the information given so far, I don't see it. Consider the following code, based on your input:

# Predictably fails, due to the [char]-to-[byte] conversion problem.
$str = '{"schemas":["urn:ietf:params:scim:schemas:core:2.0:User"],"name":{"formatted":"KiÃªn fguyá»�n KiÃªn fguyá»�n","familyName" :"KiÃªn fguyá»�n","givenName":"KiÃªn fguyá»�n"},"id":"47500169ID","userName":"5977579"}'
[System.Text.Encoding]::UTF8.GetString($str.ToCharArray()).Length

# Works - the [char]-to-[byte] conversion problem is avoided, because there are only ASCII chars.
$str = 'x' * 17689864
([System.Text.Encoding]::UTF8.GetString($str.ToCharArray())).Length

I see the same behavior in Windows PowerShell and PowerShell 7.2.10, 7.3.8, and 7.4.0-preview.6

Do you get different results?

I don't remember the earlier version. Once I applied the PS update the bug appeared.
The code solution was provided by MS. If as you say it's conceptionally flawed then why did it work for processing over 3 million json objects in PS 5.1 without issue?
Based on your response, it sounds like PS 5.1 is the app to use when processing large responses with non-ascii.

why did it work for processing over 3 million json objects in PS 5.1 without issue?

Because 5.1 mis-decodes UTF-8 responses as ISO-8859-1-encoded in the absence of the response header explicitly indicating UTF-8. Given that Unicode ISO-8859-1 is a subset of Unicode, namely in the 8-bit range, all resulting (misinterpreted) non-ASCII characters in effect retained their original byte values (in the sense that the Unicode code points of the resulting [char] instances matched the original bytes).

That is a workaround that is only required in v5.1, and breaks in v7+.

sounds like PS 5.1 is the app to use when processing large responses with non-ascii.

No, PowerShell Core (7+) is the best edition, because the mis-decoding doesn't happen there, given that UTF-8 is now the default - no workarounds needed.

For cross-edition scripts, use the following:

$utfPost = 
  if ($PSVersionTable.PSVersion.Major -ge 7) { 
     $post.Content # No workaround needed
  } else { 
      # Functionally equivalent and conceptually clearer (not sure which is faster):
      #    [System.Text.Encoding]::UTF8.GetString($post.RawContentStream.ToArray())
     [System.Text.Encoding]::UTF8.GetString($post.Content.ToCharArray())
  }

To summarize:

What did change was that since v7.0 UTF-8 is now used to decode JSON responses by default, i.e. when the response doesn't have a charset header field.
- If memory serves, in v7.4 UTF-8 will become the consistent default for all other formats too - see Webcmdlets set default charset encoding to UTF8 #18219
Your [System.Text.Encoding]::UTF8.GetString() workaround is only needed in Windows PowerShell (the legacy edition whose latest and last version is v5.1), and in now-obsolete v6.x versions of PowerShell (Core), where the default encoding is ISO-8859-1, resulting in misinterpreted UTF-8 responses.

Thank you very much for the code. I quick tested your code recommendations and here is what I found.

PS 7.3.8
$utfPost = $post.Content - Does not work for me in PS 7.
In output file I see: 668001E1EO,876689988,,,,07-21-2023_02:37:16:464
On concole I see: 668001E1EO,876689988,,,,07-21-2023_02:37:16:464

$utfPost = [System.Text.Encoding]::UTF8.GetString($post.RawContentStream.ToArray()) - with PS 7, in console non-ascii characters look wrong but if copy and paste to another app looks good. Output file looks correct. I can work with this even if it looks incorrect on console. Tried adding the following to top of PS file for console: [Console]::OutputEncoding = [System.Text.UTF8Encoding]::new() but no change.

PS 5.1
$utfPost = [System.Text.Encoding]::UTF8.GetString($post.Content.ToCharArray()) - does not work for PS 5.1
$utfPost = [System.Text.Encoding]::UTF8.GetString($post.RawContentStream.ToArray()) - Works for PS 5.1 on console and output file.

I will do more extensive testing to confirm the code solution, but again I appreciate your code suggestion. If you would like to close this it's fine with me.

Glad to hear my code helped. I do not have the power to close your issue, but feel free to close it yourself.

The following may not matter if you've found a solution, but I am baffled by some of the symptoms you describe.

A discrepancy between to-console output and captured output normally only arises in calls to external programs (perhaps due to lack of code formatting in your previous comment, both strings look the same).

If $post.Content doesn't work in v7+, the only explanation I can think of is that the the response isn't declared as a JSON response (application/json):

If the response isn't recognized as text-based, the .Content property contains a [byte[]] array (because an instance of a different type is then returned, Microsoft.PowerShell.Commands.WebResponseObject vs. Microsoft.PowerShell.Commands.BasicHtmlWebResponseObject) - which would also explain why $post.Content.ToCharArray() doesn't work in 5.1 (and why [string]::new($post.Content).ToCharArray() did, but it is unnecessary).

If that is the case, then the following should work in both editions:

# Should work, if $post.Content is of type [byte[]]
[System.Text.Encoding]::UTF8.GetString($post.Content)

$utfPost = [System.Text.Encoding]::UTF8.GetString($([String]::new($post.Content)).ToCharArray())
looks wrong to me .

I have an API which returns a bytestream and I need to do this with - my current client is stuck in Windows PowerShell 5.1

$UtfEncoder = [System.text.Utf8Encoding]::new($false) # false = no B.o.M 
$raw = Invoke-web request @params 
$cooked = $utfEncoder.getString($raw.content)

That only works for a byte stream $UtfEncoder.GetString($raw.RawContentStream.toarray()) may be needed if the data is sent as text

$raw= Invoke-WebRequest "https://google.com"
$UtfEncoder = [System.Text.UTF8Encoding]::new($false)
$UtfEncoder.GetString($raw.RawContentStream.toarray())

Works on PS 7.3 and 5.1

Thank you everyone for the good information. I'm going to close this issue.

mtarkow3

📣 Hey mtarkow3, how did we do? We would love to hear your feedback with the link below! 🗣️

🔗 https://forms.office.com/r/P926k48jRJ

Microsoft Forms

Navigation Menu

Repository navigation

System.Byte error occurs when data size large #20488

Prerequisites

Steps to reproduce

Response body contains non-ascii characters.

If $($post.Content).length is too large than error returned on next line.

Expected behavior

Actual behavior

Error details

Environment data

Visuals

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

System.Byte error occurs when data size large #20488

Description

Prerequisites

Steps to reproduce

Response body contains non-ascii characters.

If $($post.Content).length is too large than error returned on next line.

Expected behavior

Actual behavior

Error details

Environment data

Visuals

Activity

mklement0 commented on Oct 16, 2023

mtarkow3 commented on Oct 16, 2023

mklement0 commented on Oct 16, 2023

mtarkow3 commented on Oct 16, 2023

mklement0 commented on Oct 16, 2023

mklement0 commented on Oct 16, 2023

mtarkow3 commented on Oct 16, 2023

mklement0 commented on Oct 16, 2023

jhoneill commented on Oct 16, 2023

mtarkow3 commented on Oct 18, 2023

microsoft-github-policy-service commented on Oct 18, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions