ConvertFrom-Json sends objects converted from a JSON array as an *array* through the pipeline.

PetSerAl

Note: Perhaps the behavior in question will rarely be noticed as problematic in the real world, so perhaps the only thing that's needed is adding a note to the documentation - I'm unclear on whether this ultimately a non-issue.

alpha.17 fixed issue #3153 with respect to ConvertTo-Json no longer wrapping ConvertFrom-Json array output in an extraneous object with value and count properties.

However, ConvertFrom-Json still sends the objects resulting from converting a JSON array as a single array through the pipeline, which is a deviation from the usual cmdlet behavior of unwrapping collections and sending the items one by one.

> ('[ 1, 2 ]' | ConvertFrom-Json | Measure-Object).Count
1  # !! PS array `1, 2` was sent as a *single* object through the pipeline.

Workaround: Enclose the ConvertFrom-Json part of the pipeline in (...), which forces enumeration:

> (('[ 1, 2 ]' | ConvertFrom-Json) | Measure-Object).Count
2  # OK, (...) forced enumeration

Alternatively, simply insert a Write-Output pipeline segment:

> ('[ 1, 2 ]' | ConvertFrom-Json | Write-Output | Measure-Object).Count
2  # OK, Write-Output forced enumeration

^{Note: On Windows PowerShell, as of v5.1, you paradoxically need Write-Output -NoEnumerate to prevent unwrapping from getting applied to the individual array elements as well.}

On the plus side, as PetSerAl points out, this preserves the input structure in a round-trip in the case of single-element JSON arrays:

> '[ 1 ]' | ConvertFrom-Json | ConvertTo-Json
[
  1
]

It is only ConvertFrom-Json's send-the-array-as-a-whole approach that enables the above round trip.

By contrast, when you start with a PS array, it doesn't work that way (which illustrates the difference in pipeline behavior):

> , 1 | ConvertTo-Json
1 # !! Input PS array was unwrapped on sending it through the pipeline

To preserve the array, you must explicitly nest the array:

> , , 1 | ConvertTo-Json
[
  1
]

# Alternative: explicitly prevent enumeration
> Write-Output -NoEnumerate (, 1) | ConvertTo-Json
[
  1
]

Environment data

PowerShell Core v6.0.0-alpha (v6.0.0-alpha.17) on Darwin Kernel Version 16.4.0: Thu Dec 22 22:53:21 PST 2016; root:xnu-3789.41.3~3/RELEASE_X86_64

An alternate illustration of this behavior is '[ 1, 2 ]' | ConvertFrom-Json | foreach { $_.Count } which returns the value 2 indicating that the array is passed as a single value as opposed to '[ 1, 2 ]' | ConvertFrom-Json | write-output | foreach { $_.Count } which will return 1,1.

Bruce Payette (@BrucePay)

Thanks, Bruce Payette (@BrucePay), but what I'm unclear on is: is this behavior - given that it deviates from usual collection-unwrapping behavior - problematic in practice?

The more I think about it, the more I believe this behavior should be changed:

I just came across a real-world problem in which the behavior is problematic, and I found that despite being aware of the issue in principle, I simply forgot, which stumped me for a while.

Having to remember this exception is a burden, especially given that there's no obvious reason for it and that the only benefit is the ability to round-trip a single-item array - something that it isn't even expected to work with PowerShell's own arrays.

@PowerShell/powershell-committee reviewed this and agrees that we should be consistent with expected PowerShell semantics and a single element array would be a single item. However, we also don't want to revert the original problem in #3153

This behavior needs to be fixed--it means you can't effectively compose pipelines that involve JSON arrays, unless you either assign to an intermediate variable, or wrap in a "subshell" to terminate the pipeline.

That is by no means obvious to discover--I found it before I found this issue by pure accident, mostly because I suspected something wrong with the pipeline objects and wanted to call GetType on the output of ConvertFrom-Json. Another engineer had already spent at least 20 minutes wondering why his pipeline didn't work.

This looks really bad as a user--Powershell's object-oriented nature should make pipelines like this a breeze, instead you get hard-to-diagnose Heisenbugs (this would have disappeared as soon as someone tried assigning it to a variable, only to reappear again later).

I think the way to fix the issue could be by introducing a switch to the ConvertFrom-Json that would modify the behaviour, either returning an array or pipeline objects, e.g.:

ConvertFrom-Json -PipelineFriendly

I wrote an advanced function while trying to solve the issue

function Split-Array {
    [CmdletBinding()]
    param (
        [Parameter(Mandatory = $true, Position = 0, ValueFromPipeline = $true)]
        [System.Object[]] $Array
    )
    
    process {
        foreach ($item in $Array) {
            $item
        }
    }
}

However, when reading throughout this ticket, I realized that Write-Output could be used instead.

I've added a simple workaround to the original post: in short: enclosing the Convert-FromJson part of the pipeline in (...) forces enumeration; simply inserting a Write-Output pipeline segment would work too (except in Windows PowerShell, where you'd paradoxically need Write-Output -NoEnumerate)).

I like the idea of introducing the switch, but I'd reverse the logic: pipeline-friendliness should be the default; thus, something like -NoEnumerate - in line with Write-Output's parameter - would be more appropriate.

Obviously, however, that would be a breaking change.

I'm glad I found this post. I spent a couple hours trying to figure out what was wrong. I originally thought the API I was working with was doing something improper. I switched over to Python and realized that the API was returning good results, the problem was Convert-FromJson was not working the way I expected it to.

I came across this in a StackOverflow question where the user was trying to pipeline the output of ConvertFrom-Json. The difference in these behaviors is very much unexpected:

PS> ConvertFrom-Json '[1, 2, 3]' | ForEach-Object  {": $_"}
: 1 2 3

PS> $x = ConvertFrom-Json '[1, 2, 3]'
PS> $x | ForEach-Object  {": $_"}
: 1
: 2
: 3
PS> ,$x | ForEach-Object  {": $_"}
: 1 2 3

PS> (ConvertFrom-Json '[1, 2, 3]') | ForEach-Object  {": $_"}
: 1
: 2
: 3

Once you know that ConvertFrom-Json is wrapping the item in an array it's clear what's going on, but until then it's very confusing. ConvertFrom-Csv certainly doesn't behave like this, but, of course, JSON is much more structured than CSV.

AikenBM

AikenBM

Once you know that ConvertFrom-Json is wrapping the item in an array

Yes, but the problem is remembering it - given that it works differently from the rest of PowerShell; the benefits of this deviation (round-tripping of single-item JSON arrays) are too slim to warrant it, in my opinion.

Steve Lee (@SteveL-MSFT)

Steve Lee (@SteveL-MSFT) I have to agree with Michael Klement (@mklement0) here, I'm also disappointed. I picked this issue not just because it's a nice entry level problem to look at the code base but also because not only have I fallen into this trap, but I also had to help lots of colleagues with the same issue.

Not only is this behavior completely unintuitive, since it behaves contrary to almost (? I can't think of a single counterexample actually) everything else, it is also as far as I can see nowhere documented.

And not only is the behavior unintuitive, when trying to analyse an issue (which will often involve assigning partial expressions to variables to make it easier to debug), you're likely to unknowingly fix the underlying problem which means you might not even understand the problem at all, just that the problem magically went away after a bit of debugging.

The same problem also makes the current solution very fragile, since a simple refactoring will change the observable behavior.

To add one further point to what's already mentioned, I think we should avoid the inherent inconsistency of having one common command with -NoEnumerate (i.e., Write-Output) and other with -Enumerate

Variant behaviours may be desired, and that is why we have the option to select non-standard behaviour, but we should not have two completely different standard behaviours. Documentation or no, this difference is not an immediately obvious behaviour in that users not already very familiar with the commands will be able to diagnose. As such, I think the default behaviours should be consistent.

Daniel Sturm (@danstur)

Daniel Sturm (@danstur) I believe the @PowerShell/powershell-committee misunderstood the situation when we discussed it, we'll discuss this again on Monday.

@PowerShell/powershell-committee reviewed this again. There is a case of an array of arrays that would break with this change, but this is unlikely and can be addressed with the -noenumerate switch. So we support adding the -noenumerate switch to return an array as a single object and have ConvertFrom-Json return elements individually to the pipeline following expected PowerShell behavior by default for arrays.

ConvertFrom-Json: Unwrap Collections by default

PMC Tools: Work around PowerShell/PowerShell#3424

PMC Tools: Work around PowerShell/PowerShell#3424 (#22134)

After #10861 can we close the issue?

Ilya (@iSazonov)

Ilya (@iSazonov): Yes, we can:

PS> ('[ 1, 2, 3]' | ConvertFrom-Json | Measure-Object).Count
3 # implies that the array elements were sent one by one through the pipeline.

Navigation Menu

Repository navigation

ConvertFrom-Json sends objects converted from a JSON array as an array through the pipeline. #3424

Environment data

48 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ConvertFrom-Json sends objects converted from a JSON array as an *array* through the pipeline. #3424

Description

Environment data

Activity

BrucePay commented on Mar 31, 2017

mklement0 commented on Mar 31, 2017

mklement0 commented on Jun 10, 2017

SteveL-MSFT commented on Jun 14, 2017

bojanrajkovic commented on Nov 17, 2017

san7hos commented on Nov 24, 2017

mklement0 commented on Nov 24, 2017

elijahgagne commented on Dec 2, 2017

AikenBM commented on Mar 6, 2018

mklement0 commented on Mar 6, 2018

48 remaining items

danstur commented on Nov 2, 2019

vexx32 commented on Nov 2, 2019

SteveL-MSFT commented on Nov 3, 2019

SteveL-MSFT commented on Nov 7, 2019

iSazonov commented on Sep 9, 2020

mklement0 commented on Sep 9, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

ConvertFrom-Json sends objects converted from a JSON array as an array through the pipeline. #3424