AutomationNull Behaviour

Patrick Meinecke (@SeeminglyScience)

See the comments in #9794 for the full extent of the existing discussion, starting with #9794 (comment)

AutomationNull.Value is sometimes detectable and distinguishable from $null in certain cases, for example:

using namespace System.Management.Automation.Internal

$null -is [psobject] # false
[AutomationNull]::Value -is [psobject] # true

@($null).Count # 1
@([AutomationNull]::Value).Count # 0

Patrick Meinecke (@SeeminglyScience) mentioned a possible way we could have it be handled more closely like [dbnull] and [nullstring] are being handled as of #9794, without losing its current function in the pipeline internals, in #9794 (comment):

Ideally if this were to be fixed it would just be removed from assignment, e.g. $obj = [AutomationNull]::Value would populate $obj with true null. The -is operator is one of the few things that never "lie" on occasion and I think it would be a detriment to change that.

You can also make [AutomationNull]::Value -is [AutomationNull] work by:

Removing the static keyword from the class decl

Make it inherit PSObject

Change the singleton instance to new AutomationNull()

That would allow $autoNull -is [psobject] to still work while enabling $autoNull -is [AutomationNull]. Though I think there are a few places in the compiler where it uses a pattern like if (expr.ExpressionType == typeof(PSObject) && expr.Value == AutomationNull.Value) so that might need to change.

/cc Dongbo Wang (@daxian-dbw) Michael Klement (@mklement0)

I'm personally in favor of Patrick's solution, as it enables the very clear and concise $item -is [AutomationNull] with few downsides (the biggest downside being potential implementation complications, but in my opinion these are probably worth tackling). Interested to hear any further discussion on this! 💖

Jason Shirk (@lzybkr)

using namespace System.Management.Automation.Internal
[AutomationNull]::Value -is [AutomationNull]

returning $true makes perfect sense both intrinsically and for symmetry with [DbNull]::Value -is [DbNull] and [NullString]::Value -is [NullString] returning $true.

The current workaround for detecting [System.Management.Automation.Internal.AutomationNull]::Value actually requires two tests (to also distinguish [AutomationNull]::Value from an empty collection object).

$null -eq $someValue -and @($someValue).Count -eq 0

As for why there is an occasional need to distinguish between a true $null and [System.Management.Automation.Internal.AutomationNull]::Value:

A command that produces no output technically outputs [System.Management.Automation.Internal.AutomationNull]::Value, which is distinct from a command explicitly returning $null.

While $null and [System.Management.Automation.Internal.AutomationNull]::Value are treated the same in an expression context, their behavior differs fundamentally in the pipeline:

# A true $null is sent through the pipeline. 
PS> & { $null }  | % { 'here' }
here

# A command with no output technically outputs [System.Management.Automation.Internal.AutomationNull]::Value, which is NOT sent through the pipeline - it is an empty enumeration.
PS> & { }  | % { 'here' }
# NO output

Also, because the switch statement treats its operand as an enumeration, a [System.Management.Automation.Internal.AutomationNull]::Value value causes the statement to be skipped altogether; that is, it is effectively ignored:

$val = & {}
# Because $val contains [System.Management.Automation.Internal.AutomationNull]::Value,
# the switch statement is effectively ignored;
switch ($val) {
  default { 'hi' }
}

Related issues:

You cannot pass [System.Management.Automation.Internal.AutomationNull]::Value as an argument - it is invariably converted to $null, which Jason Shirk (@lzybkr) surmised may have been an oversight - see No any way to pass empty pipeline (AutomationNull) to function parameter #9150 (comment)
The foreach statement does not distinguish between $null and [System.Management.Automation.Internal.AutomationNull]::Value - neither is enumerated.
The mystifying [AutomationNull]::Value -is [AutomationNull] being $false has echoes of [pscustomobject] @{ foo = 1 } -as [System.Management.Automation.PSCustomObject] being $null and -as [pscustomobject] always returning the LHS, with a true $null LHS being the only exception - see -as operator does not recognize System.Management.Automation.PSCustomObject instances as such #4343 and below.

Just like [AutomationNull]::Value -is [AutomationNull] currently being $false is a head-scratcher, so is [AutomationNull]::Value -is [psobject] being $true and - given the unfortunate identity of [psobject] and [pscustomobject] - [AutomationNull]::Value -is [pscustomobject] being equally $true.

[AutomationNull]::Value -is [psobject] being $true makes no sense from an end-user perspective, and looks like another case of [psobject] peeking from behind the curtain - see #5579.

The AutomationNull is defined in System.Management.Automation.Internal namespace that implies only internal non-public use. We free to change the internal AutomationNull but it make sense only if it is really needed for addressing important scenarios.

If it is purely for internal use it should never be leaking in the ways described above.

However, due to the way it's intended to behave, I'm not sure it can be purely internal and still provide the same utility.

Ilya (@iSazonov)

Ilya (@iSazonov) is correct. AutomationNull is for internal use by the runtime. It exists to distinguish an empty stream from a stream containing null.

Rain Sallow (/u/ta11ow) (@vexx32)

AutomationNull.Value is sometimes detectable and distinguishable ... for example...

All of your examples explicitly reference the type. I would hardly call that a "leak".

However, due to the way it's intended to behave, I'm not sure it can be purely internal and still provide the same utility

What do you think it should do that it currently doesn't do and why do you think that behaviour is useful? For example, this

using System.Management.Automation.Internal
[AutomationNull]::Value -is [AutomationNull]

does not qualify as useful because it's an internal class.

Michael Klement (@mklement0)

Sure, for simplicity of definition I opted to use the explicit class name. Michael Klement (@mklement0) offered several other ways to obtain the value. Also, it's not internal. I can reference the type name from PS directly. It might be in the internal namespace, but it's public.

Here's another way to get an auto-null:

$a = if ($false) { Do-Thing }

Contrived? Yes. Easily found in a real application? Absolutely. Even more so in a pipeline which may not return any value.

Rain Sallow (/u/ta11ow) (@vexx32)

Rain Sallow (/u/ta11ow) (@vexx32)

It's in the internal namespace:

System.Management.Automation.Internal

This namespace contains classes and interfaces that must be public for some reason but are not considered part of the public PowerShell API.

{master}PSCore (1:1) >  $a = if ($false) { Do-Thing }
{master}PSCore (1:2) >  $a -eq $null
True
{master}PSCore (1:3) >  $a.GetType().FullName
You cannot call a method on a null-valued expression.
At line:1 char:1
+ $a.GetType().FullName
+ ~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : InvokeMethodOnNull

{master}PSCore (1:4) >

So where does AutomationNull show up in this?

As Dongbo mentioned in the linked PR, AutomationNull doesn't respect GetType(); it's special-cased in that instance to appear as $null. However, compare $a in that example with true $null as follows:

$a -is [psobject] # true
$null -is [psobject] # false

@($a).Count # 0
@($null).Count # 1

If it's not meant to be part of the public API, then y'all need to figure out how to hide it better. 😉

Bruce Payette (@BrucePay)

Bruce Payette (@BrucePay) I've had to account for it before with methods that have a parameter typed as object[]. For example:

$a = if ($false) { Do-Thing }
[type]::GetTypeArray(@(0, $a, 1))

And to be fair, the documentation does not do a great job at explaining that it shouldn't be used (and by that I mean it says that it's imperative that you do use it). You also have to remember the audience, most PowerShell folks won't think twice about the namespace because they aren't familiar with that pattern. Granted, most folks who end up needing AutomationNull will probably be pretty technical, but still.

Whether or not it was intended to be used, I'd still advise that care be taken when it comes to breaking changes.

Rain Sallow (/u/ta11ow) (@vexx32)

Good points, Rain Sallow (/u/ta11ow) (@vexx32) and Patrick Meinecke (@SeeminglyScience).

I don't think [AutomationNull] can be hidden, because users need to understand how it differs from $null and need to be able to inspect a given value to infer its pipeline behavior - and also more obscure differences such as Patrick Meinecke (@SeeminglyScience)'s method-argument example.

In other words: it is insufficient for PowerShell to pretend that a type('s singleton) is $null when it actually behaves differently, situationally.

That difference must be discoverable, and having to resort to obscure and cumbersome workarounds to detect [AutomationNull]::Value - which you'll never discover unless you know about this "internal" type - is obviously unsatisfactory.

So, yes,

$var = & {}
$var -is [AutomationNull]

is useful - and that it currently returns $false is self-contradictory.

Furthermore, the type must be documented in the end-user docs.

Patrick Meinecke (@SeeminglyScience)

Patrick Meinecke (@SeeminglyScience), stumbling upon this again I've noticed bizarre behavior around your example:

# This FAILS, as in your example.
$anull = & {}; [Type]::GetTypeArray(@(0, $anull)).Name | Should -Be 'Int32', 'PSObject'

# This SUCCEEDS - single-element array, with just AutomationNull.
$anull = & {}; [Type]::GetTypeArray(@(, $anull)).Name | Should -Be 'PSObject'

# RETRYING the ORIGINAL command NOW SUDDENLY SUCCEEDS.
$anull = & {}; [Type]::GetTypeArray(@(0, $anull)).Name | Should -Be 'Int32', 'PSObject'

Yeah, looks like a binder bug. There must be a difference in how it approaches converting single item arrays vs more populated arrays. That would explain why the third works, the binder would still be cached.

Patrick Meinecke (@SeeminglyScience)

Thanks, Patrick Meinecke (@SeeminglyScience); I've created #11118.

Rain Sallow (/u/ta11ow) (@vexx32)

Rain Sallow (/u/ta11ow) (@vexx32)

What risk?

The significant risk of breaking all kinds of things in obscure ways because we've done something to make AutomationNull more visible rather than less visible. As I mentioned before, getting a usable system with AutomationNull took months of bug hunting.

And I have yet to hear a credible rational for changing the semanitcs of AutomationNull. In a value/expression content AutomationNull should work exactly like null. In a pipeline, it's purpose is to indicate that the pipeline returned no results which is not the same as null.

Bruce Payette (@BrucePay)

Bruce Payette (@BrucePay) Sure, but it doesn't work "exactly" like null in all cases. Therefore having a reliable and accessible way to distinguish the two is desirable so that you can properly handle it when you need to.

Your arguments are hypotheticals that we "might" break something, with very little information . The responses from Michael Klement (@mklement0) and Patrick Meinecke (@SeeminglyScience) as well as the occasional question that crops up on a fairly frequent basis in the community chat channels indicates that having a reliable way to distinguish $null from AutomationNull is worth implementing.

Unless you can demonstrate something that the proposed changes will actually break, I'm not sure your concerns can be tested in any way. An untestable hypothesis isn't especially useful as a talking point.

Rain Sallow (/u/ta11ow) (@vexx32)

To add to Rain Sallow (/u/ta11ow) (@vexx32)' s helpful comment: No one in this thread has advocated changing the semantics of AutomationNull - the proposed change in no way modifies the existing behavior, it merely enables reflection on a preexisting behavioral difference that has hitherto been undiscoverable without insider knowledge.

Anecdotal references to past struggles and insisting on not being personally convinced are no substitutes for rational debate, especially if the points that have been made aren't being addressed.

We were sure that # 9794 is correct but it was reverted. This experience says that each application area can have its own 'null' and it is probably not worth crossing them as far as possible (even if it raises a lot of questions).

Ilya (@iSazonov)

Ilya (@iSazonov), #9794 was a different story altogether, and in a sense the opposite (complement) of what is being proposed here: it tried to extend the set of types that are conflated with $null when compared it via -eq (in keeping with -eq's loose concept of equality); by contrast, here we're asking to unambiguously identify a "specific flavor of $null". Again, we're only talking about a reflection feature here that makes real-world behavioral differences discoverable - no existing behavior relating to using these values will change.

Rain Sallow (/u/ta11ow) (@vexx32)

we're only talking about a reflection feature here that makes real-world behavioral differences discoverable - no existing behavior relating to using these values will change.

Rain Sallow (/u/ta11ow) (@vexx32) Michael Klement (@mklement0) The issue description is based on #9794 and mentioned dbnull and stringnull. This is misleading. I think you need to close this discussion and create a new one where you clearly indicate the desired changes as 1, 2, 3.

Please strongly take into account a history which Bruce revealed.
Also please take into account that AutomationNull is "public" mainly for SDK (see C# vs PowerShell - ~3000 vs ~0 results)

Please strongly take into account a history which Bruce revealed.

The history he revealed is that there were a lot of bugs when they made the change that allowed autonull to be saved to a variable. Nothing is revealed there, that's a huge change that pretty obviously increases risk.

It's sort of like saying that there was a lot of obscure bugs when PowerShell's parser switched from purely token based to AST based so we shouldn't consider adding an overload to Parser.ParseInput.

If the argument was "this comes up so infrequently that it's not even worth discussing" I'd be with ya 100%. If instead the argument is "significantly more drastic and involved changes we made 10 years ago caused problems" then I have a hard time seeing that as anything other than off topic.

I mean mainly the old history and experience:

AutomationNull is primarily designed for use in the pipeline to distinguish between the value $null and no value (the empty stream.) When a pipeline returns AutomationNull it is converted into a $null on variable assignment.

AutomationNull needed to be publicly visible for various reasons. For example, a compiled cmdlet might want to explicitly return it. (Though generally, if the cmdlet doesn't call write, the pipeline processor will automatically return AutiomationNull.)

Our new history and experience is that in #9794 we got married [dbnull], [stringnull] and [null] and then stumbled upon AutiomationNull that was highlighted in the separate discussion. In the history no real scenario presents. So question is - what scenarios do we need to fix to help (1) script writers, (2) binary module developers? what can they not do? what can they not overcome? What annoys them and forces them to make bad workarounds?
Although Bruce mentioned only one scenario where AutiomationNull might be used (binary cmdlet), perhaps there are others. I would rather expect that these scenarios were discussed with the aim of hiding or getting rid of AutiomationNull because it is the visibility of AutiomationNull (knowledge that it exists) that causes questions for users, although most likely they do not need it and they can bypass the emerging problem easily.

Patrick Meinecke (@SeeminglyScience)

@PowerShell/powershell-committee discussed this. We feel that using AutomationNull directly comes up infrequently enough that it doesn't warrant the time spent on discussion already. We would also be concerned about potential breaking change as noted by Patrick Meinecke (@SeeminglyScience) and Bruce Payette (@bpayette).

Patrick Meinecke (@SeeminglyScience)

comes up infrequently enough

How often this comes up isn't the point, as argued before.

using AutomationNull directly

That wasn't the point either; it's not about using, but about discovering something that you can't help but encounter in real life, without the language giving you the ability to understand why not all apparent $null values are created equal.

Recent case in point: https://stackoverflow.com/q/60515757/45375

concerned about potential breaking change

What the proposal turned into would not result in any breaking change (as Patrick Meinecke (@SeeminglyScience) has conclusively argued, aside from his not being personally convinced of the need for a change).

Ilya (@iSazonov):

I think you need to close this discussion and create a new one

Re-reading the OP, I agree; unfortunately I didn't get around to it in time.

it is the visibility of AutomationNull (knowledge that it exists) that causes questions for users

The exact opposite is the case: most users are unaware of AutomationNull, but they see its effects, which is why they need to be given a way to discover this value.

with the aim of hiding or getting rid of AutomationNull

In light of the above: AutomationNull must neither be hidden nor can it be gotten rid of.

Stack Overflow
Powershell filter ignored in pipeline
I am writing a Chef library to make writing a custom resource for managing Microsoft MSMQ resources on Windows Server easier. Chef interfaces with Windows using Powershell 5.1.
I want to raise an...

Ilya (@iSazonov)

Ilya (@iSazonov), I finally got around to this suggestions of yours:

I think you need to close this discussion and create a new one

Please see #13465, which focuses just on -is [AutomationNull] and hopefully avoids the confusion that arose in this discussion around non-existent breaking changes.

Navigation Menu

Repository navigation

AutomationNull Behaviour #9997

21 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

AutomationNull Behaviour #9997

Description

Activity

mklement0 commented on Jun 25, 2019

iSazonov commented on Jun 25, 2019

vexx32 commented on Jun 25, 2019

bpayette commented on Jun 25, 2019

vexx32 commented on Jun 25, 2019

bpayette commented on Jun 25, 2019

vexx32 commented on Jun 25, 2019

SeeminglyScience commented on Jun 25, 2019

mklement0 commented on Jun 28, 2019

mklement0 commented on Nov 19, 2019

SeeminglyScience commented on Nov 19, 2019

mklement0 commented on Nov 19, 2019

mklement0 commented on Dec 19, 2019

21 remaining items

BrucePay commented on Feb 26, 2020

vexx32 commented on Feb 26, 2020

mklement0 commented on Feb 26, 2020

iSazonov commented on Feb 27, 2020

mklement0 commented on Feb 27, 2020

iSazonov commented on Feb 28, 2020

SeeminglyScience commented on Feb 28, 2020

iSazonov commented on Feb 28, 2020

SteveL-MSFT commented on Mar 4, 2020

mklement0 commented on Mar 4, 2020

mklement0 commented on Aug 17, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions