Get-Item ./.hidden-file fails with "Could not find item"

This behavior is on macOS 11.3.

Steps to reproduce

> Get-Item ~/.profile

Expected behavior

    Directory: /Users/nimazzuc

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-----           4/28/2021 12:25 PM            107 .profile

Actual behavior

Get-Item: Could not find item /Users/nimazzuc/.profile.

Environment data

WSManStackVersion = 3.0
PSEdition = Core
Platform = Unix
PSCompatibleVersions = 1.0 2.0 3.0 4.0 5.0 5.1.10032.0 6.0.0 6.1.0 6.2.0 7.0.0 7.1.3
GitCommitId = 7.1.3
OS = Darwin 20.4.0 Darwin Kernel Version 20.4.0: Fri Mar  5 01:14:02 PST 2021; root:xnu-7195.101.1~3/RELEASE_ARM64_T8101
PSRemotingProtocolVersion = 2.3
SerializationVersion = 1.1.0.1
PSVersion = 7.1.3

Additionally, the Mode thing is weird, shouldn't that show the unix file mode?

-Force

I'm able to reproduce this:

> gi ~/.vimrc
Get-Item: Could not find item /Users/rjmholt/.vimrc.

> gi -force ~/.vimrc

    Directory: /Users/rjmholt

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
---h-           4/22/2021  3:46 PM           1030 .vimrc

> stat ~/.vimrc
16777233 1451798 -rw-r--r-- 1 rjmholt staff 0 1030 "Apr 22 16:28:44 2021" "Apr 22 15:46:55 2021" "Apr 22 15:46:55 2021" "Apr 22 15:46:55 2021" 4096 8 0 /Users/rjmholt/.vimrc

The difference in the modes shown seems to imply that PowerShell lacks permissions in some sense perhaps.

James Truher (@JamesWTruher)

/cc James Truher (@JamesWTruher) Steve Lee (@SteveL-MSFT)

Get-Error output:

> get-error

Exception             :
    Type    : System.IO.IOException
    Message : Could not find item /Users/rjmholt/.vimrc.
    HResult : -2146232800
TargetObject          : /Users/rjmholt/.vimrc
CategoryInfo          : ObjectNotFound: (/Users/rjmholt/.vimrc:String) [Get-Item], IOException
FullyQualifiedErrorId : ItemNotFound,Microsoft.PowerShell.Commands.GetItemCommand
InvocationInfo        :
    MyCommand        : Get-Item
    ScriptLineNumber : 1
    OffsetInLine     : 1
    HistoryId        : 169
    Line             : gi ~/.vimrc
    PositionMessage  : At line:1 char:1
                       + gi ~/.vimrc
                       + ~~~~~~~~~~~
    InvocationName   : gi
    CommandOrigin    : Internal
ScriptStackTrace      : at <ScriptBlock>, <No file>: line 1
PipelineIterationInfo :

This is standart syntax for hidden "items".

> help Get-item -Parameter Force

-Force <System.Management.Automation.SwitchParameter>
    Indicates that this cmdlet gets items that can't otherwise be accessed, such as hidden items. Implementation varies from provider to provider.....

This has always been the norm as far as I'm aware, yeah. -Force has always been required when querying for hidden and sometimes system files/folders.

This is... very weird to me, but fair enough. I still think it's a bug (or at least a thing that should be changed), since hidden files aren't "different" from regular files on unices.

It is not bug, it is default PowerShell behavior for years. PowerShell never gets/enumerates objects with hidden and system attributes.

since hidden files aren't "different" from regular files on unices.

It is Unix standard - first dot in name assumes that the file is hidden. .Net maps sign to FileAttribute.Hidden and PowerShell follows the Unix and .Net behavior.

In Get-ChildItem we can use -Hidden and -System switches to get such files in output.

Even if it isn't a bug, the behavior is counterintuitive and inconvenient, given that a specific, known-to-exist file is being asked for: If I know that it's there, its hidden-or-not status should be irrelevant.

Contrast that with the arguably more sensible behavior of POSIX-compatible shells such as bash, and the (POSIX-mandated) ls utility:

ls behavior (shell-independent):
- Enumeration behavior (no path / only parent path argument, e.g., ls ~):
  - Parallels Get-ChildItem's enumeration behavior: hidden items are not included by default, and there's an opt-in to include them (-A, the equivalent of -Force)
- Literal direct-target path-argument behavior (e.g, ls ~/.bashrc):
  - The item is listed, irrespective of whether it is hidden or not.
Globbing behavior, which is purely a shell feature (that is, the shell itself resolves the wildcard expression up front and passes the resulting literal paths to the target command):
- Unless the pattern starts with ., hidden items are not included by default; e.g.:
  - sh -c 'printf %s\\n ~/*bashrc' does not find ~/.bashrc (unfortunately, bash, dash, and ksh then pass the unexpanded pattern through, which is unfortunate, but that's a separate issue)
  - but sh -c 'printf %s\\n ~/.*bashrc' does, due to the name-pattern-initial .
- Specific shells may have shell-environment options to include hidden items by default, such as bash's dotglob option.
- Note that the only reason I'm using sh -c in the examples above is the need to bypass PowerShell's own native globbing (see below).

The following following changes make sense to me:

Make Get-Item / Get-ChildItem find hidden items directly specified by their literal path, without the need for -Force.
Make .-initial wildcards such as .*rc match hidden items without the need for -Force.
- Ditto for native globbing (wildcard resolution when calling external programs), to align with native-shell behavior.
- Note that native globbing fundamentally lacks a mechanism to request inclusion of hidden items, as discussed in Native globbing on Unix lacks ability to include hidden items #4683

The hidden dot-files behaviour was originally a bug in ls, so I don't think it gets any credit there. What's more, the dot-file concept is specific to ls as a utility — things like stat don't care what a file is called.

On the other hand, Get-ChildItem and Get-Item (which I think is arguably more like stat than ls) share a common implementation because they are provider traversal commands under which the filesystem provider is just one implementation, so the provider must have a single consistent behaviour underneath both of them. This isn't an implementation detail; PowerShell providers are a user-level concept and each *-Item* cmdlet corresponds to an established provider API.

Incidentally this is actually much closer in spirit to Plan9's walk and reminds me of the fact that Plan9 has no recursive -r flag on utilities because their philosophy was that there should only be one code path for that.

With that said though, PowerShell providers do differentiate between getting an item and getting child items. Get-Item uses the former API:

PowerShell/src/System.Management.Automation/namespaces/FileSystemProvider.cs

Lines 1242 to 1256 in a95b2cb

    
           /// <summary> 
        
           /// Gets the item at the specified path. 
        
           /// </summary> 
        
           /// <param name="path"> 
        
           /// A fully qualified path representing a file or directory in the 
        
           /// file system. 
        
           /// </param> 
        
           /// <returns> 
        
           /// Nothing.  FileInfo and DirectoryInfo objects are written to the 
        
           /// context's pipeline. 
        
           /// </returns> 
        
           /// <exception cref="System.ArgumentException"> 
        
           ///     path is null or empty. 
        
           /// </exception> 
        
           protected override void GetItem(string path)

While Get-ChildItem uses the latter:

PowerShell/src/System.Management.Automation/namespaces/FileSystemProvider.cs

Lines 1504 to 1529 in a95b2cb

    
           /// <summary> 
        
           /// Gets the child items of a given directory. 
        
           /// </summary> 
        
           /// <param name="path"> 
        
           /// The full path of the directory to enumerate. 
        
           /// </param> 
        
           /// <param name="recurse"> 
        
           /// If true, recursively enumerates the child items as well. 
        
           /// </param> 
        
           /// <param name="depth"> 
        
           /// Limits the depth of recursion; uint.MaxValue performs full recursion. 
        
           /// </param> 
        
           /// <returns> 
        
           /// Nothing.  FileInfo and DirectoryInfo objects that match the filter are written to the 
        
           /// context's pipeline. 
        
           /// </returns> 
        
           /// <exception cref="System.ArgumentException"> 
        
           ///     path is null or empty. 
        
           /// </exception> 
        
           protected override void GetChildItems( 
        
               string path, 
        
               bool recurse, 
        
               uint depth) 
        
           { 
        
               GetPathItems(path, recurse, depth, false, ReturnContainers.ReturnMatchingContainers); 
        
           }

So conceivably we could alter the behaviour in only Get-Item without introducing deep inconsistency, here:

PowerShell/src/System.Management.Automation/namespaces/FileSystemProvider.cs

Lines 1407 to 1414 in a95b2cb

    
           if (!isContainer) 
        
           { 
        
               if (!hidden || Force || showHidden || filterHidden || switchFilterHidden) 
        
               { 
        
                   s_tracer.WriteLine("Got file info: {0}", result); 
        
                   return result; 
        
               } 
        
           }

It's worth noting that PowerShell itself doesn't decide that files whose names start with . are hidden. We rely on an attribute that .NET gives us:

PowerShell/src/System.Management.Automation/namespaces/FileSystemProvider.cs

Line 1380 in a95b2cb

bool hidden = attributes.HasFlag(FileAttributes.Hidden);

.NET has decided to inherit ls's baggage here and as a result, we have no way of knowing why a file was hidden. So this behaviour would need to apply to all hidden files (e.g. if a file has the BSD hidden flag set and starts with a ., we have no .NET-native way to know about the former). More importantly, this would imply changes on Windows as well, since again, we rely on .NET's abstraction here.

The main problem I see, clearly from the responses here, is that this is an established behaviour — users have learned that this happens with hidden files and that -Force is required much like with ls -a.

Anyway, I don't have a particular horse in this race (I was mainly interested to learn more on the topic). But generally speaking, I'd say you need a pretty compelling argument to change an established behaviour in a core functionality like this.

Rob Holt (@rjmholt)

Rob Holt (@rjmholt), after having just posted #10238 (comment), I figured: what the heck, one for the road, for old time's sake:

the dot-file concept is specific to ls as a utility

As shown above, all major POSIX-like shells have adopted this convention.
That this convention may have been a historical accident is an interesting tidbit, but no more (a shell with an ironclad commitment to backward-compatibility should appreciate that).

users have learned that this happens with hidden files and that -Force is required much like with ls -a.

That's great. And it's appropriate for enumeration behavior and for wildcards that don't start with .

But it's absurd otherwise - when you know you're looking for a hidden file, either by literal path or by deliberately formulated wildcard - as evidenced by the following:

# Tab-complete a hidden file:
PS> Get-Item ~/.ba<tab> # tab-completes to '/Users/jdoe/.bashrc'
Get-Item: Could not find item /Users/jdoe/.bashrc.

Now, you could argue that the tab-completion behavior should be fixed.
But I hope you won't.

Yeah, the fact that Get-Item requires -Force to "find" a hidden file just means that one should always pass -Force to Get-Item, since:

function Get-JsonElement {
  [CmdletBinding()]
  Param([Parameter(Mandatory)][String]$Path, [Parameter(Mandatory)][String]$JPointer)

  $File = Get-Item $Path
  ...
}

this is entirely reasonable code to write; additionally:

> Get-JsonElement .config.json '.foo.bar'

is also entirely reasonable code to write. There is no reason that putting these two bits of code together should fail, assuming .config.json exists, yet the default is that it does fail with a File Not Found error (which is... a very odd error, I must say).

EDIT: I would also argue that the behavior makes no sense on Windows either; why does Get-Item ~/AppData fail. That is very, very weird behavior.

I know this isn't the solution many of you are looking for, but as a workaround at least you can put this at the top of your scripts (or inside your $PROFILE):

$script:PSDefaultParameterValues = @{
    "Get-Item:Force" = $true
    "Get-ChildItem:Force" = $true
}

Documentation here: https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_parameters_default_values?view=powershell-7.1

It sets -Force by default on all your subsequent Get-Item and Get-ChildItem calls.

about_Parameters_Default_Values - PowerShell
Describes how to set custom default values for cmdlet parameters and advanced functions.

jantari

Thanks, jantari, but just to spell it out: you workaround makes Get-ChildItem and Get-Item return hidden items unconditionally.

A proper fix would mean:

Always allow items targeted by literal path, irrespective of their hidden status; never require -Force in that case.
- As an aside: requiring -Force for removal is justified, and Remove-Item already behaves correctly in this case: if -Force is absent, instead of pretending that the item doesn't exist (the way Get-Item always and Get-ChildItem situationally does), it acknowledges the existence and states that the hidden nature of the item requires -Force.
Omit hidden items by default when enumerating items in a directory, which happens in two cases:
- Explicitly, with Get-ChildItem
- Implicitly, via globbing (resolving wildcard expressions to matching items)
To include hidden items in these enumeration cases, -Force is needed.
- Additionally, on Unix only, a wildcard expression (path component) that starts with . should implicitly act as if -Force had also been passed (on Windows, an item's name doesn't signal its hidden status, so this doesn't apply)
- A related challenge - again on Unix only - is that the globbing applied when calling native utilities provides doesn't allow for a -Force switch (Native globbing on Unix lacks ability to include hidden items #4683).

Conversely, to summarize what's broken with respect to literal paths to hidden items specified without -Force:

Get-Item is broken with respect to both files and directories - it pretends they don't exist.
Get-ChildItem is broken with respect to files only - it happily targets hidden directories (see code below) and non-hidden files inside them (by direct path).
- It is in bad company there: cmd.exe's dir exhibits the same partially broken behavior; additionally, del is broken in the sense that it pretends that hidden items don't exist (cf. the sensible Remove-Item behavior described above).

Furthermore, Rob Holt (@rjmholt):

.NET has decided to inherit ls's baggage here

To recap: .-initial items being considered hidden from enumeration is a decades-old convention observed on all Unix-like platforms.

and as a result, we have no way of knowing why a file was hidden.

We don't need to know, we can rely on .NET's enumeration behavior when exclusion of hidden items is requested (by default, nothing is excluded).

So this behaviour would need to apply to all hidden files (e.g. if a file has the BSD hidden flag set

At least as of .NET 5, the .NET file-system enumeration APIs don't know about this alternative, name-independent BSD/macOS mechanism of hiding files, and, similarly, neither does ls on macOS (where only the platform's file manager, Finder, observes it). [Update: This may change in .NET 6 - see https://github.com/dotnet/runtime/pull/52235]

Again, deferring to .NET should be fine.

To further illustrate the inconsistency in Get-Item's (and partially Get-ChildItem's) behavior with respect to literal paths directly targeting a hidden item: The following demonstrates that other file-processing cmdlets do find hidden items by literal path, by (often invariable) default, and also juxtaposes the behavior with that of the platform-native shell / utilities:

# Switch to a temporary directory.
Push-Location -ea Stop ($tmpDir = (New-Item -Type Directory -Force (Join-Path Temp:/ $PID)).FullName)

try {

  # Create a hidden directory... 
  $null = New-Item -Type Directory -ea Stop .hiddenTmpDir; if ($IsWindows) { attrib +h .hiddenTmpDir }
  # ... and a hidden file.
  $null = New-Item -Type File -ea Stop .hiddenTmpFile; if ($IsWindows) { attrib +h .hiddenTmpFile }

  # Add content to the hidden file.
  # Add-Content targets the hidden file just fine, if it is targeted with a literal path.
  'hi1' | Add-Content -ea Stop .hiddenTmpFile
  # Create a non-hidden file in the hidden directory.
  # Note using the hidden directory as part of the file path works just fine.
  'hi2' > .hiddenTmpDir/test.txt
  # Ditto, with Set-Content
  'hi3' | Set-Content -ea Stop .hiddenTmpDir/test.txt

  # Get-ChildItem accepts literal paths to hidden directories and to files inside hidden directories,
  # but NOT if you directly target a file that is itself hidden, .hiddenTmpFile.
  (Get-ChildItem .hiddenTmpDir).Name
  (Get-ChildItem .hiddenTmpDir/test.txt).Name

  # Test the existence of the hidden file, the hidden dir. and the non-hidden
  # file inside the hidden dir.
  Test-Path .hiddenTmpFile
  Test-Path .hiddenTmpDir
  Test-Path .hiddenTmpDir/test.txt

  # Get the content of the hidden file and of one of the non-hidden files in 
  # the hidden directory.
  Get-Content .hiddenTmpFile
  Get-Content .hiddenTmpDir/test.txt

  # Set-Location / Push-Location accepts literal paths to hidden directories.
  Push-Location
  Set-Location .hiddenTmpDir
  Pop-Location

  # Removing a non-hidden file in a hidden dir. works as-is
  Remove-Item .hiddenTmpDir/test.txt

  # By contrast, removing an item that is itself hidden - justifiably - 
  # requires -Force, but note that Remove-Item does NOT pretend that the
  # the item *isn't there* if you neglect to use -Force.
  Remove-Item -Force .hiddenTmpFile
  Remove-Item -Force -Recurse .hiddenTmpDir

  '---- Native shell / external utilities:'

  # Create a hidden directory... 
  $null = New-Item -Type Directory -ea Stop .hiddenTmpDir; if ($IsWindows) { attrib +h .hiddenTmpDir }
  # ... and a hidden file.
  $null = New-Item -Type File -ea Stop .hiddenTmpFile; if ($IsWindows) { attrib +h .hiddenTmpFile }
  # ... and a non-hidden file in the hidden dir.
  'hi3' > .hiddenTmpDir/test.txt

  if ($isWindows) {
    cmd /v /c 'cd .hiddenTmpDir & echo !CD!'
    cmd /c 'dir /b .hiddenTmpDir'
    cmd /c 'type .hiddenTmpFile'
    cmd /c 'type .hiddenTmpDir\test.txt'
    cmd /c 'if exist .hiddenTmpFile echo exists'
    cmd /c 'if exist .hiddenTmpDir echo exists'
    cmd /c 'if exist .hiddenTmpDir\test.txt echo exists'
    cmd /c 'del .hiddenTmpDir\test.txt'
    cmd /c 'rd /q /s .hiddenTmpDir'
  }
  else {
    sh -c 'cd .hiddenTmpDir; echo $PWD'
    ls   .hiddenTmpFile
    stat .hiddenTmpFile
    ls -d .hiddenTmpDir
    sh -c 'test -e .hiddenTmpFile && echo exists'
    sh -c 'test -e .hiddenTmpDir && echo exists'
    sh -c 'test -e .hiddenTmpDir/test.txt && echo exists'
    cat .hiddenTmpFile
    cat .hiddenTmpDir/test.txt
    rm .hiddenTmpFile
    rm .hiddenTmpDir/test.txt
    rm -r .hiddenTmpDir
  }

}
finally {
  # Clean up.
  Pop-Location; Remove-Item -Force -Recurse $tmpDir
}

I think I've said all I have to say, so I'm unsubscribing from this thread.

The Engine Working Group discussed this issue today, and we agree it's an enhancement that we should consider. But for this particular issue, it's more of a question, and it was answered, so we will resolve this issue as "Answered". A separate enhancement issue #15477 is opened and linked to this issue.

Navigation Menu

Repository navigation

Get-Item ./.hidden-file fails with "Could not find item" #15323

Steps to reproduce

Expected behavior

Actual behavior

Environment data

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	/// <summary>
	/// Gets the item at the specified path.
	/// </summary>
	/// <param name="path">
	/// A fully qualified path representing a file or directory in the
	/// file system.
	/// </param>
	/// <returns>
	/// Nothing. FileInfo and DirectoryInfo objects are written to the
	/// context's pipeline.
	/// </returns>
	/// <exception cref="System.ArgumentException">
	/// path is null or empty.
	/// </exception>
	protected override void GetItem(string path)

	/// <summary>
	/// Gets the child items of a given directory.
	/// </summary>
	/// <param name="path">
	/// The full path of the directory to enumerate.
	/// </param>
	/// <param name="recurse">
	/// If true, recursively enumerates the child items as well.
	/// </param>
	/// <param name="depth">
	/// Limits the depth of recursion; uint.MaxValue performs full recursion.
	/// </param>
	/// <returns>
	/// Nothing. FileInfo and DirectoryInfo objects that match the filter are written to the
	/// context's pipeline.
	/// </returns>
	/// <exception cref="System.ArgumentException">
	/// path is null or empty.
	/// </exception>
	protected override void GetChildItems(
	string path,
	bool recurse,
	uint depth)
	{
	GetPathItems(path, recurse, depth, false, ReturnContainers.ReturnMatchingContainers);
	}

	if (!isContainer)
	{
	if (!hidden \|\| Force \|\| showHidden \|\| filterHidden \|\| switchFilterHidden)
	{
	s_tracer.WriteLine("Got file info: {0}", result);
	return result;
	}
	}

Get-Item ./.hidden-file fails with "Could not find item" #15323

Description

Steps to reproduce

Expected behavior

Actual behavior

Environment data

Activity

237dmitry commented on Apr 29, 2021

rjmholt commented on Apr 29, 2021

rjmholt commented on Apr 29, 2021

rjmholt commented on Apr 29, 2021

237dmitry commented on Apr 29, 2021

vexx32 commented on Apr 29, 2021

strega-nil commented on Apr 29, 2021

iSazonov commented on Apr 30, 2021

mklement0 commented on Apr 30, 2021

rjmholt commented on Apr 30, 2021

mklement0 commented on Apr 30, 2021

strega-nil commented on May 1, 2021

jantari commented on May 3, 2021

mklement0 commented on May 4, 2021

daxian-dbw commented on May 28, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions