7.4.0 Breaking Change: Tab expansion no longer converts tildes to full paths, breaking path passing to executables on Windows

Prerequisites

Write a descriptive title.
Make sure you are able to repro it on the latest released version
Search the existing issues.
Refer to the FAQ.
Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce

With versions prior to 7.4.0, pressing Tab to get completion on a path with ~ converted that path into the full path for my home folder.

Type dir ~\down and press Tab:

In 7.3.9, it will be completed as dir C:\Users\bradwilson\Downloads\
In 7.4.0, it will be completed as dir ~\Downloads\

For built-in commands (like dir), this is fine. For executables, this passes a path with ~ in it, which makes using such paths impossible unless the executable has its own support for mapping ~ to the home folder. For example, if I try to run notepad ~\.config\git\config, it will tell me the path isn't found; if I run notepad C:\Users\bradwilson\.config\git\config, it will open the file appropriately.

This breaks more than a decade of muscle memory expecting ~ to be translated into the correct path, as I've been using PowerShell as my shell since it was called Monad.

This appears to have been purposefully introduced in #19489. I cannot find any way to restore the old behavior short of sticking w/ version 7.3.9.

Expected behavior

PS> TabExpansion2 '~\Downloads' | select -ExpandProperty CompletionMatches

CompletionText                ListItemText        ResultType ToolTip
--------------                ------------        ---------- -------
C:\Users\bradwilson\Downloads Downloads    ProviderContainer C:\Users\bradwilson\Downloads

Actual behavior

PS> TabExpansion2 '~\Downloads' | select -ExpandProperty CompletionMatches

CompletionText ListItemText        ResultType ToolTip
-------------- ------------        ---------- -------
~\Downloads    Downloads    ProviderContainer C:\Users\bradwilson\Downloads

Error details

No response

Environment data

Name                           Value
----                           -----
PSVersion                      7.3.9
PSEdition                      Core
GitCommitId                    7.3.9
OS                             Microsoft Windows 10.0.22631
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0


Name                           Value
----                           -----
PSVersion                      7.4.0
PSEdition                      Core
GitCommitId                    7.4.0
OS                             Microsoft Windows 10.0.22631
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Visuals

No response

In fact, when this behavior finally appeared in 7.4.0-rc.1, I was glad about it. But perhaps this is not as relevant in Windows as it is in Linux.

It's very counterproductive in Windows.

You could argue it's somewhat counterproductive in Linux as well unless every single application you use from the command line already maps ~. Many of the built-in utilities probably already have to do this, so it's less obvious when failures will materialize.

PowerShell follows POSIX (partially) and translates ~ before calling executables on Linux/macOS.

On Linux, all native applications interpret the tilde as a synonym for the user's home directory. It’s just that the command line is shorter due to the fact that it does not expand the path to the full one. The same cannot be said about Windows; most utilities probably simply will not understand what this path is.

It's worth pointing out that the Unix-y apps on Windows (for example, all the tools that ship with Git) do map ~ but will only do it if you pass Unix-y separators.

Using less version 563 (shipped with Git 2.29.2.windows.2), these work:

7.3.9: less C:\Users\bradwilson\.config\git\config
And crafted by hand: less ~/.config/git/config

But this doesn't:

7.4.0: less ~\.config\git\config results in ~.configgitconfig: No such file or directory

Since PowerShell uses the backslash when tab completing, it's creating command lines that will be broken with these Unix-y tools.

To be clear, I don't think the use of the backslash is wrong for PowerShell Core on Windows, as the Unix-y tools do work as expected with backslashes but only with fully qualified paths (which Tab completion no longer generates in 7.4.0).

I was talking specifically about Linux as the actual installed OS. Not WSL, not ported utilities, but about my main OS, which is the only one on the computer (Windows on the older notebook).

Therefore, I would like everything to remain as it is, at least on Linux.

I'm fine with the new behavior being the default, as long as I have a trivial way to opt out.

Yes, as you've discovered this was an intentional change and there's currently no way to change the behavior. A quick workaround you can implement on your own machine is to update the tabexpansion2 function to replace the ~ character in the completion results with your home path.

Assuming this is something that people want fixed, I can think of the following solutions:

Permanently revert this change
Add a command to control the behavior, like in: Add commands for controlling completion options #19518 or if that will take too long, revert the change until such a command is added
Make PS resolve ~ before passing it in as an argument to native commands.

Quick and dirty tabexpansion2 workaround:

Details

function TabExpansion2
{
    <# Options include:
         RelativeFilePaths - [bool]
             Always resolve file paths using Resolve-Path -Relative.
             The default is to use some heuristics to guess if relative or absolute is better.

       To customize your own custom options, pass a hashtable to CompleteInput, e.g.
             return [System.Management.Automation.CommandCompletion]::CompleteInput($inputScript, $cursorColumn,
                 @{ RelativeFilePaths=$false }
    #>

    [CmdletBinding(DefaultParameterSetName = 'ScriptInputSet')]
    [OutputType([System.Management.Automation.CommandCompletion])]
    Param(
        [Parameter(ParameterSetName = 'ScriptInputSet', Mandatory = $true, Position = 0)]
        [AllowEmptyString()]
        [string] $inputScript,

        [Parameter(ParameterSetName = 'ScriptInputSet', Position = 1)]
        [int] $cursorColumn = $inputScript.Length,

        [Parameter(ParameterSetName = 'AstInputSet', Mandatory = $true, Position = 0)]
        [System.Management.Automation.Language.Ast] $ast,

        [Parameter(ParameterSetName = 'AstInputSet', Mandatory = $true, Position = 1)]
        [System.Management.Automation.Language.Token[]] $tokens,

        [Parameter(ParameterSetName = 'AstInputSet', Mandatory = $true, Position = 2)]
        [System.Management.Automation.Language.IScriptPosition] $positionOfCursor,

        [Parameter(ParameterSetName = 'ScriptInputSet', Position = 2)]
        [Parameter(ParameterSetName = 'AstInputSet', Position = 3)]
        [Hashtable] $options = $null
    )

    End
    {
        $TempRes = if ($psCmdlet.ParameterSetName -eq 'ScriptInputSet')
        {
            [System.Management.Automation.CommandCompletion]::CompleteInput(
                <#inputScript#>  $inputScript,
                <#cursorColumn#> $cursorColumn,
                <#options#>      $options)
        }
        else
        {
            [System.Management.Automation.CommandCompletion]::CompleteInput(
                <#ast#>              $ast,
                <#tokens#>           $tokens,
                <#positionOfCursor#> $positionOfCursor,
                <#options#>          $options)
        }

        [System.Management.Automation.CompletionResult[]]$NewCompletionList = foreach ($Item in $TempRes.CompletionMatches)
        {
            if ($Item.CompletionText -like "*~*")
            {
                [System.Management.Automation.CompletionResult]::new(
                    $Item.CompletionText.Replace('~', $HOME),
                    $Item.ListItemText,
                    $Item.ResultType,
                    $Item.ToolTip
                )
            }
            else
            {
                $Item
            }
        }

        $TempRes.CompletionMatches = $NewCompletionList
        $TempRes
    }
}

MartinGC94

Let me provide some context:

On Unix-like platforms it is the (POSIX-compatible) shell (such as bash) that performs tilde (~) expansion, not the standard utilities (such as as ls) themselves; the same applies to globbing (wildcard expansion in filenames), among other such so-called shell expansions. You can verify this with /bin/ls --% ~ from PowerShell.

On Unix-like platforms only, PowerShell emulates this behavior (as well as globbing) when calling external programs:
- That is, it expands an unquoted ~ - either in isolation of if followed by/ (but not \) - to the value of $HOME.
- As an aside: This emulation is incomplete:
  - Syntactically: Use of ` to escape ~ - /bin/echo `~- should work like /bin/echo '~' and /bin/echo "~", i.e. should suppress expansion, but currently doesn't; the inverse is true for /bin/echo ~/'foo bar', which should expand -d due to the ~ itself being unquoted, but doesn't (if you remove the space, it works): Unix: Emulation of ~ (tilde) expansion for native utilities is syntactically incomplete #20754
  - Lack of support for other users' home directories: ~user1 works in POSIX-compatible shells to refer to the home directory of user1 (verify from PowerShell with sh -c "echo ~$env:USER"), but not in PowerShell: Implement ~username expansion on Windows and Linux #12387
On Windows, where the native shell (cmd.exe) has no such expansion features, PowerShell does not perform them.
- Individual cross-platform utilities, especially those with a Unix heritage, may compensate for this by implementing these expansions themselves, though possibly only globbing.

Given the above, I propose the following variation of MartinGC94's proposal 1:

Revert the change on Windows only, i.e. go back to making ~ expand to the value of $HOME only there - conceivably, it could even expand to verbatim $HOME, so as to preserve the intent to express the path abstractly.

While making PowerShell emulate literal ~ expansion on argument-passing to external programs on Windows also may seem appealing, there's a greater risk of breaking existing Windows-only code that never anticipated ~ as having special meaning and therefore requiring quoting.
Also, emulating globbing too is definitely not an option, as a program expecting, say, file*.txt verbatim, then receiving multiple arguments (file1.txt file2.txt ...) could break.

Taking a step back, re proposal 3 (unquoted ~ expansion on external-program argument-passing also on Windows):
It may be an option IF:

We're not concerned about the resulting asymmetry (tilde expansion then works everywhere, globbing (filename expansion) only on Unix).
More importantly, that the technically breaking change resulting from expanding a token-initial, unquoted ~ (either in isolation or if followed by \ or /) to the current user's home directory falls into bucket 3, i.e. is unlikely to break existing code.

For what it's worth, the problem I have is that I don't think you went far enough. I really don't want TabCompletion to replace generic values like ~ with the specific value on my computer. The same goes for or $home and $pwd and any other variable that I might have a partial path in.

How about instead of reverting a good feature, we replace the literal ~ at the front of the path with $HOME and then ... leave all the variables as variables in the path?

Since PowerShell expands variables when passing arguments to native apps, this gets us the best of all worlds:

Tab-completed paths work with native apps, like they used to
The shortest syntax like ~/Doc{Tab} still tab completes
Tab-completion will not make my carefully generic "$HOME/Docu{Tab}" into a script that only works on my computer.
Does not require potentially breaking changes to argument passing

NOTE: the catch to this is that it doesn't work if they're single-quoting the path they are tab-completing, in which case we might want to resort to the old behavior...

I am planning on creating a PR to stop it from replacing variables with their resolved values but that's a separate issue from this.
I don't like the idea of replacing ~ with $home when tab completing because it would feel weird and confusing for anyone outside of this discussion. It also wouldn't be accurate because the home location that ~ refers to is set in the provider, while $home just points to the same static location always. See:

PS C:\Users\Martin> $Provider=Get-PSProvider FileSystem
PS C:\Users\Martin> $Provider.Home
C:\Users\Martin
PS C:\Users\Martin> $Provider.Home = 'C:\Windows'
PS C:\Users\Martin> TabExpansion2 'ls ~\' | select -ExpandProperty CompletionMatches | select -First 1

CompletionText ListItemText        ResultType ToolTip
-------------- ------------        ---------- -------
~\appcompat    appcompat    ProviderContainer C:\Windows\appcompat

PS C:\Users\Martin> $HOME
C:\Users\Martin

On Linux, all native applications interpret the tilde as a synonym for the user's home directory

Not exactly true. The tilde is expanded by the shell (bash, sh, zsh etc). The applications don't need to map the tilde in command line arguments. Just like on UNIX the shell expands file wildcards, so the application does not need to enumerate directories, the shell did it for them.

$ cat test.c
#include <unistd.h>
#include <string.h>

int main(int argc,char **argv)
{
        write(1,argv[1],strlen(argv[1]));
        return 0;
}
$ cc test.c
$ ./a.out ~
/home/bythesea

No interpretation by the native app.

The shell does not expand if quoted

$ ./a.out '~'
~

and

$ ./a.out "~"
~

Now that is a gotcha, because shell does expand environment variables within double quotes but not within single quotes. So shell is doing special treatment for tilde, it is not treating it simply as $HOME.

Bash Manual - Tilde-Expansion

rhubarb-geek-nz

Thanks, rhubarb-geek-nz - this has all been covered by my previous comment.

MartinGC94: good point about the provider-specific nature of ~.

However, tab-completion was previously already provider-aware and would not expand ~ while the current location was, say, the Env: drive (for which a home location isn't defined) - that shouldn't change (or, rather: categorically leave ~ alone for all other providers).

(On a side note: On Unix-like system, when tilde expansion happens in the context of argument-passing to external programs, the current location's provider does not matter; it is always the value of $HOME that is used - which makes sense, given that the outside world only knows the file-system, and nothing about PowerShell's other providers.)

Given that among the built-in providers only the FileSystem one has a home location predefined, the slight loss of level of abstraction that comes from translating ~ to $HOME - if the FileSystem provider underlies the current location (which, however, is almost always the case, if I were to guess) - is still preferable to expanding to the full, literal home directory path.

To me, translating ~ to verbatim $HOME would be no more confusing than the old behavior of expanding to, say, C:\Users\jdoe.

That said, we could avoid this problem if we went with proposal 3, which is the cleanest solution.
Personally, that is my preference, but the question is whether the concerns raised previously prevent it.

Paul Moore (@pfmoore)

Paul Moore (@pfmoore): MartinGC94 has posted a temporary fix in the form of a modified TabExpansion2 function, which you can add to your $PROFILE: expand the Details section in the following comment: #20750 (comment)

Sorry, everybody, but I'm closing this issue. I've been nearly half a year with tons of discussion (and way too many heated words), and never once has anybody from the PowerShell team come to say anything about this.

It's done and over, and we're not getting it back.

Brad Wilson (@bradwilson)

📣 Hey Brad Wilson (@bradwilson), how did we do? We would love to hear your feedback with the link below! 🗣️

🔗 https://aka.ms/PSRepoFeedback

Microsoft Forms

Brad Wilson (@bradwilson)

Brad Wilson (@bradwilson), as much as I can relate to the I'm-gonna-take-my-ball-and-go-home sentiment, the community is better served by keeping this issue open.

The fact that the issue is labeled WG-NeedsReview presumably:

(a) would keep this issue open indefinitely (unless closed explicity)
and (b) at least implies that at some point there'll be an official adjudication.

In short: I encourage you to reopen this issue, if only to - for now - increase its visibility to future users.

You're welcome to open a new issue and even link back here for "history". I have zero confidence that this one will yield any attention. 5 months is long enough to wait for even a simple acknowledgement.

You're welcome to open a new issue and even link back here for "history". I have zero confidence that this one will yield any attention. 5 months is long enough to wait for even a simple acknowledgement.

Thanks for sticking with it as long as you did. I've been affected by other hot-button issues that the pwsh team have refused to acknowledge. This is nothing new for this team.

fwiw, I just downgraded to https://github.com/PowerShell/PowerShell/releases/tag/v7.3.12

Sorry, everybody, but I'm closing this issue. I've been nearly half a year with tons of discussion (and way too many heated words), and never once has anybody from the PowerShell team come to say anything about this.

Sadly that's the way the PowerShell team has gone. Not engaging, and indulging in "Developer Driven Development".

Brad Wilson (@bradwilson)

📣 Hey Brad Wilson (@bradwilson), how did we do? We would love to hear your feedback with the link below! 🗣️

🔗 https://aka.ms/PSRepoFeedback

Microsoft Forms

Steve Lee (@SteveL-MSFT)

Thank you, thank you, thank you Steve Lee (@SteveL-MSFT) for the fix! And thanks to Jared Parsons (@jaredpar) for getting it attention.

That gif is amazing.

And just when you give up, someone engages and does the right thing :-) Tip-of-the-hat to you Steve.

Navigation Menu

Repository navigation

7.4.0 Breaking Change: Tab expansion no longer converts tildes to full paths, breaking path passing to executables on Windows #20750

Prerequisites

Steps to reproduce

Expected behavior

Actual behavior

Error details

Environment data

Visuals

114 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

7.4.0 Breaking Change: Tab expansion no longer converts tildes to full paths, breaking path passing to executables on Windows #20750

Description

Prerequisites

Steps to reproduce

Expected behavior

Actual behavior

Error details

Environment data

Visuals

Activity

237dmitry commented on Nov 22, 2023

bradwilson commented on Nov 22, 2023

237dmitry commented on Nov 22, 2023

bradwilson commented on Nov 22, 2023

bradwilson commented on Nov 22, 2023

237dmitry commented on Nov 22, 2023

bradwilson commented on Nov 22, 2023

MartinGC94 commented on Nov 22, 2023

mklement0 commented on Nov 22, 2023

mklement0 commented on Nov 22, 2023

Jaykul commented on Nov 23, 2023

MartinGC94 commented on Nov 23, 2023

rhubarb-geek-nz commented on Nov 23, 2023

mklement0 commented on Nov 23, 2023

114 remaining items

mklement0 commented on Apr 23, 2024

bradwilson commented on Apr 24, 2024

microsoft-github-policy-service commented on Apr 24, 2024

mklement0 commented on Apr 24, 2024

bradwilson commented on Apr 24, 2024

Gonkers commented on Apr 24, 2024

jhoneill commented on Apr 24, 2024

microsoft-github-policy-service commented on Apr 24, 2024

bradwilson commented on Apr 24, 2024

jaredpar commented on Apr 24, 2024

jhoneill commented on Apr 25, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions