Repository navigation
Insecure Proxy Behavior with Invoke-(RestMethod|WebRequest) on 7.4.0-preview2 #19461
Description
Prerequisites
- Write a descriptive title.
- Make sure you are able to repro it on the latest released version
- Search the existing issues.
- Refer to the FAQ.
- Refer to Differences between Windows PowerShell 5.1 and PowerShell.
Steps to reproduce
I use a number of script-based PWSH modules. In addition to using Invoke-RestMethod for a lot of API calls. These calls/modules are executed in a corporate environment with insecure/zero-trust proxies all over the place. The proxies cannot be avoided.
When testing PWSH v7.4.0-preview2, all script-based modules and IRM/IWR throw an error.
Eg. https://github.com/dell/OpenManage-PowerShell-Modules
I believe this is part of PR #18595. It seems that in certain scenarios, the change is breaking.
Adding the following to script-based modules and repos would be a large effort with external modules being out of my control. Unclear of best approach to mitigate in order to work before/after the 7.4.0 threshold.
if ($psVersionTable.PSVersion.Major -ge 7 -and $psVersionTable.PSVersion.Minor -ge 4) {
$querySplat.Add('AllowInsecureRedirect',$true)
}
Expected behavior
[2881.31]ms GitHub> # Works before 7.4.0-preview2
$querySplat = @{
Method = 'POST'
URI = $URI
Body = $bodyJSON
SkipCertificateCheck = $true
ContentType = 'application/json'
}
Invoke-RestMethod @querySplat
Id : a1bf17dd-d590-4929-8f0a-7357bed828f8
Description : abcde.ab@OU=Groups,DC=test,DC=Contosa,DC=com
Name : API
...Actual behavior
[2881.31]ms GitHub> # Works before 7.4.0-preview2
$querySplat = @{
Method = 'POST'
URI = $URI
Body = $bodyJSON
SkipCertificateCheck = $true
ContentType = 'application/json'
}
Invoke-RestMethod @querySplat
Invoke-RestMethod:
Line |
9 | Invoke-RestMethod @querySplat
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| This operation is not supported for a relative URI.
### Error details
```console
Exception :
Type : System.InvalidOperationException
TargetSite :
Name : get_Scheme
DeclaringType : uri
MemberType : Method
Module : System.Private.Uri.dll
Message : This operation is not supported for a relative URI.
Source : System.Private.Uri
HResult : -2146233079
StackTrace :
at System.Uri.get_Scheme()
at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.ProcessRecord()
at System.Management.Automation.CommandProcessor.ProcessRecord()
CategoryInfo : NotSpecified: (:) [Invoke-RestMethod], InvalidOperationException
FullyQualifiedErrorId : System.InvalidOperationException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand
InvocationInfo :
MyCommand : Invoke-RestMethod
ScriptLineNumber : 8
OffsetInLine : 1
HistoryId : 8
Line : Invoke-RestMethod @querySplat
Statement : Invoke-RestMethod @querySplat
PositionMessage : At line:8 char:1
+ Invoke-RestMethod @querySplat
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
InvocationName : Invoke-RestMethod
CommandOrigin : Internal
ScriptStackTrace : at <ScriptBlock>, <No file>: line 8
Environment data
Name Value
---- -----
PSVersion 7.4.0-preview.2
PSEdition Core
GitCommitId 7.4.0-preview.2
OS Microsoft Windows 10.0.19045
Platform Win32NT
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0Visuals
No response
Activity
CarloToso commentedon Apr 6, 2023
Andy (@crlogic) Does this work if you use -AllowInsecureRedirect? (if it does, probably there's a bug and I'll try to fix it ASAP)
crlogic commentedon Apr 6, 2023
Affirmative. Adding
-AllowInsecureRedirectworks on 7.4.0-preview2 but that has to be conditionally applied since below that version (of which I only really tested 7.3.x) doesn't have the param.CarloToso commentedon Apr 6, 2023
Could you post the
$URI?The error originates here:
response.RequestMessage.RequestUri.Scheme == "https" && response.Headers.Location?.Scheme == "http"It tries to get the scheme from a relative uri (like
www.google.com) and fails to gethttp or httpsbecause they are missing.crlogic commentedon Apr 6, 2023
Here you go :)
crlogic commentedon Apr 6, 2023
I live in proxy central (Axis/ZTNA/others) where all sort of mangling likely occurs.
CarloToso commentedon Apr 6, 2023
Steve Lee (@SteveL-MSFT) Ilya (@iSazonov) Maybe we should just revert #18595, if a proxy mangles the url checking for the scheme is useless and we can't determine if a redirect would be insecure. In this case
-AllowInsecureRedirectshould not be needed, it only skips che check and the error related toget_Scheme()iSazonov commentedon Apr 7, 2023
A silent redirect from HTTPS to HTTP is a potential security hole even on the internal network.
I don't think PowerShell should encourage insecure scripts and insecure ways to build applications and environments.
CarloToso commentedon Apr 7, 2023
A silent redirect from https to http is impossible without the -AllowInsecureRedirect parameter, it's not supported by System.Net.Http.HttpClient. Before 7.4 preview it failed silently. I added an error to check if there was an insecure redirect and help the discovery of the new parameter, unfortunately it looks for the Scheme property and fails if the url is relative.
I see two options now:
crlogic commentedon Apr 7, 2023
Understandable..
I don't feel like this is something that PowerShell has any say in; hear me out..
Eg. As an end-user that is mandated to leverage the "security policies" of the corporation, I have no say in those policies and the argument that "PowerShell will no longer work" will merely receive a response from the "security team" that "I should use better tools."
I should have to ability to make the decision to "write an insecure script because I have been mandated to do so". After all, the "security team mandates I do it that way." Even if they are wrong from an ideological perspective.
I don't know how these zero-trust tools/tunnels/redirects work which doesn't help my case. All I know is that, "it works in pwsh 7.3.3 and below" and a lot of tools will break if the preview2 behavior is maintained.
Perhaps a third option is a warning and continuing?
Eg. VMware PowerCLI outputs a warning if deprecated cmdlets/parameters are used.
doctordns commentedon Apr 7, 2023
The problem I see with the proposed 7.,4 behavior is that it introduces a breaking change. A script that runs in 7.3 now fails. While the script could be fixed in 7.4.,x (ie add the
-AllowInsecureRedirectparameter) but then that script no longer works in 7.3.I agree that issuing a warning and continuing may be a better approach.
matsmcp commentedon Apr 9, 2023
+1
As it stands -we need the SSH options enhancement that is in 7.3 and we want it in a LTS release (aka 7.4). We do not want breaking changes preventing a smooth upgrade from 7.3 -> 7.4
doctordns commentedon Apr 9, 2023
matsmcp Unless I am mistaken, the SSH option enhancements you mention in 7.3 should be in 7.4.
I have not tested it, but what happens when this code is on PowerShell 7.2? Does it work the same way as in 7.3? If so. this also be a breaking change for 7.,2 to 7.,4 upgrades (ie LTS to LTS)? I would not be in favour of adding more breaking changes if it can be avoided (and it feels like it can). Especially in an LTS to LTS update.
crlogic commentedon Apr 9, 2023
This code has gone through the lineage of 7.0 through latest 7.3. I didn't go all the way back in specific testing, here is 7.2.10
Eg. 1
Eg. 2
crlogic commentedon Apr 10, 2023
Adding more detail should the PR be revisited in the future.
Further investigation into the symptoms revealed that my focus on tunnel/proxy/fw may have been pre-mature. I primarily work remotely via tunnels that require 10+ hops to the API endpoint. And the way Axis/ZTNA appear to work is that they highjack IP/name-resolution as any type of
Resolve-DNSNamedoes not return true address information. Therefore I RDP'd directly to the local (L2) subnet of a suspect API endpoint to repeat the testing above and received the same results.This made me wonder if it was actually the API doing something funny so I tested a half-dozen other API's and found some that actually did work w/ 7.4.0-preview2. And some that always failed, ala Dell iDrac & Dell OME, regardless of execution proximity to endpoint.
matsmcp commentedon Apr 10, 2023
Yes it should be but if we get a breaking change like the one being discussed here, in worst case we cant upgrade meaning we are stuck on 7.3 (at least until the scripts can be fixed). I guess you could do an if on the version and adjust the call based on that but - thanks - no thanks ;)
daxian-dbw commentedon Apr 11, 2023
Andy (@crlogic) Now that CarloToso has a potential fix #19468, can you please try out the private build produced from that PR and confirm whether it actually works for your scenario?
You can download the build.zip package from https://dev.azure.com/powershell/PowerShell/_build/results?buildId=127283&view=artifacts&pathAsName=false&type=publishedArtifacts
After you download, please remember to unblock the build.zip file before unzip it, otherwise you won't be able to start the pwsh.exe:
crlogic commentedon Apr 11, 2023
Dongbo Wang (@daxian-dbw) - three good test results below.
using DellOpenMange module which is a script-based module around
Invoke-WebRequestInvoke-RestMethodto same Dell OpenManage APIInvoke-RestMethodto a Dell iDrac Redfish APIdaxian-dbw commentedon Apr 11, 2023
Andy (@crlogic) Thank you for confirming the fix works! Appreciate your help.