Repository navigation
Add warning to web cmdlets if it defaults to HTTP #19316
Description
Summary of the new feature / enhancement
In WebRequestPSCmdlet.Common.cs line 1399 if we provide a -Uri with no scheme it defaults to http, is this the desired behaviour in 2023? Discussed with Ilya (@iSazonov) in #19162 comments
private static Uri CheckProtocol(Uri uri)
{
ArgumentNullException.ThrowIfNull(uri);
return uri.IsAbsoluteUri ? uri : new Uri("http://" + uri.OriginalString);
}Test:
$x = Invoke-WebRequest google.com
$x.BaseResponse.RequestMessage.RequestUri.Scheme
#--> httpProposed technical implementation details (optional)
http --> https
Reacted by Michael Klement
Activity
iSazonov commentedon Mar 13, 2023
HTTPS is preferred as default in modern world.
But it can be a breaking change.
Perhaps we could try HTTPS first and then HTTP - since the user didn't specify the exact protocol, this is probably acceptable, but I'd prefer not to.
jhoneill commentedon Mar 13, 2023
We have conflict between modern world assumptions and things that existing scripts may rely on, so I think there must be a fall back to http to avoid breaking scripts.
Even then a new script for a server which does not allow http which relies on the assumption of https would work on updated PowerShell and fail on older versions (e.g. Windows PowerShell). Documentation must be very clear that the behaviour has changed and if you require https AND can't rely on the script running on a recent version then you must specify it explicitly.
I'll bring this to the cmdlets working group and see what the view is.
iSazonov commentedon Mar 14, 2023
I suppose that specifying a URL without a protocol is too careless and it is very unlikely that someone does that in a working script, rather only on the command line which is not critical. Then we can conclude that switching from HTTPS to HTTP is unnecessary, even harmful.
jhoneill commentedon Mar 14, 2023
I'm not sure what is safe to conclude.
Generally if something can be called in a particular way someone, somewhere will call it that way.
It's good practice to write
httporhttpsexplicitly and everyone SHOULD do that, but not everyone WILL.If the service supports https and http then using https is preferable.
But if the service doesn't support https, then (a) should the command line short cut stop working? and (b) is it acceptable to break the small number of scripts that contain that shortcut ?
I think the answer to both is no, and the expectation is things work like browsers work - try https first and only if that fails try http. (There should probably be a warning message if the non secure version is used but not explicitly specified).
iSazonov commentedon Mar 14, 2023
There is a security aspect to this fallback. As long as the default is HTTP this is not an issue, but if the default is HTTPS and we add a fallback to HTTP, then there is a security issue.
So it must either be rejected altogether or be bound to the AllowInsecureRedirect parameter, or it must be approved by the WG-Security.
But it all looks unnecessarily complicated.
I wouldn't overcomplicate it, just make HTTPS the default and expect feedback. If users do run into a problem, but then we will have more information to make things more right.
SteveL-MSFT commentedon Mar 15, 2023
@PowerShell/wg-powershell-cmdlets reviewed this and concerned about the impact of a breaking change. We looked at curl as a precedent and they also default to http. We suggest adding a warning message if no scheme is specified and HTTP is being used.
[-]Should Invoke-WebRequest default to http or https?[/-][+]Add warning to web cmdlets if it defaults to HTTP[/+]iSazonov commentedon Mar 16, 2023
curl has very old history and it is unlikely that they will change anything there.
jhoneill commentedon Mar 22, 2023
rhubarb-geek-nz one could. However that would break existing scripts.
3 remaining items
jhoneill commentedon Apr 14, 2023
CarloToso I think that change of plan needs wider discussion, it was set as up for grabs because
Therefore a warning that says "you didn't specify http or https, so for legacy reasons I'm using http even though the server might support https" was seen as the best solution.
It could break a script which runs with
-warningAction stopbut that's rare and unlikely to combine with a naked URL.Annoying people who don't write their URLs properly was understood when it was set as up for grabs.
iSazonov commentedon Apr 14, 2023
Can you demonstrate such scripts?
jhoneill commentedon Apr 15, 2023
I have seen them at 2 of the last 3 clients. My last long contract included enabling HTTPS on a bunch of internal servers and the about half the places I needed to change http to https and half I was adding https where there had been nothing. 2 clients before that had some of the oddest PowerShell I've ever seen (their main scripter had never heard of "if" and used switch for everything) They included server names as constants at the start of some scripts sometimes they would build a unc as a "\$name\path" and sometimes a url as "$name/api_path", some cases did have http:// in them, others didn't
I wrote this https://github.com/jhoneill/OctopusTools/blob/main/Public/Connect-Octopus.ps1 on the recent engagement, and if the user supplies the URL as a command line parameter without the http, I don't add it. I'm fairly sure the octopus servers still didn't support https when I finished so a CI/CD pipeline which uses my code where someone has specified the server name instead of a URL would break.
I've said before I believe in the original formulation of Murphy's law "If it can be done wrongly, sooner or later someone will do it wrongly". The same is true for APIs (and command lines are APIs), if you allow something to be used in some fashion, someone will come to depend on its continued working in that fashion.
The view that this only breaks things for people whose practices were sloppy (omitting the protocol is sloppy) therefore it's OK, is dangerous. It sends the message that if the powers that be don't like how you write code they can break your past work, and therefore upgrades carry a risk, so staying on Windows PowerShell 5 forever is the smart thing to do - I have seen aversion to upgrading in many organisations over a long career.
iSazonov commentedon Apr 15, 2023
My question was very specific. A better answer would be to give examples of popular apps that would stop working, it would be hard to fix, it would degrade the UX, etc.
Right now all we have are statements of concern.
We were in the same situation when the question arose about changing the default encoding to Utf8 in pwsh. The situation was even more uncertain. What did we do? We released a preview and then a release version afterwards. It was the only way to move forward with the uncertainty.
The uncertainty of this issue is much less and the scope is much smaller since it only affects a couple of cmdlets. And I see no reason why we can't release a preview with this change. Name them if you know the specific reasons but not the general reasoning.
Another very strange point is that the concern expressed is about scripts (sloppy in nature), but not about the more obvious danger when the user uses these cmdlets in an interactive session and could leak credentials and other sensitive information. Obviously, we all often omit http/https when typing in the console. Then why shouldn't these cmdlets work in a secure way like their big brothers?
jhoneill commentedon Apr 16, 2023
Ilya (@iSazonov) "Right now all we have are statements of concern." Well yes.
Do we have any numbers for how many internal servers support / don't support HTTPS? I don't think so. Do we know how many scripts call http (only) services on those ? Doubt it. And of those scripts how many are sloppily written without the protocol. Complete unknown.
What I can tell you is I have worked in places where, if the default changed, they would either need to get their act together and ensure all services could be accessed over https OR they must review every script which uses these cmdlets to (1) ensure that everywhere any parameter passed in is checked and their script adds http if it is not specified, and (2) fix hard-coded urls OR Simply not deploy the version concerned. Realistically they have enough other work to do that they will simply not deploy. You would put me in the position of telling my clients not to deploy releases after X.
As for UTF-8 - if there are text files which are written with codepage 1252 - which is what PowerShell 5 assumes with files without a byte-order-mark, which a script needs to read, or other tools which read correctly if the file is UTF-16 but assume another code page with no bom , then it is possible to set encoding as a parameter default and work around the problem.
Working round the problem being a parameter which has become necessary is easy; working round a parameter that has become wrong needs a Proxy function to be deployed which fixes them.
The reason for printing a warning was precisely to say to sloppy cmdline users "hey if you're going through an insecure network to a server that supports https, don't write it like that". curl doesn't work in a secure way and the model is to provide native functionality which is like curl (which is now pre-installed on Windows) . If users find this annoying which was the reason cited for not giving a warning, they can set a default parameter for warningaction in these cmdlets.
As Steve posted above the cmdlets wg decided that a warning was the least bad of a set of imperfect choices. It was not appropriate for CarloToso to change this back to "not planned" without at least asking the steve or the wg to reconsider.
microsoft-github-policy-service commentedon Nov 15, 2023
This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.
microsoft-github-policy-service commentedon Nov 15, 2023
This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.
microsoft-github-policy-service commentedon Nov 15, 2023
This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.
microsoft-github-policy-service commentedon Nov 22, 2023
This issue has been marked as "No Activity" as there has been no activity for 6 months. It has been closed for housekeeping purposes.