Repository navigation
Implement WebCdmlets to save response content to filename specified in Content-Disposition #6618
Description
Currently the WebCmdlets only allow to save the content of a response via -OutFile to a filename specified when invoking the Cmdlets. As many services are responding with the Content-Disposition header, which contains details of the response content, including a filename, it would be good if the WebCmdlets would allow to save a file to a path specified e.g. via a parameter -OutPath but using the filename from the Content-Disposition header. This would be similar to the behaviour of curl, wget or Web Browsers.
Reacted by Mr Beedell, Roke Julian Lockhart
Activity
markekraus commentedon Apr 11, 2018
I do not want to overload
Invoke-WebRequestandInvoke-RestMethodwith download specific functionality. Instead, I would like to consider this functionality in the design of the new Download Specific cmdlet.Please comment on PowerShell/PowerShell-RFC#124
The current plan doesn't include using Content-Disposition, but I'm open to guidance on how to incorporate that for HTTP/HTTPS.
ffeldhaus commentedon Apr 12, 2018
Mark Kraus (@markekraus) I like the suggestion in the RFC you mentioned, but in my opinion what you are suggesting there is something like a download utility similar to
wgetor a download manager (e.g. something like CloudBerry, Cyberduck or others).In my opinion the WebCmdlets are about handling Web Requests and their responses. The response to a POST, PUT, GET or DELETE request can contain content and the
Content-Dispositionheader is adding information about the content such as a filename which should be used to save the content. A download utility, as you suggested in the RFC, should be targeted at downloading files (e.g. via a HTTP GET request) and not handling responses of POST, PUT, GET or DELETE requests, which may contain files in their responses.As the WebCmdlets already include functionality to save the content to a file via the
-OutFileparameter, I don't think we are overloading any functionality here.I noted that
-OutFileaccepts String as input, and only the documentation states that the parameter should either be path plus filename or only filename (and then$PWDwill be used as path).Therefore my suggestion is, to extend the current behaviour and documentation:
-OutFileparameter to contain either a path OR a path and a filename OR only a filenameContent-Dispositionheader if available. If not available, the filename will be the last part of the URL without query parameters (e.g. for https://example.org/path/to/file.txt it would befile.txt)This would bring the WebCmdlets closer to the
curlbehavior and would meet what several users are expecting from the WebCmdlets when using the-OutFileparameter (see discussions on StackOverflow, Reddit.This is an example how curl handles the filename of the
Content-Dispositionheader (parameter-Ois required to save to file and-Jto use extract the filename from theContent-Dispositionheader instead from the URL:This page has several examples for testing the
Content-Dispositionheader.SteveL-MSFT commentedon Apr 12, 2018
Rather than overload
-OutFile, it may be better to have a-OutPathparameter that only takes a directory path.markekraus commentedon Apr 13, 2018
I'm an 100% against adding this.
There are way too many download specific requests for these cmdlets that are better served on the new cmdlet. The code for dealing with downloads in the web cmdlets is not ideal because all of the value of these cmdlets is focused on proccessing web responses, not downloaded content.
Cmdlets and functions should "do one thing". The web cmdlets already provide a download feature as a convenince, and perhaps that was a mistake in the initial design.. Any further download features should be focued on the new download specific cmdlets.
markekraus commentedon Apr 13, 2018
Also. the existing web cmdlets do not need to reach full parity with curl. PowerShell does. curl is overloaded and and that caries with it a ton of baggage.
processing web content, downloading binary data (this includes text downloads), and processing web streams are 3 distinct tasks. Yes, they are sharing a common protocol, HTTP. But that protocol is overloaded with a bunch of different functionalities.
PowerShell can and should accommodate those functions. But I see no compelling reason why they should all be done in the existing cmdlets. It makes them too complex and difficult to change. Adding more download features makes it harder to make feature changes that deal with processing web content, and adding features to process web content makes it harder to deal with downloads.
We can avoid that coupling behavior by using separate cmdlets.
markekraus commentedon Apr 13, 2018
Steve Lee (@SteveL-MSFT) Even with that, we now have to add logic in these cmdlets to deal with naming when content-distposition isn't available, we have to figure out the behavior when a file with the chosen name already exists. Do we overwrite? Do we error out? fdo we add an
-AllowClobber? We have to now complicate the resume feature and extend run time checks to include more mutex params. It's just too complex and too restricting for too little gain, IMO. With the use of-PassThruor-ResponseHeadersVariablethe user has access to the content-dispostiion header and can rename the file if they wish.SteveL-MSFT commentedon Apr 13, 2018
Mark Kraus (@markekraus) I'm with you 100% that we want full parity with curl, but that doesn't mean everything jammed into just one or two cmdlets.
ffeldhaus commentedon Apr 13, 2018
Mark Kraus (@markekraus) I don't understand why you insist that handling attachments is something which needs to be done in a different Cmdlet. This issue is not about downloading a file. It is about extracting the filename for the response content from the
Content-Dispositionheader, which is considerably different. A user may need to issue a complex Web Request with HTTP Header, Body and a Method different than GET (most often POST, but also PUT or DELETE) and where the response happens to contain an attachment which should be saved locally. As-OutFilealready exists as a parameter, this issue is only about allowing to use the filename provided in the response instead of specifying it upfront. This issue is not about adding download functionality to WebCmdlets.The Cmdlet you proposed in the RFC is about downloading (e.g. retrieving) a file. I agree, that functionality specific to downloading files could (and probably should) be handled in a different Cmdlet and that also includes extracting the filename from the
Content-Dispositionheader. There are some features which would really bloat the existing WebCmdlets like multipart downloads (e.g. via multiple parallel HTTP Range Requests), Async IO for saving files, resumable downloads, protocols other than HTTP and probably several more features. Saving an attachment in resposne to a complex Web Request is not something which requires a different Cmdlet in my opinion, as it would bloat the Download Cmdlet and require it to allow specifying HTTP Header, Body and different HTTP Methods which are what the existing WebCmdlets seem to be designed for. The Download Cmdlet could reuse the existing WebCmdlets and enhance them with additional features for downloading files.There is a good reason, that both
wgetandcurlexist and are both used in different uses cases.wgethandles downloading of files, similar to what you suggest.curlhandles Web Requests and their responses.I'd be glad if others would add their view here. It may also be a good idea to exactly specify what the community (and the PowerShell Committee) want to achieve with the existing WebCmdlets and what they do not want to achieve with them, maybe even through dedicated RFCs for the WebCmdlets.
markekraus commentedon Apr 14, 2018
I know I sound like a broken record, but: All of the value of the web cmdlets derives from their ability process web content. That is to say, doing meaningful things with HTML, XML, JSON, and plain-text. Their code is geared towards it. Adding features to support downloads or "attachments" or any other way you want to paint them is a completely different functionality. That is why.
You can already rename the file after the fact by inspecting the content disposition header on the response. So this lack of convenience is non-blocking. But, adding this feature is a non-trivial amount of work that requires limiting the cmdlets' flexibility to extend and improve features relevant to their core functionality.
If you think you may need POST functionality in order to retrieve remote files, please comment on the RFC.
markekraus commentedon Apr 17, 2018
And that is a design decision that has not served us well.
markekraus commentedon Jan 24, 2019
I'm closing this as the conversation has gone stale and there is decent enough consensus and reason not to implement this in the existing cmdlets.
mklement0 commentedon Jan 24, 2020
With the recent addition of #11671, we now have 3 distinct requests to implement the same thing.
I find @ffeldhaus proposal and reasoning convincing: this is not about "feature creep" or "responsibility creep", this is about a simple, non-breaking enhancement to an existing feature that has great value.
I suggest we revisit this.
mklement0 commentedon Jan 27, 2020
If someone is willing to take this on: Implementing it as an experimental feature has now been green-lighted: #11671 (comment)