Web cmdlets: support compression

I have noticed that there currently appears to be no support for requesting a compressed response (gzip/deflate) with the Invoke-[RestMethod|WebRequest] cmdlets. It would be useful to have support for this (curl implements this via the --compressed flag).

Conditional support can be added via the AutomaticDecompression property of the WebRequest class for FullCLR, and using a new HttpClientHandler with HttpClient for CoreCLR. Support would be added with a new cmdlet parameter (suggest -Compressed ?)

I'm happy to put a pull request in for this - however just making sure this functionality is desirable before adding.

The web cmdlets should support gzip compressed requests.

Following commands work in PS5/6 on Windows:

Invoke-RestMethod httpbin.org/gzip

gzipped headers
------- -------
   True @{Host=httpbin.org; User-Agent=Mozilla/5.0 (Windows NT; Windows NT 10.0; en-US) WindowsPowerShell/5.1.14393....

(Invoke-WebRequest httpbin.org/gzip).Content

{
  "gzipped": true,
  "headers": {
    "Host": "httpbin.org",
    "User-Agent": "Mozilla/5.0 (Windows NT; Windows NT 10.0; en-US) WindowsPowerShell/5.1.14393.693"
  },
  "method": "GET",
  "origin": "REDACTED"
}

However deflate is not supported:

Invoke-RestMethod httpbin.org/deflate
Invoke-WebRequest httpbin.org/deflate

PS5 errors

Invoke-RestMethod : Block length does not match with its complement.
At line:1 char:1
+ Invoke-RestMethod httpbin.org/deflate
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-RestMethod], InvalidDataException
    + FullyQualifiedErrorId : System.IO.InvalidDataException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Invoke-WebRequest : Block length does not match with its complement.
At line:1 char:1
+ Invoke-WebRequest httpbin.org/deflate
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], InvalidDataException
    + FullyQualifiedErrorId : System.IO.InvalidDataException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

PS6 errors

Invoke-RestMethod : The read operation failed, see inner exception.
At line:1 char:1
+ Invoke-RestMethod httpbin.org/deflate
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-RestMethod], IOExcepti
   on
    + FullyQualifiedErrorId : System.IO.IOException,Microsoft.PowerShell.Comma
   nds.InvokeRestMethodCommand

Invoke-WebRequest : The read operation failed, see inner exception.
At line:1 char:1
+ Invoke-WebRequest httpbin.org/deflate
+  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], IOExcepti
   on
    + FullyQualifiedErrorId : System.IO.IOException,Microsoft.PowerShell.Comma
   nds.InvokeWebRequestCommand

I just verified, that PS5/6 indeed doesn't send Accept-Encoding header. My initial test with Requestb.in was stating wrong that Accept-Encoding: gzip is included in the header. Seems to be a bug on Requestb.in side.
Another test with http://xhaus.com/headers and http://www.reliply.org/tools/requestheaders.php showed that Accept-Encoding is missing.
Furthermore http://httpbin.org/gzip always delivers gzip compressed data, regardless if the header is included or not. Another strange behavior in my opinion.

Anyway, what seems to work is:

Invoke-WebRequest www.google.com -Headers @{"Accept-Encoding"="gzip"}

This returns a gzip compressed version of the website.

I would simply suggest to always include the Accept-Encoding header for all supported formats without the need of a specific parameter. If the client (PowerShell) and the server both support the compression they should use it transparently.

So to clarify, you verified that we do already decompress gzip'd content? You just want to default to requesting compressed content in the header?

Sounds right to me, but I don't know enough to know if there's a downside to that default. Only thing I can think of is a little extra server-side compute, but it's probably worth the bandwidth tradeoff. And certainly none of that should affect the client, right?

Yes, gzip and even deflate response streams are supported!

Streamhelper.cs Line 472

#if CORECLR

        internal static Stream GetResponseStream(HttpResponseMessage response)

        {

            Stream responseStream = response.Content.ReadAsStreamAsync().GetAwaiter().GetResult();

            var contentEncoding = response.Content.Headers.ContentEncoding;



            // HttpClient by default will automatically decompress GZip and Deflate content.

            // We keep this decompression logic here just in case.

            if (contentEncoding != null && contentEncoding.Count > 0)

            {

                if (contentEncoding.Contains("gzip"))

                {

                    responseStream = new GZipStream(responseStream, CompressionMode.Decompress);

                }

                else if (contentEncoding.Contains("deflate"))

                {

                    responseStream = new DeflateStream(responseStream, CompressionMode.Decompress);

                }

            }



            return responseStream;

        }

#else

        internal static Stream GetResponseStream(WebResponse response)

        {

            Stream responseStream = response.GetResponseStream();



            // See if it had a content-encoding, wrap in a decoding stream if so.

            string contentEncoding = response.Headers["Content-Encoding"];

            if (contentEncoding != null)

            {

                if (contentEncoding.IndexOf("gzip", StringComparison.OrdinalIgnoreCase) >= 0)

                {

                    responseStream = new GZipStream(responseStream, CompressionMode.Decompress);

                }

                else if (contentEncoding.IndexOf("deflate", StringComparison.OrdinalIgnoreCase) >= 0)

                {

                    responseStream = new DeflateStream(responseStream, CompressionMode.Decompress);

                }

            }



            return responseStream;

        }

#endif

Honestly I have no idea, why the deflate decompression doesn't work, since I don't have a Debug setup here on my laptop.

Regarding the default headers: Here is the W3 RFC

14.3 Accept-Encoding
The Accept-Encoding request-header field is similar to Accept, but restricts the content-codings (section 3.5) that are acceptable in the response. 
       Accept-Encoding  = "Accept-Encoding" ":"
                          1#( codings [ ";" "q" "=" qvalue ] )
       codings          = ( content-coding | "*" )
Examples of its use are: 
       Accept-Encoding: compress, gzip
       Accept-Encoding:
       Accept-Encoding: *
       Accept-Encoding: compress;q=0.5, gzip;q=1.0
       Accept-Encoding: gzip;q=1.0, identity; q=0.5, *;q=0
A server tests whether a content-coding is acceptable, according to an Accept-Encoding field, using these rules: 
      1. If the content-coding is one of the content-codings listed in
         the Accept-Encoding field, then it is acceptable, unless it is
         accompanied by a qvalue of 0. (As defined in section 3.9, a
         qvalue of 0 means "not acceptable.")
      2. The special "*" symbol in an Accept-Encoding field matches any
         available content-coding not explicitly listed in the header
         field.
      3. If multiple content-codings are acceptable, then the acceptable
         content-coding with the highest non-zero qvalue is preferred.
      4. The "identity" content-coding is always acceptable, unless
         specifically refused because the Accept-Encoding field includes
         "identity;q=0", or because the field includes "*;q=0" and does
         not explicitly include the "identity" content-coding. If the
         Accept-Encoding field-value is empty, then only the "identity"
         encoding is acceptable.
If an Accept-Encoding field is present in a request, and if the server cannot send a response which is acceptable according to the Accept-Encoding header, then the server SHOULD send an error response with the 406 (Not Acceptable) status code. 
If no Accept-Encoding field is present in a request, the server MAY assume that the client will accept any content coding. In this case, if "identity" is one of the available content-codings, then the server SHOULD use the "identity" content-coding, unless it has additional information that a different content-coding is meaningful to the client. 
      Note: If the request does not include an Accept-Encoding field,
      and if the "identity" content-coding is unavailable, then
      content-codings commonly understood by HTTP/1.0 clients (i.e.,
      "gzip" and "compress") are preferred; some older clients
      improperly display messages sent with other content-codings.  The
      server might also make this decision based on information about
      the particular user-agent or client.
      Note: Most HTTP/1.0 applications do not recognize or obey qvalues
      associated with content-codings. This means that qvalues will not
      work and are not permitted with x-gzip or x-compress.

Maybe we should open an issue fixing the deflate decompression bug and another issue for the default headers?

ThreeFive-O

I wrongly presumed this to not support automatic decompression (was looking at reference source) - thanks for clarifying ThreeFive-O

I will take a look at the deflate issue later, think it's definitely worth a new git issue

I've been able to set up a Debug environment to search for the deflate decompression problem.

Fun fact: The above StreamHelper class I referenced is totally unnecessary and always gets skipped by the code, since the contentEncoding variable doesn't contain gzip or deflate.

Found this instead in WebRequestPSCmdlet.CoreClr.cs for CoreCLR:

        internal virtual HttpClient GetHttpClient()
        {
            // By default the HttpClientHandler will automatically decompress GZip and Deflate content

Hence, there is no need for the StreamHelper class to check for gzip or deflate, since the HttpClient for CoreCLR should do it automatically. On FullCLR the WebRequest or HttpWebRequest class probably does the same thing (not verified).
I guess the problem with deflate decompression is on those .NET classes.

Any ideas how to tackle the problem?

According to #3046 there are many bug fixes planned for the web cmdlets. Maybe some cleanup of the code would be good, if all those problems are getting fixed.

UPDATE: One more hint about the implementation. HttpWebRequest (which seems to be preferable over HttpClient) is supported by .NET CoreFX. The CoreCLR part of WebRequestPSCmdlet should probably share as much code as possible with FullCLR (Less code maintenance and better testability). Just in case someone needs it for bug-fixing.

ThreeFive-O

ThreeFive-O HttpWebRequest is still not in CoreCLR.

Well, I couldn't find any information on the .NET Standard where it is supported. I only checked the master branch of CoreFX: https://github.com/dotnet/corefx/blob/master/src/System.Net.Requests/src/System/Net/HttpWebRequest.cs

Maybe as part of .NET Standard 2.0? I guess PS is on 1.6?

Our plan is to move to .NET Core vNext that supports .NET Standard 2.0 #3046
.Net Core RoadMap https://github.com/dotnet/core/blob/master/roadmap.md

Yes, .Net Core 1.0 = .Net Standart 1.6
Full version table https://docs.microsoft.com/en-us/dotnet/articles/standard/library

Ilya (@iSazonov)

Ilya (@iSazonov) Thank you for clarification. Should I open an issue for moving from HttpClient to HttpWebRequest? Or should I wait until PS moved to .NET vNext?

lee303 Would you please submit a PR which fixes the Accept-Encoding problem with the solution you initially stated (AutomaticDecompression and HttpClientHandler)? I'm glad to assist you debug the problem, why HttpClientHanlder doesn't send the correct header by default as stated in the source comment. Do you also want to open a separate issue for the deflate decompression problem or do you want to fix it under this issue?

Just found that deflate problem is already known for Invoke-WebRequest #1753

Steve Lee (@SteveL-MSFT)

.Net Core new version release date is very optimistic. I believe that we should wait.
It would be useful to know the opinion of Powershell team.
/cc Steve Lee (@SteveL-MSFT)

It looks like there's been some movement here. When issuing this command in PS6:

Invoke-RestMethod https://httpbin.org/deflate

You get this error now:

Invoke-RestMethod : The archive entry was compressed using an unsupported compression method.
At line:1 char:1
+ Invoke-RestMethod https://httpbin.org/deflate -Headers @{'accept-encoding' =  ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : NotSpecified: (:) [Invoke-RestMethod], InvalidDataException
+ FullyQualifiedErrorId : System.IO.InvalidDataException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

Interestingly enough, if you try Brotli:

Invoke-RestMethod https://httpbin.org/brotli

IRM doesn't throw an error, but returns the raw Brotli-encoded response:

║��mF╚��>�&����0�� ~Q,����<�����̳���t�d
                                      L�w5�╔|�=�~38�B��O��X6��I�D�<�{!�)��r�UI�       +D╚>������H*$��'
                                                                                                      8

Seems like a decent workaround in the meantime is to manually send a Accept-Encoding header with gzip and identity in every request and hope that the server responds accordingly.

Brotli is not yet supported in HttpClient and thus not in the web cmdlets.

Deflate, however, is supported... but only per RFC. the reality is many sites (httpbin.org included) use a non-rfc compression for deflate.. and there isn't much we can do about that. If DeflateStream could support automatic detection and deflation that would solve our problems, but there's not much we can do at the higher level abstraction where the web cmdlets sit.

Would it make sense to have any response that is brotli encoded throw the same System.IO.InvalidDataException that deflate throws? At least until brotli is natively supported in HttpClient?

Can someone explain the difference between -Headers @{"Accept-Encoding"="gzip"} (that works with our server) and -TransferEncoding GZip (that doesn't, yielding 400 Bad Request)?

I think the compression is supported by new HttpClient from .Net 3+.

Stéphane BARIZIEN (@sba923)

Stéphane BARIZIEN (@sba923) commented on Jul 10, 2020

Can someone explain the difference between -Headers @{"Accept-Encoding"="gzip"} (that works with our server) and -TransferEncoding GZip (that doesn't, yielding 400 Bad Request)?

Accept-Encoding and Transfer-Encoding are two different headers:

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Encoding

The Accept-Encoding request HTTP header indicates the content encoding (usually a compression algorithm) that the client can understand. The server uses content negotiation to select one of the proposal and informs the client of that choice with the Content-Encoding response header.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding

Transfer-Encoding is a hop-by-hop header, that is applied to a message between two nodes, not to a resource itself. Each segment of a multi-node connection can use different Transfer-Encoding values. If you want to compress data over the whole connection, use the end-to-end Content-Encoding header instead.

This answer on StackOverflow goes into the differences between Content-Encoding (which is related to Accept-Encoding) and Transfer-Encoding a bit more.

Ilya (@iSazonov)

Ilya (@iSazonov) I think this issue was fixed by #17955

Navigation Menu

Repository navigation

Web cmdlets: support compression #3178

16 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Web cmdlets: support compression #3178

Description

Activity

ThreeFive-O commented on Feb 21, 2017

ThreeFive-O commented on Feb 21, 2017

joeyaiello commented on Feb 22, 2017

ThreeFive-O commented on Feb 22, 2017

0x4c6565 commented on Feb 22, 2017

ThreeFive-O commented on Feb 22, 2017

iSazonov commented on Feb 22, 2017

ThreeFive-O commented on Feb 22, 2017

iSazonov commented on Feb 22, 2017

ThreeFive-O commented on Feb 22, 2017

ThreeFive-O commented on Feb 22, 2017

iSazonov commented on Feb 22, 2017

16 remaining items

burkasaurusrex commented on Mar 13, 2019

markekraus commented on May 5, 2019

burkasaurusrex commented on Jun 7, 2019

sba923 commented on Jul 10, 2020

iSazonov commented on Jul 16, 2020

God-damnit-all commented on Nov 29, 2021

CarloToso commented on Nov 8, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions