Invoke-WebRequest removes the Content-Type header from WebResponseObject's

The Content-Type header appears to be missing everywhere except $response.BaseResponse.Content.Headers.ContentType. it is even being removed from $response.RawContent . it happens regardless of using -UseBasicParsing or not, regardless of -method and regardless of -UserAgent.

Steps to reproduce

Describe 'Invoke-WebRequest' {
    Context 'Response: Content-Type Header' {
        $Uri = 'http://httpbin.org/headers'
        $response = Invoke-WebRequest -Uri $Uri
        It "Has BaseResponse.Content.Headers.ContentType" {
            $response.BaseResponse.Content.Headers.ContentType | 
            Should Not BeNullOrEmpty
        }
        It "Has Headers.'Content-Type'" {
            $response.Headers.ContainsKey('Content-Type') | Should Be $True
            $response.Headers.'Content-Type' | Should Not BeNullOrEmpty
        }
        It "Has BaseResponse.Headers.'Content-Type'" {
            $response.BaseResponse.Headers.Keys -contains 'Content-Type' | Should Be $True
            $response.BaseResponse.Headers.Get('Content-Type') | Should Not BeNullOrEmpty
        }
        It "Has BaseResponse.ContentType" {
            $response.BaseResponse.ContentType | Should Not BeNullOrEmpty
        }
        It "Has Content-Type in the RawContent" {
            $response.RawContent -match 'Content-Type' | Should Be $True
        }
    }
}

Expected behavior

Describing Invoke-WebRequest

  Context Response: Content-Type Header
    [+] Has BaseResponse.Content.Headers.ContentType.MediaType 12.62s
    [+] Has Headers.'Content-Type' 11.36s
    [+] Has BaseResponse.Headers.'Content-Type' 28ms
    [+] Has BaseResponse.ContentType 10ms
    [+] Has Content-Type in the RawContent 18ms

Actual behavior

Describing Invoke-WebRequest
   Context Response: Content-Type Header
    [+] Has BaseResponse.Content.Headers.ContentType 13.59s
    [-] Has Headers.'Content-Type' 42ms
      Expected: {True}
      But was:  {False}
      at line: 10 in
      10:             $response.Headers.ContainsKey('Content-Type') | Should Be $True
    [-] Has BaseResponse.Headers.'Content-Type' 23ms
      Expected: {True}
      But was:  {False}
      at line: 14 in
      14:             $response.BaseResponse.Headers.Keys -contains 'Content-Type' | Should Be $True
    [-] Has BaseResponse.ContentType 24ms
      Expected: value to not be empty
      at line: 18 in
      18:             $response.BaseResponse.ContentType | Should Not BeNullOrEmpty
    [-] Has Content-Type in the RawContent 28ms
      Expected: {True}
      But was:  {False}
      at line: 21 in
      21:             $response.RawContent -match 'Content-Type' | Should Be $True

Environment data

> $PSVersionTable
Name                           Value
----                           -----
PSVersion                      6.0.0-beta
PSEdition                      Core
GitCommitId                    v6.0.0-beta.4
OS                             Microsoft Windows 10.0.15063
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Compared to Windows PowerShell 5.1 it seems that Content-Length is also missing

Looks like this is by design of HttpResponseMessage (you'll have to open an issue with corefx if you want it changed). You can still get them, but they are in a different part of the object:

PS C:\Users\slee\repos\PowerShell> $o = Invoke-WebRequest http://bing.com
PS C:\Users\slee\repos\PowerShell> $o.Headers

Key                       Value
---                       -----
Cache-Control             {max-age=0, private}
Date                      {Thu, 03 Aug 2017 16:15:23 GMT}
P3P                       {CP="NON UNI COM NAV STA LOC CURa DEVa PSAa PSDa OUR IND"}
Set-Cookie                {SRCHD=AF=NOFORM; domain=.bing.com; expires=Sat, 03-Aug-2019 16:15:24 GMT; path=/, SRCHUID...
Vary                      {Accept-Encoding}
Strict-Transport-Security {max-age=10886400; includeSubDomains; preload}
X-MSEdge-Ref              {Ref A: 54EF10511D70472CACD48B2E03FDB68F Ref B: WSTEDGE0518 Ref C: 2017-08-03T16:15:24Z}


PS C:\Users\slee\repos\PowerShell> $o.BaseResponse.Content.Headers

Key            Value
---            -----
Content-Length {127796}
Content-Type   {text/html; charset=utf-8}


PS C:\Users\slee\repos\PowerShell> $o.BaseResponse.Headers

Key                       Value
---                       -----
Cache-Control             {max-age=0, private}
Date                      {Thu, 03 Aug 2017 16:15:23 GMT}
P3P                       {CP="NON UNI COM NAV STA LOC CURa DEVa PSAa PSDa OUR IND"}
Set-Cookie                {SRCHD=AF=NOFORM; domain=.bing.com; expires=Sat, 03-Aug-2019 16:15:24 GMT; path=/, SRCHUID...
Vary                      {Accept-Encoding}
Strict-Transport-Security {max-age=10886400; includeSubDomains; preload}
X-MSEdge-Ref              {Ref A: 54EF10511D70472CACD48B2E03FDB68F Ref B: WSTEDGE0518 Ref C: 2017-08-03T16:15:24Z}

As you can see, the Headers property of the output object just mirrors .BaseResponse.Headers

The new location on the objects is somewhat understandable, but, what about it being missing from RawResponse on Microsoft.PowerShell.Commands.BasicHtmlWebResponseObject?

$Uri = 'http://httpbin.org/headers'
$response = Invoke-WebRequest -Uri $Uri
$response.RawContent

5.1:

HTTP/1.1 200 OK
Connection: keep-alive
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
X-Processed-Time: 0.000962972640991
Content-Length: 181
Content-Type: application/json
Date: Thu, 03 Aug 2017 16:23:14 GMT
Server: meinheld/0.6.1
Via: 1.1 vegur
X-Powered-By: Flask

{
  "headers": {
    "Connection": "close", 
    "Host": "httpbin.org", 
    "User-Agent": "Mozilla/5.0 (Windows NT; Windows NT 10.0; en-US) WindowsPowerShell/5.1.14393.1480"
  }
}

6:

HTTP/1.1 200 OK
Connection: keep-alive
Date: Thu, 03 Aug 2017 16:21:22 GMT
Via: 1.1 vegur
Server: meinheld/0.6.1
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
X-Powered-By: Flask
X-Processed-Time: 0.000738143920898

{
  "headers": {
    "Accept-Encoding": "peerdist",
    "Connection": "close",
    "Host": "httpbin.org",
    "User-Agent": "Mozilla/5.0 (Windows NT; Microsoft Windows 10.0.14393 ; en-US) WindowsPowerShell/6.0.0",
    "X-P2P-Peerdist": "Version=1.1",
    "X-P2P-Peerdistex": "MinContentInformation=1.0, MaxContentInformation=2.0"
  }
}

One would expect the RawContent to truly be a "Raw Content". Is this also related to the CoreFX change? Is it something that should be worked around in PowerShell 6? Or should the expectation change that Contet-Type and Content-Length will not be available in RawContent?

The big difference between 6 and 5.1 is that 5.1 uses the WebResponse class while 6 uses the HttpResponseMessage class due to differences between CoreFx and .Net. In the case of RawContent, it's just a "manual" serialization of HttpResponseMessage where those headers are not present. If coreFx put it "back", they would just show up with this cmdlet.

We could add some specialized code to just add them, but it seems the correct change is in corefx.

Digging around the CoreFX source for System.Net.Http.HttpResponseMessage, It appears the separation of the content headers is very much deliberate. The ToString() override is calling HeaderUtilities.DumpHeaders() and passing both _headers and _content. I don't think there is any chance of that being reversed.

I'll do a PR for replicating that functionality in ContentHelper

Navigation Menu

Repository navigation

Invoke-WebRequest removes the Content-Type header from WebResponseObject's #4467

Steps to reproduce

Expected behavior

Actual behavior

Environment data

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Invoke-WebRequest removes the Content-Type header from WebResponseObject's #4467

Description

Steps to reproduce

Expected behavior

Actual behavior

Environment data

Activity

SteveL-MSFT commented on Aug 3, 2017

SteveL-MSFT commented on Aug 3, 2017

markekraus commented on Aug 3, 2017

SteveL-MSFT commented on Aug 3, 2017

markekraus commented on Aug 3, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions