Repository navigation
Link header pagination only parsing first rel #5257
Description
Steps to reproduce
$uri_base = 'https://MYDEVORG.okta.com'
$apikey = 'MYAPIKEY'
$uri = $uri_base + '/api/v1/users/?filter=status eq "STAGED"&limit=200'
$method = 'GET'
$headers = @{'Accept'='application/json';'Content-Type'='application/json';'Authorization'="SSWS $apikey"}
PS /> $json = Invoke-RestMethod -Uri $uri -Method $method -Headers $headers -FollowRelLink
PS /> $json.Count
200
PS /> $webreq= Invoke-WebRequest -Uri $uri -Method $method -Headers $headers
PS /> $webreq.RelationLink
Key Value
--- -----
self https://MYDEVORG.okta.com/api/v1/users?limit=200&filter=status+eq+%22STAGED%22
PS /> $webreq.Headers.Link
<https://MYDEVORG.okta.com/api/v1/users?limit=200&filter=status+eq+%22STAGED%22>; rel="self"
<https://MYDEVORG.okta.com/api/v1/users?after=OKTAUSERID&limit=200&filter=status+eq+%22STAGED%22>; rel="next"
PS />Expected behavior
Invoke-RestMethod should follow the next rel link and return additional results.
Invoke-WebRequest RelationLink dictionary should contain both "self" and "next".
Actual behavior
Only the first value from $webreq.Headers.Link is parsed into RelationLink.
Environment data
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial
> $PSVersionTable
Name Value
---- -----
PSVersion 6.0.0-beta.9
PSEdition Core
GitCommitId v6.0.0-beta.9
OS Linux 4.4.0-43-Microsoft #1-Microsoft Wed Dec 31 14:42:53 PST 2014
Platform Unix
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0References
Related commit: 40446c8
Related RFC: https://github.com/PowerShell/PowerShell-RFC/blob/master/2-Draft-Accepted/RFC0021-Link-header-based-pagination-for-WebCmdlets.md
Activity
dddugan commentedon Oct 27, 2017
May be due to WebRequestPSCmdlet.CoreClr.cs ParseLinkHeader function using .FirstOrDefault() in the foreach? Also appears that IWR Headers.Link is a comma-separated string on Windows PowerShell, but an array of strings on PowerShell Core.
SteveL-MSFT commentedon Oct 28, 2017
dddugan can you post the the header section of the RawContent property of the result using Invoke-WebRequest? Feel free to scrub the data, but I'd like to see the format of the returned headers specifically the Link header.
dddugan commentedon Oct 28, 2017
There seem to be multiple headers for the multiple values. From my reading of the RFC a single Link: header should support multiple links. I don't see that it's prohibited to do multiple Link: headers instead, but no mention. So maybe I've run into a case where the vendor is doing something unexpected...
SteveL-MSFT commentedon Oct 28, 2017
My reading of the RFC particularly:
Is that multiple headers with the same name MUST be comma separated. So having two Link: headers that aren't a comma separated list seems to not be conforming.
markekraus commentedon Oct 28, 2017
Steve Lee (@SteveL-MSFT) that RFC is so poorly worded, What it means is that the following two must be equvilent:
That means that you cannot do
instead,
Value1, Value2becomes a single value and not 2 separate values. This allows for multiple headers to support commas as a character where as a single header, commas are always a value separator with no escape character.In Full CLR they mash these all together for you. In CoreFX they treat multiple headers as separate elements in the header collection.
The solution is to implement something like what is in #4494 in
ParseLinkHeader(). Since I'm familiar with it, I'll take this one. but it should be High Priority.dddugan commentedon Oct 29, 2017
Thanks, friends! Appreciate the quick response. Looking forward to testing the fix.
markekraus commentedon Oct 29, 2017
dddugan If you want to validate it now you can build from the MultipleRelLinkFix branch on my repo: https://github.com/markekraus/PowerShell/tree/MultipleRelLinkFix
On windows you can use the MSI or zip produced by AppVeyor https://ci.appveyor.com/project/PowerShell/powershell/build/6.0.0-beta.9-6425/artifacts