Repository navigation
Question Regarding New HTTP Persistent Connection Feature in Web Cmdlets #20713
Description
Summary of the new feature / enhancement
One of the changes in 7.4.0 I'm the most excited for is the HTTP Connection Reuse (#19249) primarily to help speed up API calls with pagination where a call might need to be sent to the same endpoint 100+ times to retrieve all results.
A fairly common example of that however is taking advantage of the -FollowRelLink parameter to only have a single Invoke-RestMethod command in my script. I can't find any documentation stating as such, but would that parameter by default take advantage of this new improvement without specifying a WebSession? Since there would rarely be a time you specified that parameter if you were not expecting multiple calls to be made. Or would a dummy call still have to be made first and thrown away just to obtain the WebSession variable for use?
Proposed technical implementation details (optional)
No response
Activity
JAK1047 commentedon Nov 20, 2023
Answered my own question that a websession isn't implicitly made when using -FollowRelLink. It would be a useful feature if feasible to implicitly reuse the HTTP connection when calling -FollowRelLink as a test on my end shows if I make a dummy call just to establish the web session first it cuts my command time from 6.7 minutes to 3.2 which is substantial. But in my own example I'll have to hard code a throwaway call first to establish a web session which many users unaware of this change may not think to do.
stevenebutler commentedon Nov 24, 2023
Have you tried creating your websession object manually first? I haven't tried this myself but it may do the trick.
I don't have an API that has rellinks like you suggest but as an example, something like below might work for you?
JAK1047 commentedon Nov 25, 2023
Yep that's essentially what I wound up doing in the end and can confirm it works. To expand on my point a bit though not many users will know about the new reuse functionality of 7.4 to use a WebSession and since -FollowRelLink is designed to be used in situations where multiple calls to the same address will be made (and as such benefits the most from this change implemented in 7.4) having the function automatically create and use a web session if one is not provided may be a nice improvement to the cmd.
iSazonov commentedon Nov 25, 2023
Web cmdlets always create WebSession if they haven't WebSession as argument. So in one cmdlet invocation the same HttpClient is used.
JAK1047 commentedon Nov 25, 2023
And every subsequent call made from -FollowRelLink reuses that same WebSession created in the initial cmd as well? That was what the original question was, but from my own experimentation that did not seem to be the case. Specifying a web session first for the command to use was cutting the same command from 6.7 minutes to 3.2. I don't know of any public APIs with pagination support, but on Monday I can test against Graph API as well to see if there was just an issue with the API I was testing against (Okta in this case)
JAK1047 commentedon Nov 28, 2023
Graph apparently uses Odata links for pagination so -FollowRelLink didn't work, but tested with a SolarWinds implementation we have and saw similar timings. Specifying a WebSession when using -FollowRelLink sped up the command time by nearly half. So, my assumption at the moment is still that the command does not reuse the initial WebSession created when FollowRelLink is used unless a web session is implicitly passed to it.
stevenebutler commentedon Dec 1, 2023
My reading of the code suggests Ilya (@iSazonov) is correct, so if it's taking double time without specifying a WebSession parameter then I have to assume there's some subtle interaction (bug?) that's causing it to recreate the WebSession client for each request. There's a lot of features/code in the path for following rel links so not easy to tell what without a test case to debug.
If you could provide a test case I can look into it when I have a chance. Note that the WebSession only exists for a single CmdLet invocation - if it is not persisted through being passed as a parameter or being requested to be saved in the session variable it will be disposed at the end of each cmdlet invocation.
JAK1047 commentedon Dec 1, 2023
Definitely thank you for looking. I only have two separate test cases within my own company, so I'd love to have someone else try to confirm there's not just a flaw to my testing methodology. Finding an API that's free, public, and follows pagination using rellinks to use as an example test case has turned up nada so far however. If I find anything I'll make sure to reply back.
stevenebutler commentedon Dec 1, 2023
If you are familiar with Fiddler Classic it is easy to see if multiple connections are being made by proxying the powershell commands through Fiddler.
JAK1047 commentedon Dec 1, 2023
Sadly my company laptop is locked down and software installations can only come through SCCM after approval, but I can probably do the same with plain old netsh trace and then opening the etl file in Wireshark on my personal PC. I'll look into it.
stevenebutler commentedon Dec 1, 2023
I setup a test with fiddler in autoresponder mode and it looks like pwsh maintained a persistent connection for FollowRelLink.
Perhaps it behaves differently when a proxy is not in use - if you can show evidence it's not keeping a persistent connection please update here and I'll look into it further.
