Repository navigation
WebClient needs to expose the timeout property (or improve Invoke-WebRequest implementation) when downloading large files #4679
Description
I have been having some issues interacting with REST full APIs, specifically with our product (Pexip) which is writing with Python and Django and uses Apache. Some of these issues are outlined in #2112 and #4274 and this involves a similar an linked problem. I also added a comment to https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/invoke-webrequest?view=powershell-5.1 that outlines this issue that I have repeated below:
Attempting to download a large file, GETting from a REST API.
The Basic Invoke-WebRequest
Invoke-WebRequest -Method Get -Uri $resource -cred $cred -OutFile $Destination -TimeoutSec 1800
This would be the ideal. I can set timeout values and seemingly achieve what I want in one line. Unfortunately, this completely locked up the machine the script was running on. Basically, all the memory was consumed and after an hour or so, I had to end the script – of the 3.5 GB file, I think 200 MB was downloaded! What I needed is a way to stream the remote file to a local file without loading everything into memory.
Using WebClient
This seemed a lot better, however, the WebClient has no timeout property!!!
$wc = New-Object System.Net.WebClient
$wc.Credentials = New-Object System.Net.NetworkCredential($User, $Pass)
$wc.Headers.Add("Authorization", "Basic $encoded")
$wc.DownloadFile($resource, $DestinationPath)
This is what I ended up using and in order to get the timeout to work, I used an extended class – luckily someone clever than I has achieved this for PowerShell using a C# class (https://koz.tv/setup-webclient-timeout-in-powershell/) although I assume the native classes in PS could also be used – but why?
I have also found that I must add the additional Authorisation Headers, otherwise the client will get challenged then end up download the file effectively twice. Using Fiddler and watching the Ethernet stats, you can see this happen. Indeed, this seemingly occurred for all methods
Using HTTP client.
Well I tried and failed dismally. However, I seem to create the Httpclient, it only ever seemed to stream the remote file into memory before dumping to a file. I know what I wanted to do, but do not have the wherewithal to do it.
Activity
iSazonov commentedon May 26, 2018
Mark Kraus (@markekraus) Is the Issue already fixed?
markekraus commentedon May 26, 2018
No. we added a resume feature but not a way to timeoul a download.
This is, however, something we could add. Since we do a looped read of the stream, we could insert logic in their to stop when the time out is reached.
iSazonov commentedon May 27, 2018
Ah, I again confused these two PR.
CarloToso commentedon Mar 15, 2023
stevenebutler This issue can be fixed with the
-ReceiveTimeout(the name is a placeholder) parameter discussed in #19315. I think you could fix this issue after we merge #19330iSazonov commentedon Mar 16, 2023
This issue touches several problems but the main request is "What I needed is a way to stream the remote file to a local file without loading everything into memory."
Does this problem still exist and in what specific scenarios?
CarloToso commentedon Mar 16, 2023
There are two problems here:
Invoke-WebRequest https://speed.hetzner.de/10GB.bin -OutFile 10gb.bin(it still writes to memory if we use
-PassThru, but I think it's by design)-ReceiveTimeoutpropertyiSazonov commentedon Mar 16, 2023
If so we should close the issue.
It is another request. I think it makes no sense. Otherwise we will have to add timeouts in all cmdlets like Copy-Item, Get-Content and so on. If you want to discuss this please open new issue.
andrei-epure-sonarsource commentedon Aug 17, 2023
CarloToso starting what version does the download streaming work by design, please?
Thanks!
CarloToso commentedon Aug 17, 2023
I tested it works correctly in 5.1, 7.2, 7.3, 7.4. I hypotesize it was fixed before 6.1, but I didn't check.