Download Web Folder Contents with Powershell (wget -r replacement)

Technology keeps moving but this post has not.

What you're about to read hasn't been updated in more than a year. The information may be out of date. Let me know if you see anything that needs fixing.

We've been working lately to use HashiCorp Packer to standardize and automate our VM template builds, and we found a need to pull in all of the contents of a specific directory on an internal web server. This would be pretty simple for Linux systems using wget -r, but we needed to find another solution for our Windows builds.

A coworker and I cobbled together a quick PowerShell solution which will download the files within a specified web URL to a designated directory (without recreating the nested folder structure):

 1$outputdir = 'C:\Scripts\Download\'
 2$url = ''
 4# enable TLS 1.2 and TLS 1.1 protocols
 5[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12, [Net.SecurityProtocolType]::Tls11
 7$WebResponse = Invoke-WebRequest -Uri $url
 8# get the list of links, skip the first one ("[To Parent Directory]") and download the files
 9$WebResponse.Links | Select-Object -ExpandProperty href -Skip 1 | ForEach-Object {
10    $fileName = $_.ToString().Split('/')[-1]                        # 'filename.ext'
11    $filePath = Join-Path -Path $outputdir -ChildPath $fileName     # 'C:\Scripts\Download\filename.ext'
12    $baseUrl = $url.split('/')                                      # ['https', '', '', 'stuff', 'files']
13    $baseUrl = $baseUrl[0,2] -join '//'                             # ''
14    $fileUrl  = '{0}{1}' -f $baseUrl.TrimEnd('/'), $_               # ''
15    Invoke-WebRequest -Uri $fileUrl -OutFile $filePath 

The latest version of this script will be found on GitHub.