0

I have to download some files through a URL pattern: https://www.server.org/data/2021/{i} where i can vary from 1 to 1000.

Files are pdfs and they all have they own name, e.g.:

https://www.server.org/data/2021/1 ====> document_about_feature.pdf   
https://www.server.org/data/2021/2 ====> "activity report.pdf"    
https://www.server.org/data/2021/3 ====> "2021-financial analysis from bla bla.pdf" 
...    

For the moment I'm getting those files using wget as follow:

#!/bin/bash

for i in {1..1000}
do
  URL="https://www.server.org/data/2021/${i}"
  wget -nv --content-disposition "${URL}"
done

It's the only way I found to keep the original file names when downloading them (without the --content-disposition flag, the files are named 1.pdf, 2.pdf, 3.pdf, etc). So it works fine, but I would like to keep the link between i and the file name by appending a prefix (or suffix) to the file name:

https://www.server.org/data/2021/1 ====> 1_document_about_feature.pdf   
https://www.server.org/data/2021/2 ====> "2_activity report.pdf"    
https://www.server.org/data/2021/3 ====> "3_2021-financial analysis from bla bla.pdf"   
...  

How could I properly achieve this (Ubuntu 22.04.3)?

Version: GNU Wget (1.21.2 built on linux-gnu).

1 Answer 1

1

I'd probably just download and afterwards rename; something like

#!/bin/bash
# use this directory as tempfile prefix
# (so that move becomes a single-filesystem operation)
tmpdirname="$(mktemp -d -p --tmpdir . "fetch_XXXXXXX")"
for i in {1..1000}; do
  URL="https://www.server.org/data/2021/${i}"

  # I'm almost certain you'll want file names 0001,…,0999,1000; not 1…1000
  formatted="$(printf '%04d' i)"

  dirname="${tmpdirname}/${formatted}"
  mkdir "${dirname}"
  pushd "${dirname}"
    wget -nv --content-disposition "${URL}"
    filename=*
  popd
  mv -- "${dirname}/${filename}" "${formatted} ${filename}"
  rmdir "${dirname}"
done
rmdir "${tmpdirname}

Personally, I remember wget being buggy around --content-disposition, at least according to its man page.

I tend towards curl these days¹, so I'd replace wget -nv --content-disposition "${URL}" with curl --remote-name --remote-header-name:

#!/bin/bash
# use this directory as tempfile prefix
# (so that move becomes a single-filesystem operation)
tmpdirname="$(mktemp -d -p --tmpdir . "fetch_XXXXXXX")"
for i in {1..1000}; do
  URL="https://www.server.org/data/2021/${i}"

  # I'm almost certain you'll want file names 0001,…,999,1000; not 1…1000
  formatted="$(printf '%04d' i)"

  dirname="${tmpdirname}/${formatted}"
  mkdir "${dirname}"
  pushd "${dirname}"
    curl --remote-name --remote-header-name -- "${URL}"
    filename=*
  popd
  mv -- "${dirname}/${filename}" "${formatted} ${filename}"
  rmdir "${dirname}"
done
rmdir "${tmpdirname}

¹ good testing, very modern protocol support, unlike wget properly reuses TLS sessions, leading to faster connections at lower server load; and no logical bug where -- can be a valid argument…

3
  • Thanks for your answer, I'm facing an error at line 4: tmpdirname=... : mktemp: too many templates Try 'mktemp --help' for more information.
    – s.k
    Commented Nov 25, 2023 at 12:47
  • After the doc, should't the current dir (i.e. the .) be placed just after the -p flag instead? : mktemp -p . -d --tmpdir 'fetch_XXXXXXX'
    – s.k
    Commented Nov 25, 2023 at 12:51
  • Also, after popd, the filename variable which was the actual filename before executing popd is set to the star * character. I don't know why it changed.
    – s.k
    Commented Nov 25, 2023 at 13:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .