55

I'm looking for a way to show all of the URLs in a redirect chain, preferably from the shell. I've found a way to almost do it with curl, but it only shows the first and last URL. I'd like to see all of them.

There must be a way to do this simply, but I can't for the life of me find what it is.

Edit: Since submitting this I've found out how to do it with Chrome (CTRL+SHIFT+I->Network tab). But, I'd still like to know how it can be done from the Linux command line.

4 Answers 4

72

How about simply using wget?

$ wget http://picasaweb.google.com 2>&1 | grep Location:
Location: /home [following]
Location: https://www.google.com/accounts/ServiceLogin?hl=en_US&continue=https%3A%2F%2Fpicasaweb.google.com%2Flh%2Flogin%3Fcontinue%3Dhttps%253A%252F%252Fpicasaweb.google.com%252Fhome&service=lh2&ltmpl=gp&passive=true [following]
Location: https://accounts.google.com/ServiceLogin?hl=en_US&continue=https%3A%2F%2Fpicasaweb.google.com%2Flh%2Flogin%3Fcontinue%3Dhttps%3A%2F%2Fpicasaweb.google.com%2Fhome&service=lh2&ltmpl=gp&passive=true [following]

curl -v also shows some info, but looks not as useful as wget.

$ curl -v -L http://picasaweb.google.com 2>&1 | egrep "^> (Host:|GET)"
> GET / HTTP/1.1
> Host: picasaweb.google.com
> GET /home HTTP/1.1
> Host: picasaweb.google.com
> GET /accounts/ServiceLogin?hl=en_US&continue=https%3A%2F%2Fpicasaweb.google.com%2Flh%2Flogin%3Fcontinue%3Dhttps%253A%252F%252Fpicasaweb.google.com%252Fhome&service=lh2&ltmpl=gp&passive=true HTTP/1.1
> Host: www.google.com
> GET /ServiceLogin?hl=en_US&continue=https%3A%2F%2Fpicasaweb.google.com%2Flh%2Flogin%3Fcontinue%3Dhttps%253A%252F%252Fpicasaweb.google.com%252Fhome&service=lh2&ltmpl=gp&passive=true HTTP/1.1
> Host: accounts.google.com
4
  • That looks like it'll do it. I'm just curious though, why did you send STDERR to STDOUT?
    – felwithe
    Jul 15, 2015 at 10:12
  • 3
    If I'm not mistaken, it will only work for redirection using the location header, not for 301 http codes
    – greg
    Mar 3, 2017 at 10:29
  • 1
    I cannot reproduce the wget example. Returns "503: Service Unavailable." - Maybe they are blocking the wget but not the curl http-user-agent? Is this just me? Nov 17, 2017 at 12:01
  • When I have a redirect from www to the canonical URL, a bit later I have redirect from HTTP to HTTPS or otherwise. How can I collect an array of all redirects per domain? Was it the point of a question?
    – Max Yudin
    Oct 23, 2019 at 19:18
3

To show all of the URLs in a redirect chain, including the first:

wget -S --spider https://rb.gy/x7cg8r 2>&1 \
 | grep -oP '^--[[:digit:]: -]{19}--  \K.*'

Result (tested on Fedora Linux):

https://rb.gy/x7cg8r
https://t.co/BAvVoPyqNr
https://unix.stackexchange.com/

wget options used:

-S
--server-response

    Print the headers sent by HTTP servers and responses sent by FTP servers.

--spider

    When invoked with this option, Wget will behave as a Web spider, which
    means that it will not download the pages, just check that they are there
    ...

Source: https://www.mankier.com/1/wget

The combination of -S and --spider causes wget to issue HEAD requests instead of GET requests.

GNU grep options used:

-o
--only-matching

    Print only the matched (non-empty) parts of a matching line, with each such
    part on a separate output line.

-P
--perl-regexp

    Interpret PATTERNS as Perl-compatible regular expressions (PCREs).

Source: https://www.mankier.com/1/grep

The lines we are interested in look like this:

--2021-12-07 12:29:25--  https://rb.gy/x7cg8r

You see that the timestamp consists of 19 characters comprised of digits, hyphens, colons, and spaces. It is therefore matched by [[:digit:]-: ]{19}, where we used a fixed quantifier of 19.

The \K resets the start of the matched portion.

Swap grep with sed

The grep pipeline stage may be replaced with sed, if you prefer:

wget -S --spider https://rb.gy/x7cg8r 2>&1 \
 | sed -En 's/^--[[:digit:]: -]{19}--  (.*)/\1/p'

Compared to the curl-based solution...

The curl-based solution omits the first url in the redirect chain:

$ curl -v -L https://rb.gy/x7cg8r 2>&1 | grep -i "^< location:"
< Location: https://t.co/BAvVoPyqNr
< location: https://unix.stackexchange.com/

Furthermore, it incurs a 4354.99% increase in bytes sent to second pipeline stage:

$ wget -S --spider https://rb.gy/x7cg8r 2>&1 | wc -c
2728

$ curl -v -L https://rb.gy/x7cg8r 2>&1 | wc -c
121532

$ awk 'BEGIN {printf "%.2f\n", (121532-2728)/2728*100}'
4354.99

In my benchmarking, the wget solution was slightly (4%) faster than the curl-based solution.

Update: See my curl-based answer for fastest solution.

2

curl -v can show all the URLs in an HTTP redirect chain:

$ curl -v -L https://go.usa.gov/W3H 2>&1 | grep -i "^< location:"
< location: http://hurricanes.gov/nhc_storms.shtml
< Location: https://www.hurricanes.gov/nhc_storms.shtml
< location: https://www.nhc.noaa.gov:443/nhc_storms.shtml
< location: http://www.nhc.noaa.gov/cyclones
< Location: https://www.nhc.noaa.gov/cyclones
< location: http://www.nhc.noaa.gov/cyclones/
< Location: https://www.nhc.noaa.gov/cyclones/
1

A proper curl-based solution

url=https://rb.gy/x7cg8r
while redirect_url=$(
  curl -I -s -S -f -w "%{redirect_url}\n" -o /dev/null "$url"
); do
  echo "$url"
  url=$redirect_url
  [[ -z "$url" ]] && break
done

Result:

https://rb.gy/x7cg8r
https://t.co/BAvVoPyqNr
https://unix.stackexchange.com/

This is 12% faster than my wget-based solution.

Benchmark details

cd "$(mktemp -d)"

cat <<'EOF' >curl-based-solution
#!/bin/bash
url=https://rb.gy/x7cg8r
while redirect_url=$(
  curl -I -s -S -f -w "%{redirect_url}\n" -o /dev/null "$url"
); do
  echo "$url"
  url=$redirect_url
  [[ -z "$url" ]] && break
done
EOF
chmod +x curl-based-solution

cat <<'EOF' >wget-based-solution
#!/bin/bash
url=https://rb.gy/x7cg8r
wget -S --spider "$url" 2>&1 \
 | grep -oP '^--[[:digit:]: -]{19}--  \K.*'
EOF
chmod +x wget-based-solution

hyperfine --warmup 5 ./wget-based-solution ./curl-based-solution
$ hyperfine --warmup 5 ./wget-based-solution ./curl-based-solution
Benchmark #1: ./wget-based-solution
  Time (mean ± σ):      1.397 s ±  0.025 s    [User: 90.3 ms, System: 19.7 ms]
  Range (min … max):    1.365 s …  1.456 s    10 runs
 
Benchmark #2: ./curl-based-solution
  Time (mean ± σ):      1.250 s ±  0.015 s    [User: 72.4 ms, System: 23.4 ms]
  Range (min … max):    1.229 s …  1.277 s    10 runs
 
Summary
  './curl-based-solution' ran
    1.12 ± 0.02 times faster than './wget-based-solution'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.