New answers tagged


Here's an improved script: #!/bin/sh cat file.txt |while read url do lynx -listonly -dump "$url" done | awk '/^[ ]*[1-9][0-9]*\./{sub("^ [^.]*.[ ]*","",$0); print;}'| \ sort -u allowing for any type of URL recognized by lynx (including ftp for instance). The script sorts the result, eliminates duplicates (which lynx will not do by itself). Further ...


Calling list.txt your list: for i in $(cat list.txt) do lynx -accept_all_cookies -dump $i |grep "http" |sed -e "s/^.*http/http/" done I suggest to redirect the output on some file.

Top 50 recent answers are included