I need to use the text-based browser Lynx to dump 1000s of html files into text files, with the same filename.

For any given filename.html the command is

lynx filename.html --force-html --dump > filename.txt

The problem is I have tens of thousands of these files.

If I used lynx *.html --force-html --dump *.html I don't think each file will generate the text file with the same name.

  • You might use html2text instead of lynx for such conversions. – Basile Starynkevitch Jun 27 '15 at 17:41

You need to execute lynx once per file, to produce separate output files. To do something over multiple files in sequence, use a for loop. The pattern *.html matches all files in the current directory whose name ends with .html.

for x in *.html; do … done

In each run through the loop, the variable x designates the current file name. Use "$x" to refer to the file name (don't forget the double quotes or your script will break on some file names, e.g. containing spaces).

To build the .txt file name, remove the .html suffix from the value of x and add the .txt suffix. There is a parameter expansion construct to get the value of a variable minus a suffix: "${VARIABLE%SUFFIX}".

for x in *.html; do
  lynx --force-html --dump "$x" >"${x#.html}.txt"

If you want to act on HTML files in subdirectories as well, there are two possibilities. If your shell is bash, ksh or zsh, you can use the ** pattern to recurse into subdirectories. In bash, this feature needs to be enabled with shopt -s globstar first; in ksh, with set -o globstar.

shopt -s globstar
for x in **/*.html; do
  lynx --force-html --dump "$x" >"${x#.html}.txt"

Alternatively, use the find command. Since you need to do some manipulation on the file name, you need to make find run a shell.

find . -name '*.html' -exec sh -c '
  lynx --force-html --dump "$0" >"${0#.html}.txt"
' {} \;

$0 is the first argument after sh -c CODE. It's a little faster to run the shell in batches, combining find (with -exec … {} +, instructing find to pass multiple file names to each single shell invocation) with a for loop. for x do means “iterate over the shell's command line arguments except $0”.

find . -name '*.html' -exec sh -c '
  for x; do lynx --force-html --dump "$x" >"${x#.html}.txt"; done
' _ {} +
| improve this answer | |
( set ./*.html; [ -f "$1" ] || exit
  printf 'eval "$L;shift" >"${1%%%.0s.*}.txt"\n' "$@" |
  L='    lynx "$1" --force-html --dump'  sh -eCs "$@"
) &
| improve this answer | |
for x in *.html;do
    lynx $x --force-html --dump >${x%%.html}.txt
| improve this answer | |
  • 1
    Using a line-oriented tool like grep is not going to work on file names which don't have to be a single line (i.e. can have a newline embedded). – Anthon Jun 26 '15 at 5:39
  • This got the job done! – hipHopMetropolisHastings Jun 26 '15 at 16:51
  • 1
    Why on earth are you writing *.html in such a complicated, fragile way? – Gilles 'SO- stop being evil' Jun 26 '15 at 21:43
  • But it is not a good answer.... – Basile Starynkevitch Jun 27 '15 at 17:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.