3

I'm thinking of something like ls @files.lst doing what ls [contents of files.lst] would do, globbing included.

I think I'm remembering the above syntax, (at-sign, filename) from somewhere (maybe an old Digital OS from sometime before the last ice age?). Searching for permutations of Bash, indirect, arguments, list, ... isn't getting me anywhere. I know there are ways to write this into the (every?!) script, but does bash already implement it in some more general way that so has far eluded me?

Update: This is for BSD/Darwin (ATM) but I appreciate the more general answers, too. I'm on a Mac, if that's not now obvious, so computational inefficiency won't hamper anyone else and the workload is pretty small anyway. Of more interest is the ability to use existing shell scripts that were written to receive their commands from the command line, and most of the time, they will.

1
  • 1
    There are several well-known programs that implement that @ syntax for reading arguments from a file instead of the command line. GCC might be the most popular of them - maybe even the originator of the syntax, I don't know - but I bet it inspired plenty of other less popular programs to do the same thing. Perhaps that could account for what you're remembering. (I do want to point out that in general the syntax is a feature of the program, not the invoking shell.)
    – David Z
    Nov 28, 2021 at 2:00

4 Answers 4

4

If you want to do something for each line of a text file, there are generally two ways of doing it.

  1. Read the file line by line and do whatever you need to do to the line in the body of the loop:

    while IFS= read -r pathname; do
        ls -d -- "$pathname"
    done <files.list
    

    The above reads lines from the file files.list and executes ls -d -- on each.

    Related:

  2. Use a utility that makes the looping implicit for you, like xargs or GNU parallel:

    xargs -I {} -- ls -d -- {} <files.list
    

    The code does the same as the loop above but uses xargs to execute ls -d -- for each argument read from files.list. An argument is a line where newlines may be embedded if they are escaped (quotes are also processed, so they must be escaped to be retained).

    xargs -- ls -d -- <files.list
    

    The above would run ls -d -- on as many of the arguments in the file as possible at once. One needs to quote arguments containing embedded tabs or spaces and escape individual embedded newlines and quotes.

    The GNU parallel utility is more generic than xargs and allows for more control over the executed jobs.

  3. Since the shell is often the slowest language you can pick for processing data, if you need to do something other than executing a command for each line in a file, then it's probably better to do so in a language that makes it easier to do this. Examples of such languages:

    • Awk
    • Perl
    • Python

In either case, since you store newline-delimited pathnames in your file, you are disqualified from using pathnames containing newlines (these are legal on Unix systems) unless quoted adequately for use with xargs (see the xargs manual).

2
  • Note that -I'{}' still does quote processing and removes leading blanks. To process one line at a time, you need the -d '\n' GNU extension. Or use tr '\n' '\0' | xargs -r0. -0 and -r are also GNU extensions but found in more other implementations. Also note that most xargs implementations would still run cmd once (as POSIX requires) in xargs cmd < empty-file Nov 27, 2021 at 9:47
  • @StéphaneChazelas Thanks! I've rephrased it slightly now. I don't want the answer to turn into an xargs tutorial.
    – they
    Nov 27, 2021 at 10:12
2

Read the output of the man xargs command, and do something like:

xargs -r <files.lst somecommand

xargs will fill somecommand's command line with as many items from files.lst as will fit, and will use as many somecommand executions as needed. See xargs --show-limits </dev/null.

You can also use xargs to feed the filenames to somecommand 1 at a time:

xargs -r -n 1 <files.lst somecommand
1
  • Thank you all for your contributions. 'xargs' seems the best fit to my use-case, given that I don't want to customize the shell-scripts to accomplish the indirection, but be able to apply it to whatever scripts I have.
    – JRobert
    Nov 28, 2021 at 16:11
2

Learning details of Bash; www.tldp.org -> Bash guides

$ man bash tells some details, but in terse form.

Now bypassing those sources above that has most pf the details;
a very simplistic and incomplete answer:

$ ls -l $(cat file)
... will read file and insert that text as arguments for ls, replacing LF (\n) characters with space (SP) characters.

Note though that globbing is unlikely to happen unless you do tricks here.

Note another problem; if any of the lines in the file contains a single space or more, the words on that line will become separate arguments to ls.

Whether a specific command (bash specific or not) can take arguments from within a file varies;
man command_name generally will tell details, for a specific command named command_name (replace that with a true command name).

1

In the zsh shell:

cmd -- $(<file)

With the default value of $IFS, would pass each space/TAB/NL/NUL delimited word in file to cmd (where $(<file) (from ksh) expands to the contents of file without the trailing newline characters, and when left unquoted is subject to IFS-splitting¹).

cmd -- ${(f)"$(<file)"}

Would pass the contents of each non-empty line as separate arguments to cmd (where the f parameter expansion flag splits on newlines aka linefeeds, and empty arguments are discarded as the expansion is not quoted).

Replace f with 0 to split on NULs instead of NLs.

cmd -- "${(Q@)${(zZ[n])$(<file)}}"

Would interpret zsh-style quoting in the file, with the z flag doing the quote-aware tokenisation, Z[n] tuning it so that newline is just considered as a separator, Q removes the quotes and @ combined with quoting preserve empty elements.

For example, on a file like:

'' "foo bar"

whatever $'\n'

and printf '<%s>\n' as the command, that would give:

$ printf '<%s>\n' $(<file)
<''>
<"foo>
<bar">
<whatever>
<$'\n'>
$ printf '<%s>\n' ${(f)"$(<file)"}
<'' "foo bar">
<whatever $'\n'>
$ printf '<%s>\n' "${(Q@)${(zZ[n])$(<file)}}"
<>
<foo bar>
<whatever>
<
>

¹ in the Korn shell (or bash), it would subject to IFS-splitting and globbing, which you generally don't want unless those words happen to be meant to be glob patterns. Also note that ksh/bash don't split on NULs by default (and generally can't cope with those).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.