8

I have a program that expects arguments in the following syntax:

prog [-f filename | -g filename1 filename2] ...

Each filename must be prefixed with the -f flag. For example, the following are valid invocations of prog:

prog -f a.txt -g b.txt c.txt -f d.txt
prog -g a.txt b.txt -g c.txt d.txt
prog -f a.txt -f b.txt -f c.txt

…but the following are not:

prog -f a.txt b.txt
prog -f a.txt -g b.txt
prog a.txt

In my case, I only care about the -f option.

I have a lot of files in a directory, all of which end in .txt. They look like this:

important-files/
├── a.txt
├── b.txt
├── c.txt
├── d.txt
└── filename with spaces.txt

I would like to avoid needing to list out every file one by one. Normally, I would use a straightforward glob for this:

$ prog important-files/*.txt

But this doesn’t work, since it produces the following invalid invocation:

$ prog important-files/a.txt important-files/b.txt important-files/c.txt important-files/d.txt 'important-files/filename with spaces.txt'

…when I really want this invocation:

$ prog -f important-files/a.txt -f important-files/b.txt -f important-files/c.txt -f important-files/d.txt -f 'important-files/filename with spaces.txt'

…since each filename must be prefixed with -f in order for prog to understand they shouldn’t be interpreted like -g.

What is the shortest way to use a glob and prefix each of the files it expands to with a flag?

  • 2
    meanwhile in ZSH this is *.txt(P:\\-f:) – thrig May 22 '18 at 22:50
  • 3
    @thrig It's just *.txt(P:-f:) in fact (Why would you put a backslash there?). – Gilles May 23 '18 at 8:59
13

Using printf and an xargs supporting nul-delimited input:

printf -- '-f\0%s\0' important-files/*.txt | xargs -0 prog

printf loops the format string over the arguments, so for each filename in the expansion of the glob, it will print -f and the filename separated by the null character. xargs then reads this and converts it into arguments for prog. The -- needed since some implementations of printf have trouble with a leading - in the format string. Alternately, the - can be replaced with \055, which is standard.

Same principle as Rakesh's answer, but using the wildcard expansion directly.

  • Why null characters usage needed, why not just double quotes, like: printf ' -f "%s"' important-files/*.txt | xargs prog? – MiniMax May 23 '18 at 8:21
  • 4
    If the filenames contain quotes or backslashes, it's going to be difficult to appropriately escape them. The null character is not allowed in filenames, so it's generally the safest character for separating arbitrary (but valid) filenames. – muru May 23 '18 at 8:27
  • Thanks for answer. In my case is needed to add --, because printf '-f\0%s\0' important-files/*.txt gives the error: bash: printf: -f: invalid option. The printf -- '-f\0%s\0' important-files/*.txt works. – MiniMax May 23 '18 at 8:33
  • 2
    My understanding is that xargs will use its input to pass args to prog. But if there are too many args, it will split them up and invoke prog multiple times. Do we have the possibility here of splitting the args in the wrong place, e.g. calling prog -f a.txt -f b.txt ... -f m.txt -f and then calling prog n.txt -f o.txt ... ? – Digital Trauma May 23 '18 at 17:26
  • 1
    @DigitalTrauma yes, the possibility exists. But if that were to happen, every answer here would fail too, either because of splitting, or because of argument length limit being exceeded. I'm counting on the program doing argument parsing early on and failing hard, of course. – muru May 24 '18 at 1:59
9

With bash, try:

args=()
for f in important-files/*.txt
do
    args+=(-f "$f")
done
prog "${args[@]}"

Note that this will work not merely with filenames that contain blanks but also with filenames that contain single or double quotes or even newlines.

Easy interactive use

For easy interactive use, define a function:

progf() { args=(); for f in  "$@"; do args+=(-f "$f"); done; prog "${args[@]}"; }

This function can be used as follows:

progf important-files/*.txt

To make the function definition permanent, place it in your ~/.bashrc file.

  • This admittedly works, and it’s somewhat palatable in a script, but it’s not terribly pleasant when I really just want to run a one-off command interactively. – Alexis King May 22 '18 at 22:06
  • @AlexisKing OK, but for palatable interactive use, I would define a function (see update). – John1024 May 22 '18 at 22:10
7

zsh now has a P glob qualifier for that.

For older versions, you can always use:

f=(*.txt)
prog -f$^f

$^f enables the rcexpandparam option for the expansion of $f, where arrays are expanded in a style similar to that of the rc shell.

In rc (or zsh -o rcexpandparam):

f=(*.txt)
prog -f$f

Is expanded to prog -fa.txt -fb.txt, a bit as if you had written in csh (or other shells supporting brace expansion): prog -f{a.txt,b.txt}.

(note that it's different from *.txt(P:-f:) in that the file name is glued to the option -ffile.txt as opposed to passing two arguments -f file.txt. Many commands including all those that use the standard getopt() API to parse options support both ways passing arguments to options).

fish also expands arrays like that:

set f *.txt
prog -f$f

In bash, you can emulate it with:

f=(*.txt)
prog "${f[@]/#/-f}"

With ksh93:

f=(*.txt)
prog "${f[@]/*/-f\0}"

For -f and the corresponding file.txts to be passed as separate arguments, another option with zsh using its array zipping operator:

o=-f f=(*.txt); prog ${o:^^f}

You could extend that for your -g option. Assuming there's an even number of txt files:

o=(-g '') f=(*.txt); prog ${o:^^f}

Would pass -g options with two files at a time. Similar to what you'd get with:

printf -- '-g\0%s\0%s\0' *.txt | xargs -r0 prog
  • Would the bash example work? You end up with elements like -f a.txt instead of -f a.txt. – m0dular May 23 '18 at 15:08
  • 1
    @m0dular, no, you end up with -fa.txt like for the zsh/rc/fish approaches above. Applications using getopt() treat it the same as -f a.txt, I can't tell about the OP's application (though it's unlikely to be using getopt() if there are options that take more than one argument). That's mentioned in the answer. – Stéphane Chazelas May 23 '18 at 15:43
6

As thrig already mentioned in a comment:

zsh -c 'prog *.txt(P:-f:)'

This uses the glob qualifier P which prepends a separate word to each glob match. Like almost all of zsh's glob qualifiers, bash has nothing similar.

If you want to have completion for prog, run zsh interactively then type prog *.txt(P:-f:). The very first time you run zsh interactively, it'll prompt you to set a configuration. If you don't care about zsh, either pick the option to make a blank .zshrc (but then you'll miss on some zsh features that need to be activated explicitly) or pick 1 and go through the menus to activate the recommended defaults.

5

One. method to do this is with the GNU find/xargs duo:

cd important-files && \
find . -name '*.txt'  -printf '-f\0%p\0' | xargs -0 -r  prog
  • Note that this will change the working directory for prog, which may affect things like relative paths used by the program. – Kusalananda May 23 '18 at 6:08
  • 2
    Why are you changing the directory? Just do find important-files … instead. This’ll also avoid the problem mentioned by @Kusalananda. – Konrad Rudolph May 23 '18 at 11:25
  • To avoid hitting the arguments limit since the "important-files" token will be tacked to all filenames without yielding any info. – Rakesh Sharma May 23 '18 at 16:44
1

POSIXly, you can always define a function like:

prog_f() (
  for i do
    set -- "$@" -f "$i"
    shift
  done
  exec prog "$@"
)
  • Why are the subshell and exec needed here? – muru May 24 '18 at 2:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.