pipe function arguments to command line arguments in python

Question

I have a python code (cmd.py) which accepts files as command line arguments and process using parse_args. I want to pass files from another code (files.py) to cmd.py, just like passing function arguments.
Can this be done without modifying cmd.py?

Answer 1

Yes it can be done:

cmd.py `files.py`

Then first files.py is executed and the output (Example: "file1.txt file2.txt") is the input for the cmd.py.

Example:

cat `ls`

This "cats" all the files found by ls in the working directory.

Answer 2

There are a couple of things that can be done here. First is simple command substitution:

cmd.py $(files.py)

One problem here is that if there are any spaces, tabs or newlines in filename then these filenames won't be passed as a single argument and instead split into different arguments where these characters occur.

One alternative is, if cmd.py outputs each file on a separate line, you can set the shell IFS to only contain a newline:

IFS='
'
cmd.py $(files.py)

This means that only filenames which contain a newline are a problem, cutting down the number of cases where it will fail.

Another problem is what happens if there are a larger number of files output from files.py. It may be that the command line becomes longer than the system can handle (the limiting factor is when the total size of the command line plus some other stuff totals ARG_MAX on Linux). The xargs program is designed for this situation and if the command line is too long for a single run of a program, it will split it into different runs. If this would help, you can do:

files.py | xargs cmd.py

Of course this also has potential for problems since it expects filenames that contain problematic characters to be quoted/escaped as you would when entering their names directly in the shell. One option is to change cmd.py so that the output is appropriately quoted, however some versions of xargs can be made to accept newline separated filenames similarly to setting IFS above. If the -d option is supported (this is with GNU xargs on Linux):

files.py | xargs -d '\n' cmd.py

Finally, perhaps the best way (now supported by most xargs implementations) is if you can have the input files separated by a null rather than a newline. This will easily handle any filename without introducing complex quoting since null is the only character not allowed in a Unix filename:

files.py | xargs -0 cmd.py