0

I have a python program that requires 2 arguments input-file and output-file. It reads input-file and creates a copy of that file with some modifications in the output-file.

I created a text file with a set of arguments. It has multiple sets of arguments on multiple lines. But for 1 execution of a program it needs to read line by line. So each line has 2 args (separated by a space) to be passed: input-file and output-file.

Example of file with arguments (call it File-With-Multiple-Arguments-Per-Line.txt):

input-file-1-path/input-file-1-name.sql output-file-1-path/output-file-1-name.sql 
input-file-2-path/input-file-2-name.sql output-file-2-path/output-file-2-name.sql
input-file-3-path/input-file-3-name.sql output-file-3-path/output-file-3-name.sql
input-file-4-path/input-file-4-name.sql output-file-4-path/output-file-4-name.sql

I tried this:

for arg in $(< /Users/Repository/File-With-Multiple-Arguments-Per-Line.txt); 
    do python /Users/Git-Repos/MyProgram.py "$arg" "$arg" ;
done. 

Usage is such: python MyProgram.py input-file.txt output-file.txt

I does what need. Though for some reason it processes each argument 3 times (so, 1st arg is input-file is also treated both as output-file; same with 2nd arg). So I end up modifying the input file, which I should not. I know this as both Input and Output files are 2 separate git Repositories.

I've also used only one "$arg" but then I'm seeing that my program does not receive a required 2nd argument (output-file). Instead of my output-file I see output: <stdout> But using "$arg" "$arg" leads to incorrect interpretation of my program. As if does the same task 3 times - first time does what is intended; second time using input-file as output and input; third time using output-file as output and input.

I've tried the solution below as well, but then my program does not do anything. Though it displays the output in my terminal as if it did what was supposed to be done. And no error appears.

For reference: Reading multiple arguments for a shell script from file

while read x y; do
    python /Users/Git-Repos/MyProgram.py "$x" "$y"
done < /Users/Repository/File-With-Multiple-Arguments-Per-Line.txt
1
  • Try adding an echo in front of your python to confirm that the correct arguments are being passed to python. Also, if your Python script is not using the second argument to open a file and write to it, but instead writing the output to stdout, then you might want to do this: python /Users/Git-Repos/MyProgram.py "$x" > "$y". Otherwise, the likely issue is inside the python script.
    – sudocracy
    Nov 3, 2023 at 5:08

2 Answers 2

1

Running a command for each line of some file with the whitespace-separated words in those lines passed as arguments to the command is what xargs -L1 does:

So, it would be just:

xargs -L1 <File-With-Multiple-Arguments-Per-Line.txt python MyProgram.py

As xargs understands some form of quoting, it can be used for arbitrary file paths¹ including ones with whitespace or even newline characters. Note however that the quoting syntax is different from that of modern sh². Both '...' and "..." are strong quotes and cannot contain newline characters. For instance you could have:

tamed-input-file tamed-output-file
"Input: it's "'such a "cool" path'  'Output: Not as cool'\
'as one with'\
'newline characters'
$ xargs -L1 <a printf '1:<%s> 2:<%s>\n'
1:<tamed-input-file> 2:<tamed-output-file>
1:<Input: it's such a "cool" path> 2:<Output: Not as cool
as one with
newline characters>

Note that depending on the xargs implementations, pythons stdin will be either opened read-only on /dev/null or shared with that of xargs (so be coming from File-With-Multiple-Arguments-Per-Line.txt as well), so it's important that python doesn't read from its stdin. With the GNU implementation of xargs, you can use:

xargs -L1 -a File-With-Multiple-Arguments-Per-Line.txt python MyProgram.py

to work around it.

Also note that with most implementations of xargs, if File-With-Multiple-Arguments-Per-Line.txt is empty, the command is still run once without arguments. With a few xargs implementations, that can be avoided with the -r option.


About your:

for arg in $(< /Users/Repository/File-With-Multiple-Arguments-Per-Line.txt); 
    do python /Users/Git-Repos/MyProgram.py "$arg" "$arg" ;
done

First note that except in zsh, that unquoted $(<...) is subject to globbing in addition to IFS-splitting, so you'd need a set -o noglob beforehand to avoid problems with filenames containing wildcard character.

Then, in each pass of the loop, you're getting one IFS-delimited ($IFS containing space, tab and newline (and NUL in zsh) by default) and passing that same word twice to python. To get two words at a time, you'd need zsh and:

for file1 file2 in $(<File-With-Multiple-Arguments-Per-Line.txt); do
  python MyProgram.py $file1 $file2
done

Or in bash:

set -o noglob
set -- $(<File-With-Multiple-Arguments-Per-Line.txt)
while (( $# > 0 )); do
  python MyProgram.py $1 $2
  shift "$(( $# < 2 ? $# : 2 ))"
done

That means the file paths can't contain space, tab nor newline characters though.

To do something similar to xargs -L1 and pass all the $IFS-delimited word from each line, in zsh:

while IFS= read -ru3 line; do
  python MyProgram.py $=line 3<&- # $=line for explicit IFS-splitting
done 3< File-With-Multiple-Arguments-Per-Line.txt

Or:

while read -ru3 -A files; do
  python MyProgram.py $files 3<&-
done 3< File-With-Multiple-Arguments-Per-Line.txt

(would also work in ksh93 and bash except you need "${files[@]}" instead of $files there (also works in zsh), and -a instead of -A in bash).

POSIXly:

set -o noglob
while IFS= read -r <&3 line; do
  python MyProgram.py $line 3<&- # implicit split+glob with glob disabled above
done 3< File-With-Multiple-Arguments-Per-Line.txt

That still doesn't allow arbitrary file paths. For a splitting of lines into words that understand some form of quoting like xargs -L1 does, with zsh:

while IFS= read -ru3 line; do
  python MyProgram.py "${(Q@)${(z)line}}" 3<&- # $=line for explicit IFS-splitting
done 3< File-With-Multiple-Arguments-Per-Line.txt

Where the z parameter expansion flag does the same tokenisation as in the zsh language, Q removes one layer of quoting, and @ within quotes preserves empty arguments (like "$@" does).

Then you can use:

"Input: it's "'such a "cool" path' $'Output: Not as cool\nas one with\nnewline characters'

¹ as long as they're not too long and with some xargs implementations, that they can be decoded as valid text in the user's locale.

² It's closer to that of the Mashey shell (aka PWB shell, xargs being introduced in PWB Unix in the late 70s).

1
  • Thanks a lot @Stéphane Chazelas! this worked perfectly fine: xargs -L1 <File-With-Multiple-Arguments-Per-Line.txt python MyProgram.py Nov 5, 2023 at 15:20
0

It appears that you are trying to loop through the contents of a file that has the input file name and output file name on each line and then invoke a python script passing the two filenames as arguments.

Assuming that the problem is not with the python script or permissions to read the file or the existence of files, the while example in your question should work.

But, here's an alternative to using while (this only shows the arguments being passed to Python; but does not run it yet):

§ cat File-With-Multiple-Arguments-Per-Line.txt | xargs -n 2 echo python 
python input-file-1-path/input-file-1-name.sql output-file-1-path/output-file-1-name.sql
python input-file-2-path/input-file-2-name.sql output-file-2-path/output-file-2-name.sql
python input-file-3-path/input-file-3-name.sql output-file-3-path/output-file-3-name.sql
python input-file-4-path/input-file-4-name.sql output-file-4-path/output-file-4-name.sql

You can pipe that output to a file like so, and then run it later which is helpful to verify that you are passing the arguments correctly:

§ cat File-With-Multiple-Arguments-Per-Line.txt | xargs -n 2 echo python > script.sh
§ chmod +x ./script.sh
§ ./script.sh

You can also invoke it directly as follows by piping the script to bash to execute:

§ cat File-With-Multiple-Arguments-Per-Line.txt | xargs -n 2 echo python | bash -

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .