0

I have a python script that takes a path where a bunch of text files are located to process them somehow. Since there are too many files I want to use batches using a bash script to pass just some of the files on the path, say 100 at a time. Is there a simple way to do this. So for example my scripts is currently

python application.py -fp [path to all files]

Can I do a bash script where I do something like

python application.py -fp [file-1:file-100]

and on the next loop

python application.py -fp [file-101:file-200]

and so on?

Edit:

I tried Stephane solution with bash and I think it almost works but I'm still having trouble getting just a subset of the files

I do this to get the path from the parameters given to the bash script

set -- "$fp*.txt"
echo "${@}"

the result is

../../files_test/pair/*.txt

which is correct since that is the path of the files I need to get. But then I do

files=${@:1:2}
echo $files

just to test if I can get the first file but it echos the list of all files in the directory. Am I missing something?

Edit 2:

Nevermind. I realized I was doing

set -- "$fp*.txt"

instead of

set -- $fp*.txt

Now it works.

1
  • 1
    You'd probably want set -- "$fp"/*.txt. That is make sure $fp is quoted and * is not. Commented Sep 4, 2019 at 12:57

1 Answer 1

2

With GNU xargs and a shell with process substitution support (ksh, bash, zsh), you can do:

xargs -r0 -n100 -a <(printf '%s\0' ./*) python application.py -fp

Example:

$ xargs -r0n4 -a <(printf '%s\0' {1..20}) echo
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20

Without process substitution, you can also do:

printf '%s\0' ./* | xargs -r0 -n100 -python application.py -fp

But that means application.py's stdin will be /dev/null which on systems with /dev/fd/xxx you can work around by basically implementing process substitution by hand with:

{
  printf '%s\0' ./* |
    xargs -a /dev/fd/3 3<&0 <&4 4<&- -r0 -n100 -python application.py -fp
} 4<&0

With zsh:

autoload zargs
zargs -l 100 ./* -- python application.py -fp

Example:

$ zargs -l4 {1..20} -- echo
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20

You can also always do (ksh93/bash/zsh):

set -- ./*
while (($# > 0)); do
  python application.py -fp "${@:1:100}"
  shift "$(($# >= 100 ? 100 : $#))"
done

Example:

$ set -- {1..20};while (($#>0));do echo "${@:1:4}";shift "$(($#>4?4:$#))";done
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
17 18 19 20

If your files are actually called file-1, file-2... you'll probably want to use zsh and its n (for numeric sorting) glob qualifier for the list of files to be sorted numerically:

zargs -l 100 ./*(n) -- python application.py -fp

Or use GNU sort -zV (for version sort) on the output of printf '%s\0':

xargs -r0 -n100 -a <(printf '%s\0' ./* | sort -zV) python application.py -fp
3
  • Thanks so much for this very complete answer. I tried the one for bash since I can't assure the shell will have process substitution support but it's not working as expected. Can you look into the edit and tell me what I'm doing wrong. Thanks a lot
    – Atirag
    Commented Sep 4, 2019 at 0:21
  • 1
    @Atirag, bash does support process substitution. Process substitution and ((...)) are both ksh features copied by bash. While ${@:offset:length} is a ksh93 feature also copied by bash. Of the 3 features, the latter is the least portable (outside of ksh93/zsh/bash). Commented Sep 4, 2019 at 13:01
  • oh that's good to know. Thank you. So maybe I'll adapt it to other solution
    – Atirag
    Commented Sep 4, 2019 at 13:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .