I want to split a file of 10000 records into number of files with equal size of records.These new files need to be given as input to a shell script. Shell script should run in parallel for each file. Can we use any looping here?

  • 1
    splitting part of your question, you can refer to this question split file into N pieces with same name but different target directories. Nov 16 '17 at 4:58
  • 1
    Have a look at man split and have a try with option -l. If you can't succeed, edit the question to show what you tried and maybe give an easy example.
    – Philippos
    Nov 16 '17 at 7:01
  • Try parallel -a myfile --pipepart --block -1 wc or parallel -a myfile --pipepart --block -1 --cat wc or parallel -a myfile --pipepart --block -1 --fifo wc.
    – Ole Tange
    Nov 17 '17 at 0:08

Suppose your data file is called data.txt and the script you want to run is called script.sh. Then you could do something like the following:


# Create a temporary directory
splitdir="$(mktemp -d)"

# Splite the data-file into files of 1000 lines each
split --lines=1000 -d --suffix-length=3 data.txt "${splitdir}/chunk"

# Run your script on each data file separately
for chunk in "${splitdir}/"*; do nohup script.sh "${chunk}" &; done

You could also do something similar using xargs or GNU parallel instead of a Bash loop.

Not the answer you're looking for? Browse other questions tagged or ask your own question.