I use a script say; filter1.sh which contains awk command;

bzip2 -dc File_1.tsv.bz2 | awk -F '\t' 'BEGIN {OFS=FS} { if (($7) > 50)  print $0 }'  > File_1.tsv

bzip2 -dc File_2.tsv.bz2 | awk -F '\t' 'BEGIN {OFS=FS} { if (($7) > 50)  print $0 }'  > File_2.tsv

to open bz2 file and filter the values in column 7 that are greater than 50 and and write the output to a new tsv file.

I have multiple files (around 200 which are located in different directories) for which I have to perform this filtering step (note exactly the same but similar).

My question here is how do I pass multiple files for this purpose, Is it possible to pass the files through separate INFILE (like in python).

Something like;

./filter1.sh pathtofiles.in

in command line. Where pathtofiles.in contains location of files.


and so on.

thanks in advance. ~M

  • 1
    Re awk-script awk -F'\t' '$7>50' > File_1.tsv is enough – Costas Jul 30 '15 at 15:18


while read f
    bzip2 -dc "$f" | 
    awk -F '\t' 'BEGIN{OFS=FS} $7 > 50' > "$(basename "$f" .bz2)"
done < list-of-file.txt
  1. awk will write result from File_1.tsv.bz2 to File_1.tsv, and so one with File_2, ... File_960.
  2. you give the list of file in list-of-file.txt
  3. > $(basename $f .bz2) will create result in local dir
|improve this answer|||||

You could get them all in a single output stream like:

bzip2 -dc ./File_*.bz2 | filter

To write them out to separate files on output you'd need to delimit the stream in some way. This would be the ideal solution if you could arrange it - and with more information you could get help in doing so.

Else you can call up your filters individually once per input file:

for j in ./File_*.bz2
do    [ -f "$j" ] &&
      bzip2 -dc "$j" |
      awk ... >"${j%.*}"

This is not ideal in that you have to call up a new instance of your pipeline per file, but may be a practical solution for you.

To source such a for loop in another script file and then to pass it the arguments you want it to work with you could do:

for j do
      case $j in
      (*.bz2) [ -f "$j" ] &&
              bzip2 -dc --  "$j" |
              awk ...   >"${j.*}"

...which is only slightly more complicated so that it might ensure all of its arguments are definitely named with a *.bz2 extension to avoid any nastiness as might surround passing in an argument with no extension., If written to an executable file named ./script on a system with a POSIX /bin/sh, it could be called like...

./script ./File_*.bz2
|improve this answer|||||

In bash, you can try this:

for f in `cat $1`
  bzip ...

This may not work in all shells, and there are undoubtedly other ways to accomplish this. $1 means "the first parameter passed to the script".

|improve this answer|||||
  • 1
    Watch out for any files with spaces in the names as f will get each part split on IFS separately. This is basically the don't parse ls problem – Eric Renouf Jul 30 '15 at 14:31
while read -r j
    [ -f "$j" ] && 
    bzip2 -dc "$j" |
    sed -r '/^([^\t]+\t){6}0*([1-9][0-9]{2,}|[6-9][0-9]|5[1-9])/! d' > "${j%.*}"
done <"$1"
|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.