1

I have a large number of files, all with the same format.

line 1: Gene ID
line 2: chromosomal position
line 3 - x: names of genetic variants)

I want to select only files containing at least 5 variants (i.e. files that have at least 10 lines in total). If a file has at least 5 variants I want to write the content minus the first two rows to a new file. Below I provide two example input files, foo1 and foo2.

foo1:

echo {885743,4:139381:3783883,rs93487,rs82727,rs111} | tr " " "\n" > foo1

foo2:

echo {10432,1:3747548:2192993,rs10204,rs262222,rs436363,rs3636,rs9878,rs11856} | tr " " "\n" > foo2

Desired output file (in this case only 1 file, in reality there will be multiple separate output files): foo2.checked, looking like:

rs10204
rs262222
rs436363
rs3636
rs9878
rs11856

2 Answers 2

2

assuming no file with funny char in their name

 for file in *
 do
    line=$(wc -l < "$file' )
    if [ $line -ge 10 ]
    then
       tail -n +3 <"$file" > "${file}.checked"
    fi
 done

this basically count line in every file, then if over 10, print all lines, starting at the third.

1
 # for each file in the current directory you can refine the ls command to match 
 # only the files you want. or if in a script file pass in the file list 
 for file in *
 do
    # if the file has more than 10 lines.
    if (( $(<"${file}" wc -l) > 10 )); then
       # print line 3 to end of file and pipe it to a file with the same
       # name as the input file with the added .checked at the end.
       sed -n '3,$p' -- "${file}" > "${file}.checked"
    fi
 done
2
  • 1
    Thanks! I replaced ls -1 by fnames.txt containing a list of filenames, to prevent the command from being executed on other (non relevant) files in the directory.
    – mats
    Sep 8, 2015 at 13:57
  • Archemar's solution was pretty close too. :) Glad we could help.
    – Rob
    Sep 8, 2015 at 14:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .