1

I have a large number of files, all with the same format.

line 1: Gene ID
line 2: chromosomal position
line 3 - x: names of genetic variants)

I want to select only files containing at least 5 variants (i.e. files that have at least 10 lines in total). If a file has at least 5 variants I want to write the content minus the first two rows to a new file. Below I provide two example input files, foo1 and foo2.

foo1:

echo {885743,4:139381:3783883,rs93487,rs82727,rs111} | tr " " "\n" > foo1

foo2:

echo {10432,1:3747548:2192993,rs10204,rs262222,rs436363,rs3636,rs9878,rs11856} | tr " " "\n" > foo2

Desired output file (in this case only 1 file, in reality there will be multiple separate output files): foo2.checked, looking like:

rs10204
rs262222
rs436363
rs3636
rs9878
rs11856
1
 # for each file in the current directory you can refine the ls command to match 
 # only the files you want. or if in a script file pass in the file list 
 for file in *
 do
    # if the file has more than 10 lines.
    if (( $(<"${file}" wc -l) > 10 )); then
       # print line 3 to end of file and pipe it to a file with the same
       # name as the input file with the added .checked at the end.
       sed -n '3,$p' -- "${file}" > "${file}.checked"
    fi
 done
  • 1
    Thanks! I replaced ls -1 by fnames.txt containing a list of filenames, to prevent the command from being executed on other (non relevant) files in the directory. – mats Sep 8 '15 at 13:57
  • Archemar's solution was pretty close too. :) Glad we could help. – Rob Sep 8 '15 at 14:06
2

assuming no file with funny char in their name

 for file in *
 do
    line=$(wc -l < "$file' )
    if [ $line -ge 10 ]
    then
       tail -n +3 <"$file" > "${file}.checked"
    fi
 done

this basically count line in every file, then if over 10, print all lines, starting at the third.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.