Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have multiple files with the same header and different vectors below that. I need to concatenate all of them but I want only the header of first file to be concatenated and I don't want other headers to be concatenated since they are all same.

for example: file1.txt

<header>INFO=<ID=DP,Number=1,Type=Integer>
<header>INFO=<ID=DP4,Number=4,Type=Integer>
A
B 
C

file2.txt

<header>INFO=<ID=DP,Number=1,Type=Integer>
<header>INFO=<ID=DP4,Number=4,Type=Integer>
D
E 
F

I need the output to be

<header>INFO=<ID=DP,Number=1,Type=Integer>
<header>INFO=<ID=DP4,Number=4,Type=Integer>
A
B
C
D
E 
F

I could write a script in R but I need it in shell?

share|improve this question

4 Answers 4

up vote 1 down vote accepted

If you know how to do it in R, then by all means do it in R. With classical unix tools, this is most naturally done in awk.

awk '
    FNR==1 && NR!=1 { while (/^<header>/) getline; }
    1 {print}
' file*.txt >all.txt

The first line of the awk script matches the first line of a file (FNR==1) except if it's also the first line across all files (NR==1). When these conditions are met, the expression while (/^<header>/) getline; is executed, which causes awk to keep reading another line (skipping the current one) as long as the current one matches the regexp ^<header>. The second line of the awk script prints everything except for the lines that were previously skipped.

share|improve this answer
    
Thanks Gilles. Each of my files are in GBs. R won't be efficient do this. That's why I asked. –  Jana Jan 8 '13 at 17:54
    
@Jana Are there lines that look like headers but aren't at the top of the file? If not, the fastest way is to use grep (like in sputnik's answer). –  Gilles Jan 8 '13 at 18:39
    
No the header lines are similar to all files and they are just at the top of each file. Yeah grep was faster. Thanks both of you –  Jana Jan 8 '13 at 21:54
1  
@Jana By the way, if all your files have the same number of header lines, here's another way (which I expect to be even faster): head -n 10 file1.txt >output.txt && tail -q -n +11 file*.txt >>output.txt (if you have 10 header lines). Also, if your files have numbers in their names, beware that file9.txt is sorted between file89.txt and file90.txt. If your files have numbers like file001.txt, …, files009.txt, files010.txt, …, then files*.txt will list them in the right order. –  Gilles Jan 8 '13 at 22:01

Try doing this :

$ cat file1.txt; grep -v "^<header" file2.txt
<header>INFO=<ID=DP,Number=1,Type=Integer>
<header>INFO=<ID=DP4,Number=4,Type=Integer>
A
B 
C
D
E 
F

NOTE

  • the -v flag means to invert the match of
  • ^ in REGEX, means beginning of the string
  • if you have a bunch of files, you can do

:

array=( files*.txt )
{ cat ${array[@]:0:1}; grep -v "^<header" ${array[@]:1}; } > new_file.txt

It's a array slicing technique.

share|improve this answer
    
Thanks sputnick, But I have ~30 files(file1.txt,file2.txt,file3.txt..filen.txt) to be concatenated. Should I type every file name or is there any other ways to do it? –  Jana Jan 7 '13 at 21:53
    
See my edited post with slicing technique –  sputnick Jan 7 '13 at 22:02
    
Thanks sputnick –  Jana Jan 7 '13 at 22:18
    
This removes <header> lines anywhere in the files, not just at the beginning. This may not be an issue here, depending on the data. –  Gilles Jan 8 '13 at 18:39
1  
Simpler: grep '^<header>' file1.txt >output.txt && grep -v '^<header>' file*.txt >>output.txt –  Gilles Jan 8 '13 at 18:41

Another solution, similar to "cat+grep" from above, using tail and head:

  1. Write the header of the first file into the output:

    head -2 file1.txt > all.txt
    

    -- head -2 gets 2 first lines of the file.

  2. Add the content of all the files:

    tail -n +3 -q file*.txt >> all.txt
    

    -- -n +3 makes tail print lines from 3rd to the end, -q tells it not to print the header with the file name (read man), >> adds to the file, not overwrites it as >.

And sure you can put both commands in one line:

head -2 file1.txt > all.txt; tail -n +3 -q file*.txt >> all.txt

or instead of ; put && between them for success check.

share|improve this answer

The tail command (on GNU, at least) has an option to skip a given number of initial lines. To print from the second line onward, i.e. skip a one-line header, do: tail -n+2 myfile

So, to keep the two-line header of the first file but not the second, in Bash:

cat file1.txt <(tail -n+3 file2.txt) > combined.txt

Or, for many files:

head -n1 file1.txt > combined.txt
for fname in *.txt
do
    tail -n+3 $fname >> combined.txt
done

If a certain string is known to be present in all header lines but never in the rest of the input files, grep -v is a simpler approach, as sputnik showed.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.