2

I wonder what is a simpler way to do this:

awk 'NR > 1 {print $1"\t"$2"\t"$3"\t"$4"\t"$5"\t"$6"\t"$7"\t"$8"\t"$9$10$11$12$13$14$15$16}'  file.in > file.out

which is simply speaking " concatenate columns 9 to 16 by removing tabs in-between"

Merged columns 9-16 become "Notes" so may include whitespaces.

As of today there are 16 columns but this may evolve in more/less if required. Eventually column 9 (concatenated 9-16) becomes "notes" field.

Cheers,
Xi

  • Can the input file contain white spaces? – LatinSuD Jul 28 '16 at 15:25
  • Yes, merged columns 9-16 become "Notes" so may include whitespaces. – xi100f Jul 28 '16 at 15:29
  • 2
    Then how are "columns" defined? Your awk solution fails if your fields can contain whitespace. Is your input file tab-separated? – terdon Jul 28 '16 at 15:31
  • Yes it is tab separated. Sorry, I thought that including "\t" between columns 1-9 reveals that this is tab separated file. – xi100f Jul 28 '16 at 15:37
  • 1
    @spasic yes, but if one of the fields contains whitespace, it will be considered two fields. Compare printf 'one\ttwo a b\tthree' | awk '{print $2}' and printf 'one\ttwo a b\tthree' | awk -F"\t" '{print $2}' – terdon Jul 28 '16 at 15:51
5
paste <(cut -f 1-8 file) <(cut -f9- file | tr -d '\t')
  • Worked like a charm. I love simplicity and portability of this. – xi100f Jul 29 '16 at 8:04
  • Nice an simple, can be a problem for big files though, as sed and awk work per line, whilepaste needs to buffer the contents in RAM. But that should be a rare limitation. – Fiximan Jul 29 '16 at 8:05
  • 1
    @Fiximan Are you sure? As far as I can tell, GNU paste only buffers a single line from each open file. – Michael Vehrs Jul 29 '16 at 8:40
  • @MichaelVehrs Not 100%, I haven't looked into the source code - however I did experience problems with paste on two large files (with long lines, too), where an analogous awk script did not fail. Long time ago and GB-sized files, so I don't have the details anymore, but I came out with the assumption that this was the reason. Might have been a wrong guess though. – Fiximan Jul 29 '16 at 9:12
3

Assuming a few lines of tably separated values, generated thusly:

% perl -E 'say join "\t", 1..8 for 1..3'

The various columns can then be dealt with as necessary via the appropriate flags and variables and functions available in Perl.

% perl -E 'say join "\t", 1..8 for 1..3' \
| perl -pale '$_=join "\t", @F[0..3], join "", @F[4..7] if $. > 1' 
1   2   3   4   5   6   7   8
1   2   3   4   5678
1   2   3   4   5678
% 
2

Yep, many ways. I have tested the following two on a file created by:

perl -le 'next if $.==1; for(1..20){print join "\t",1..20 }' > file

That's a file with 20 lines and 20 tab-separated columns.

  1. Perl

    perl -F'\t' -ale '$"="\t";print "@F[0..7]",@F[8..$#F]' file 
    

    Note that this joins all the fields from the 10th to the end. If you only want to join 9 to 16, use this instead:

    perl -F'\t' -ale '$"="\t"; print "@F[0..7]", @F[8..15], "\t@F[16..$#F]"' file 
    
  2. awk

    awk -F'\t' 'NR>1{
                    for(i=1;i<9;i++){
                        printf "%s\t",$i
                    }
                    for(i=9;i<=NF;i++){
                        printf "%s",$i
                    }print "" 
                }' file 
    

    As before, this will join all columns after the 10th. If you only want to join 9 to 16, use this instead:

    awk -F'\t' 'NR>1{
                    for(i=1;i<9;i++){
                        printf "%s\t",$i
                    }
                    for(i=9;i<=16;i++){
                        printf "%s",$i
                    }
                    for(i=17;i<=NF;i++){
                        printf "\t%s", $i
                    }
                    print ""
                }' file 
    

Granted, the awk solutions aren't very short, but at least you don't need to specify all the fields by hand.

2

Python alternative

$ cat file | python -c "import sys
for line in sys.stdin: l=line.rstrip('\r\n').split('\t'); print('\t'.join(l[:9]) + ''.join(l[9:]))
"

sed alternative

s/(([^\t]*\t){8})/\1\n/
h
s/[^\n]*\n//
s/\t//g
G
s/([^\n]*)\n([^\n]*)\n.*/\2\1/

Usage example:

$ sed -r "s/(([^\t]*\t){8})/\1\n/;h;s/[^\n]*\n//;s/\t//g;G;s/([^\n]*)\n([^\n]*)\n.*/\2\1/" file

Explanation:

Suppose that file is

a   b   c   d   e   f   g   h   i   j   k   l

The separator in file is tab.

  1. sed reads the current line.

    • pattern space is a b c d e f g h i j k l
  2. s/(([^\t]*\t){8})/\1\n/ splits the line in two parts.

    • pattern space is a b c d e f g h \ni j k l
  3. h stores pattern space in hold space.

    • pattern space is a b c d e f g h \ni j k l
    • hold space is a b c d e f g h \ni j k l
  4. s/[^\n]*\n// removes the first part in pattern space.

    • pattern space is i j k l
    • hold space is a b c d e f g h \ni j k l
  5. s/\t//g removes tabs in pattern space.

    • pattern space is ijkl
    • hold space is a b c d e f g h \ni j k l
  6. G appends \n and hold space to the pattern space.

    • pattern space is ijkl\na b c d e f g h \ni j k l
    • hold space is a b c d e f g h \ni j k l
  7. s/([^\n]*)\n([^\n]*)\n.*/\2\1/ splits the pattern space and replace it with the second and the first parts without \n.

    • pattern space is a b c d e f g h ijkl
    • hold space is a b c d e f g h \ni j k l
  8. sed prints the pattern space.

It is possible do modify the code and remove tabs in the middle columns:

s/(([^\t]*\t){3})(([^\t]*\t){4})(.*)/\1\n\3\n\5/
h
s/[^\n]*\n([^\n]*)\n.*/\1/
s/\t//g
G
s/([^\n]*)\n([^\n]*)\n([^\n]*)\n(.*)/\2\1\4/
2

With GNU sed (assuming the fields are tab delimited in the input):

sed 's/\t//9g'

Deletes the 9th to last tab characters on each line.

If there are more than 16 fields in the input and you don't want the extra ones:

cut -f 1-16 | sed 's/\t//9g'
1

awk:

awk -v OFS="\t" '{for (i=10; i<=NF; i++) $9 = $9 " " $i; NF = 9; print}' file

Fields 10 to the end are appended to field 9, then the number of fields is limited to the first 9, and the line is printed using tab as the output field separator.

I assume you want the concatenated fields to be separated with a space.
If not, change $9 " " $i to $9 $i

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.