4

Let say I have a text file in this format

field1a
field2a
field3a
field1b
field2b
field3b

I want to club 3 (or in general case N) consecutive lines, how will I do it with sed or other command line utility in bash shell?

expected output

field1a:field2a:field3a
field1b:field2b:field3b
11
 paste -sd '::\n' file

​​​​​​​ Or:

 awk '{ORS=NR%3?":":"\n";print}' < file

(note the difference if the number of records in the input is not a multiple of 3 though).

| improve this answer | |
  • 1
    The number of colons in the paste command determines the number of clubbed together entries, that was not clear to me on first sight. – Anthon Nov 6 '14 at 11:39
  • 1
    You shouldn't redirect the file into awk as it takes files as arguments anyway. This can also cause problems( although i cannot find the thread again it was something to do with how it is interpreted on OSX). Also it could be rewritten as awk 'ORS=NR%3?":":RS' file as this will always equate to true. – user78605 Nov 6 '14 at 14:35
  • 1
    @Jidder, it's rather the other way round (file name passed to awk) that's a problem (like for files with = in their name). – Stéphane Chazelas Nov 6 '14 at 14:43
  • 1
    @Jidder, if awk < file is processed differently from awk file, then that'd be a bug in the particular awk implementation, or we're talking non-POSIX systems. While the awk ... file==x is a problem with any awk implementation (can be worked around with awk ... ./file==x though). – Stéphane Chazelas Nov 6 '14 at 15:25
  • 1
    @fedorqui. True, at the cost of legibility though. I also generally avoid using assignments as conditions, because then you need to be extra careful that the assigned value never resolves to false (see KasiyA's answer for instance ($1=$1) for a case where that's a problem). – Stéphane Chazelas Nov 7 '14 at 10:56
4

With paste:

paste -d: - - - <file

Merge a file by pasting the data into 3 columns(- - -) using a colon separator:

With Perl:

perl -pe 'if($.%3){s/\n/:/;}' file

with -p option does default printing. All we do here is, if the line number is modulus of 3(%3), replace the newline character(\n) with a colon(:).

With xargs and awk:

xargs -L 3 < file | awk '$1=$1' OFS=:

The -L argument in xargs tells how many lines to join. And awk puts the output field separator(OFS) from Space(by default) to colon(:) separator.

With awk:

awk 'NR%3{printf "%s:",$0;next}{print;}' file

Print the modulus of 3 lines using printf(no new line) with a colon, and do a normal print for the next line using print(which puts newline by default).

reference

| improve this answer | |
  • 1
    xargs's -L doesn't do what you think it does. It won't work if the input contains blanks or quotes, or some of the fields resolve to 0. See for instance on the output of printf '%s\n' 0 a b 1 '2 3' 4 '5\' 6. Also note that it would run an echo process for every 3 args. – Stéphane Chazelas Nov 6 '14 at 23:05
  • @StéphaneChazelas I don't understand what do you mean? Do you mean if input file has a blank line between two another lines? – αғsнιη Nov 6 '14 at 23:17
  • 2
    -L 3 doesn't pass 3 lines as arguments to the command, it passes the words (words being blank separated, possibly quoted) found in those 3 lines as arguments to the command. The $1=$1 condition will return false if $1 resolves to 0. Try the example I suggested. – Stéphane Chazelas Nov 7 '14 at 10:07
  • 1
    I'd give up on xargs. You'd need something like (export LC_ALL=C; sed 's/['\''"\]/\\&/g' < "$file" | xargs -n3 printf '%s:%s:%s\n') which could still be a problem for long lines or inputs with number of lines not multiple of 3, and would also run many processes. – Stéphane Chazelas Nov 7 '14 at 12:19
2

For output formatting you can use printf

IFS='
'       # split on sequences of newline characters
set -f # disable globbing
printf "%s:%s:%s\n" $(cat file)

(note that it skips empty lines).

Or sed (if you like)

sed '$!N;$!N;s/\n/:/g' file
| improve this answer | |
  • @StéphaneChazelas If you expect the number of lines in the input is not a multiple or 3 and want the last line in a:b\n form you can use sed '{N;s/\n/:/g};{N;s/\n/:/g}' – Costas Nov 6 '14 at 13:58
2

With sed:

sed '$!N;$!N;y/\n/:/'

​​​​​​​​Though I guess that is less general than N lines than it should be. If the number of lines you want will always be at the end of the field like that then:

sed '$q;N;/1.$/!s/\(..*\)\(\n\)/\2\1:/;//P;D
' <<\INPUT 
field1a
field1b
field2b
field1c
INPUT

OUTPUT

field1a
field1b:field2b
field1c

...which stacks lines following one that ends with 1 then any single character until it encounters another also ending with a 1 then something else.

It would work for any number of occurring fields, but you may want to do...

sed '.../[^0-9]1.$/!...'

... if you get into multiple-digits.

| improve this answer | |
0

If I have just a few hundred or fewer lines, or if the number of fields per group varies, then vi works pretty well for me.

In vi, if I want to join the current and following 2 lines (3 lines of text in all), then ...

    3J

At this point, my cursor is on the concatenated line, so I step down to the next one ...

    j

I do the same modification relative to the current line by using the dot ...

    .

And again ...

    j
    .

Oops! That last group had 4 lines, not 3. I undo the latest modification ...

    u

and do it again, right this time ...

    4J

Next! ...

    j

... And so forth.

Note, if you intend to do this for your 360-million-field database load, or if your consecutive fields are usually 3, with a few groups of 4 and a maybe a rare group of 5 or 6, then you will probably want to endure the arcane syntax of taking yourself out of the loop, conditioning your data by steps, and maybe writing a program to test it and alert you for errors.

But if it's a small one-time job, then I just bear a few minutes of tedium and use vi.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.