Join lines of text with repeated beginning

Question

I have a long text file (a tab-file for stardict-editor) which consists of lines in the following format:

word1  some text
word1  some other text
word2  more text
word3  even more

and would like to convert it to

word1  some text<br>some other text
word2  more text
word3  even more

This means that subsequent lines (the file is sorted) which start with the same word should be merged to a single one (here the definitions are separated with <br>). Lines with equal beginning can also appear more often than just twice. The character which separates word and definition is a tab-character and is unique on each line. word1, word2, word3 are of course placeholders for something arbitrary (except tab and newline characters) which I don't know in advance.

I can think of a longer piece of Perl code which does this, but wonder if there is a short solution in Perl or something for the command line. Any ideas?

Answer 1

This is standard procedure for awk

awk '
{
  k=$2
  for (i=3;i<=NF;i++)
    k=k " " $i
  if (! a[$1])
    a[$1]=k
  else
    a[$1]=a[$1] "<br>" k
}
END{
  for (i in a)
    print i "\t" a[i]
}' long.text.file

If file is sorted by first word in line the script can be more simple

awk '
{
  if($1==k)
    printf("%s","<br>")
  else {
    if(NR!=1)
      print ""
    printf("%s\t",$1)
  }
  for(i=2;i<NF;i++)
    printf("%s ",$i)
  printf("%s",$NF)
  k=$1
}
END{
print ""
}' long.text.file

Or just bash

unset n
while read -r word definition
do
    if [ "$last" = "$word" ]
    then
        printf "<br>%s" "$definition"
    else 
        if [ "$n" ]
        then
            echo
        else
            n=1
        fi
        printf "%s\t%s" "$word" "$definition"
        last="$word"
     fi
done < long.text.file
echo

Answer 2

perl -p0E 'while(s/^((.+?)\t.*)\n\2\t/$1<br>/gm){}' 

(It takes 2s to process a 23MB, 1.5Mlines dictionary, in my 6years old laptop)

Answer 3

With sed:

sed '$!N;/^\([^\t]*\t\)\(.*\)\(\n\)\1/!P;s//\3\1\2<br>/;D' <<\IN
word1  some text
word1  some other text
word1  some other other text
word2  more text
word3  even more
word3  and still more
IN

(note: with many seds the above \t escape is invalid and a literal <tab> character should be used in its place)

And if you have GNU sed you can write it a little easier:

sed -E '$!N;/^(\S+\t)(.*)\n\1/!P;s//\n\1\2<br>/;D' <infile

It works by gradually stacking input as it is read. If two consecutive lines do not begin with the same not-space string, then the first of these is Printed. Else the intervening newline is relocated to the head of the line and the matched string immediately following it (to include the tab) is replaced w/ the string <br>.

Note that the stacking method used here could have performance implications if the line that sed assembles grows very long. If it grows any longer than 8kb then it will exceed the minimum pattern space buffer-size specified by POSIX.

Regardless of which of the two possibilities occurred, last of all sed Deletes up to the first occurring \newline character in pattern space and starts over with what remains. And so when two consecutive lines do not begin with identical strings then the first is printed and deleted, else the substitution is performed and the Delete only deletes the \newline which formerly separated them.

And so the command above prints:

word1  some text<br>some other text<br>some other other text
word2  more text
word3  even more<br>and still more

I used a <<\HERE_DOC for input above, but you should probably drop everything from <<\IN on and use </path/to/infile instead.

Answer 4

This is indeed standard for awk. Here is a terse solution that doesn't change operational data:

awk 'BEGIN { FS="\t" }
     $1!=key { if (key!="") print out ; key=$1 ; out=$0 ; next }
     { out=out"<br>"$2 }
     END { print out }'

Answer 5

In python:

import sys

def join(file_name, join_text):
    prefix = None
    current_line = ''
    for line in open(file_name):
        if line and line[-1] == '\n':
            line = line[:-1]
        try:
            first_word, rest = line.split('\t', 1)
        except:
            first_word = None  # empty line or one without tab
            rest = line
        if first_word == prefix:
            current_line += join_text + rest
        else:
            if current_line:
                print current_line
            current_line = line
            prefix = first_word

    if current_line:  # do the last line(s)
        print current_line


join(sys.argv[2], sys.argv[1])

This expects the separator (<br>) as the first argument to the program and the file name as the second argument

Answer 6

try

awk 'BEGIN { before="" } 
{ if ( $1 == before ) { $1="" ; printf "<br>%s",$0 ; } 
  else { printf "\n%s",$0 ;} ; before=$1 ; } 
END { printf "\n"  ;}'

which give with your input

word1  some text<br> some other text
word2  more text
word3  even more

tha awk basically remember the first word on previous line, and don't print newline.