I have a long text file (a tab-file for stardict-editor) which consists of lines in the following format:

word1  some text
word1  some other text
word2  more text
word3  even more

and would like to convert it to

word1  some text<br>some other text
word2  more text
word3  even more

This means that subsequent lines (the file is sorted) which start with the same word should be merged to a single one (here the definitions are separated with <br>). Lines with equal beginning can also appear more often than just twice. The character which separates word and definition is a tab-character and is unique on each line. word1, word2, word3 are of course placeholders for something arbitrary (except tab and newline characters) which I don't know in advance.

I can think of a longer piece of Perl code which does this, but wonder if there is a short solution in Perl or something for the command line. Any ideas?

6 Answers 6

perl -p0E 'while(s/^((.+?)\t.*)\n\2\t/$1<br>/gm){}' 

(It takes 2s to process a 23MB, 1.5Mlines dictionary, in my 6years old laptop)

  • I can confirm this is much faster than the sed solution. For one file it decreased the execution time from about 8 minutes to under one second.
    – pcworld
    Commented Jun 8, 2019 at 12:14

This is standard procedure for awk

awk '
  for (i=3;i<=NF;i++)
    k=k " " $i
  if (! a[$1])
    a[$1]=a[$1] "<br>" k
  for (i in a)
    print i "\t" a[i]
}' long.text.file

If file is sorted by first word in line the script can be more simple

awk '
  else {
      print ""
    printf("%s ",$i)
print ""
}' long.text.file

Or just bash

unset n
while read -r word definition
    if [ "$last" = "$word" ]
        printf "<br>%s" "$definition"
        if [ "$n" ]
        printf "%s\t%s" "$word" "$definition"
done < long.text.file
  • Looks good! Only when I run it the output does not contain any tab characters. There should be one between each word and its definition.
    – highsciguy
    Commented Apr 1, 2015 at 9:07
  • @highsciguy Have both script edited.
    – Costas
    Commented Apr 1, 2015 at 9:24
  • Costas, your code changes the data; not only the TAB (as already mentioned in a previous comment) but also sequences of spaces. This is likely undesired behaviour.
    – Janis
    Commented Apr 1, 2015 at 9:34

This is indeed standard for awk. Here is a terse solution that doesn't change operational data:

awk 'BEGIN { FS="\t" }
     $1!=key { if (key!="") print out ; key=$1 ; out=$0 ; next }
     { out=out"<br>"$2 }
     END { print out }'
  • It is. Since the OP said that there is a unique TAB in between the "word" and the rest of the data. Mind the FS definition!
    – Janis
    Commented Apr 1, 2015 at 9:31
  • No. I assign the whole $0 (which contains the TAB) to out.
    – Janis
    Commented Apr 1, 2015 at 9:36
  • The two comments from me look a bit strange now because the one to whom I was replying to deleted his comments. To summarize; the essence is that the presented solution considers all requirements (including to keep data and TAB-separators intact). - That said; I wonder why there was (besides the upvote) also a downvote. Would the voter please add a rationale for that.
    – Janis
    Commented Apr 1, 2015 at 10:17
  • Seems to be the shortest, but for some reason it fails on many lines. Perhaps because the file contains a lot of utf-8 special characters?
    – highsciguy
    Commented Apr 1, 2015 at 10:19
  • With a not too old GNU awk you'd have UTF-8 supported. Is your locale appropriately set? (Something like LC_ALL=en_US.UTF-8.) Otherwise it would be helpful to get some of the sample lines where you see problems; it could also be that the data format is not everywhere as you expected it. Your feedback is appreciated, to track where in the chain of processing the problem lies; somewhere a fix will be necessary.
    – Janis
    Commented Apr 1, 2015 at 10:27

With sed:

sed '$!N;/^\([^\t]*\t\)\(.*\)\(\n\)\1/!P;s//\3\1\2<br>/;D' <<\IN
word1  some text
word1  some other text
word1  some other other text
word2  more text
word3  even more
word3  and still more

(note: with many seds the above \t escape is invalid and a literal <tab> character should be used in its place)

And if you have GNU sed you can write it a little easier:

sed -E '$!N;/^(\S+\t)(.*)\n\1/!P;s//\n\1\2<br>/;D' <infile

It works by gradually stacking input as it is read. If two consecutive lines do not begin with the same not-space string, then the first of these is Printed. Else the intervening newline is relocated to the head of the line and the matched string immediately following it (to include the tab) is replaced w/ the string <br>.

Note that the stacking method used here could have performance implications if the line that sed assembles grows very long. If it grows any longer than 8kb then it will exceed the minimum pattern space buffer-size specified by POSIX.

Regardless of which of the two possibilities occurred, last of all sed Deletes up to the first occurring \newline character in pattern space and starts over with what remains. And so when two consecutive lines do not begin with identical strings then the first is printed and deleted, else the substitution is performed and the Delete only deletes the \newline which formerly separated them.

And so the command above prints:

word1  some text<br>some other text<br>some other other text
word2  more text
word3  even more<br>and still more

I used a <<\HERE_DOC for input above, but you should probably drop everything from <<\IN on and use </path/to/infile instead.

  • Sorry, What is option `sed -E'?
    – JJoao
    Commented Apr 1, 2015 at 14:20
  • 2
    @JJoao - sorry for what? The -E option to GNU sed is an undocumented alternative to using -r, except that 1. it makes more sense (what was -r ever supposed to mean anyway?), 2. It also works in BSD sed, 3. POSIX has a scheduled change due to be applied to the next version of the spec that officially blesses -E as the correct syntax to enable extended regular expressions in a sed.
    – mikeserv
    Commented Apr 1, 2015 at 14:58

In python:

import sys

def join(file_name, join_text):
    prefix = None
    current_line = ''
    for line in open(file_name):
        if line and line[-1] == '\n':
            line = line[:-1]
            first_word, rest = line.split('\t', 1)
            first_word = None  # empty line or one without tab
            rest = line
        if first_word == prefix:
            current_line += join_text + rest
            if current_line:
                print current_line
            current_line = line
            prefix = first_word

    if current_line:  # do the last line(s)
        print current_line

join(sys.argv[2], sys.argv[1])

This expects the separator (<br>) as the first argument to the program and the file name as the second argument



awk 'BEGIN { before="" } 
{ if ( $1 == before ) { $1="" ; printf "<br>%s",$0 ; } 
  else { printf "\n%s",$0 ;} ; before=$1 ; } 
END { printf "\n"  ;}'

which give with your input

word1  some text<br> some other text
word2  more text
word3  even more

tha awk basically remember the first word on previous line, and don't print newline.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .