7

I have a long text file (a tab-file for stardict-editor) which consists of lines in the following format:

word1  some text
word1  some other text
word2  more text
word3  even more

and would like to convert it to

word1  some text<br>some other text
word2  more text
word3  even more

This means that subsequent lines (the file is sorted) which start with the same word should be merged to a single one (here the definitions are separated with <br>). Lines with equal beginning can also appear more often than just twice. The character which separates word and definition is a tab-character and is unique on each line. word1, word2, word3 are of course placeholders for something arbitrary (except tab and newline characters) which I don't know in advance.

I can think of a longer piece of Perl code which does this, but wonder if there is a short solution in Perl or something for the command line. Any ideas?

3

This is standard procedure for awk

awk '
{
  k=$2
  for (i=3;i<=NF;i++)
    k=k " " $i
  if (! a[$1])
    a[$1]=k
  else
    a[$1]=a[$1] "<br>" k
}
END{
  for (i in a)
    print i "\t" a[i]
}' long.text.file

If file is sorted by first word in line the script can be more simple

awk '
{
  if($1==k)
    printf("%s","<br>")
  else {
    if(NR!=1)
      print ""
    printf("%s\t",$1)
  }
  for(i=2;i<NF;i++)
    printf("%s ",$i)
  printf("%s",$NF)
  k=$1
}
END{
print ""
}' long.text.file

Or just bash

unset n
while read -r word definition
do
    if [ "$last" = "$word" ]
    then
        printf "<br>%s" "$definition"
    else 
        if [ "$n" ]
        then
            echo
        else
            n=1
        fi
        printf "%s\t%s" "$word" "$definition"
        last="$word"
     fi
done < long.text.file
echo
  • Looks good! Only when I run it the output does not contain any tab characters. There should be one between each word and its definition. – highsciguy Apr 1 '15 at 9:07
  • @highsciguy Have both script edited. – Costas Apr 1 '15 at 9:24
  • Costas, your code changes the data; not only the TAB (as already mentioned in a previous comment) but also sequences of spaces. This is likely undesired behaviour. – Janis Apr 1 '15 at 9:34
3
perl -p0E 'while(s/^((.+?)\t.*)\n\2\t/$1<br>/gm){}' 

(It takes 2s to process a 23MB, 1.5Mlines dictionary, in my 6years old laptop)

  • I can confirm this is much faster than the sed solution. For one file it decreased the execution time from about 8 minutes to under one second. – pcworld Jun 8 at 12:14
3

With sed:

sed '$!N;/^\([^\t]*\t\)\(.*\)\(\n\)\1/!P;s//\3\1\2<br>/;D' <<\IN
word1  some text
word1  some other text
word1  some other other text
word2  more text
word3  even more
word3  and still more
IN

(note: with many seds the above \t escape is invalid and a literal <tab> character should be used in its place)

And if you have GNU sed you can write it a little easier:

sed -E '$!N;/^(\S+\t)(.*)\n\1/!P;s//\n\1\2<br>/;D' <infile

It works by gradually stacking input as it is read. If two consecutive lines do not begin with the same not-space string, then the first of these is Printed. Else the intervening newline is relocated to the head of the line and the matched string immediately following it (to include the tab) is replaced w/ the string <br>.

Note that the stacking method used here could have performance implications if the line that sed assembles grows very long. If it grows any longer than 8kb then it will exceed the minimum pattern space buffer-size specified by POSIX.

Regardless of which of the two possibilities occurred, last of all sed Deletes up to the first occurring \newline character in pattern space and starts over with what remains. And so when two consecutive lines do not begin with identical strings then the first is printed and deleted, else the substitution is performed and the Delete only deletes the \newline which formerly separated them.

And so the command above prints:

word1  some text<br>some other text<br>some other other text
word2  more text
word3  even more<br>and still more

I used a <<\HERE_DOC for input above, but you should probably drop everything from <<\IN on and use </path/to/infile instead.

  • Sorry, What is option `sed -E'? – JJoao Apr 1 '15 at 14:20
  • 2
    @JJoao - sorry for what? The -E option to GNU sed is an undocumented alternative to using -r, except that 1. it makes more sense (what was -r ever supposed to mean anyway?), 2. It also works in BSD sed, 3. POSIX has a scheduled change due to be applied to the next version of the spec that officially blesses -E as the correct syntax to enable extended regular expressions in a sed. – mikeserv Apr 1 '15 at 14:58
2

This is indeed standard for awk. Here is a terse solution that doesn't change operational data:

awk 'BEGIN { FS="\t" }
     $1!=key { if (key!="") print out ; key=$1 ; out=$0 ; next }
     { out=out"<br>"$2 }
     END { print out }'
  • It is. Since the OP said that there is a unique TAB in between the "word" and the rest of the data. Mind the FS definition! – Janis Apr 1 '15 at 9:31
  • No. I assign the whole $0 (which contains the TAB) to out. – Janis Apr 1 '15 at 9:36
  • The two comments from me look a bit strange now because the one to whom I was replying to deleted his comments. To summarize; the essence is that the presented solution considers all requirements (including to keep data and TAB-separators intact). - That said; I wonder why there was (besides the upvote) also a downvote. Would the voter please add a rationale for that. – Janis Apr 1 '15 at 10:17
  • Seems to be the shortest, but for some reason it fails on many lines. Perhaps because the file contains a lot of utf-8 special characters? – highsciguy Apr 1 '15 at 10:19
  • With a not too old GNU awk you'd have UTF-8 supported. Is your locale appropriately set? (Something like LC_ALL=en_US.UTF-8.) Otherwise it would be helpful to get some of the sample lines where you see problems; it could also be that the data format is not everywhere as you expected it. Your feedback is appreciated, to track where in the chain of processing the problem lies; somewhere a fix will be necessary. – Janis Apr 1 '15 at 10:27
1

In python:

import sys

def join(file_name, join_text):
    prefix = None
    current_line = ''
    for line in open(file_name):
        if line and line[-1] == '\n':
            line = line[:-1]
        try:
            first_word, rest = line.split('\t', 1)
        except:
            first_word = None  # empty line or one without tab
            rest = line
        if first_word == prefix:
            current_line += join_text + rest
        else:
            if current_line:
                print current_line
            current_line = line
            prefix = first_word

    if current_line:  # do the last line(s)
        print current_line


join(sys.argv[2], sys.argv[1])

This expects the separator (<br>) as the first argument to the program and the file name as the second argument

-1

try

awk 'BEGIN { before="" } 
{ if ( $1 == before ) { $1="" ; printf "<br>%s",$0 ; } 
  else { printf "\n%s",$0 ;} ; before=$1 ; } 
END { printf "\n"  ;}'

which give with your input

word1  some text<br> some other text
word2  more text
word3  even more

tha awk basically remember the first word on previous line, and don't print newline.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.