0

I would like to use bash tools to solve the following two related problems.

1. Suppose that I have a file with the following lines

A foo1
B bar
A foo2
A foo3 foo4
C ccc

How could I merge the lines that have the same starting field? In other words, I would like to get the following output:

A foo1 foo2 foo3 foo4
B bar
C ccc

2. After finishing the first job, there may be duplicated fields in each line. For example, foo2 may be identical to foo4. How can I remove the duplications and keep only one of them (i.e., keep only foo2)? This has to be done for each line, but we do not care about duplications across the lines.

Background: You may assume that I am compiling a dictionary. The starting field of each line is a word, and the following fields are its meanings.

2
$ awk '
    { for (i=2; i<=NF; i++) if (!seen[$1,$i]++) map[$1] = map[$1] OFS $i }
    END { for (key in map) print key map[key] }
' file
A foo1 foo2 foo3 foo4
B bar
C ccc

The above assumes you don't care about the order of the output lines.

0
0
cat - <<\eof > file
A foo1
B bar
A foo2
A foo3 foo2
C ccc
eof

awk '
{
  word = $1
  for (i=2; i<=NF; i++) {
    meaning = $i
    sep = (!(word in dict) ? "" : OFS)
    dict[word] = dict[word] \
     (!seen[meaning]++ ? sep meaning : "")
  }
}
END {
  for (word in dict)
    print word, dict[word]
}
' file

A foo1 foo2 foo3
B bar
C ccc
3
  • 1
    this removes duplicates say foo1 for rest of the words if they had and just keep for first word, for example if there were A foo1, B foo1, you will output A foo1, B – αғsнιη Mar 19 at 11:15
  • That was my understanding of the specs. This is what happens when OPs provide skeleton inputs and leave it to the responder to divine the expected interpretation. – guest_7 Mar 19 at 12:30
  • @guest_7 Thanks. As mentioned in the question, you may assume that I am compiling a dictionary. "A foo1, B" does not seem to be what a dictionary would look like. – Nuno Mar 20 at 4:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.