I am using paste to merge three text files (which do not require sorting) into a single document with three columns.

paste a.txt b.txt c.txt

I'd like elements that the columns have in common to occupy the same row without sharing it with non-matching elements (which they currently do). By the same token, unique elements should have their own rows. The elements in each column should preserve their original order.

Here's a simple example.

Input

1 1 1
2 2 2
3 4 4
5 5 5
1 1 2
3 3 3

Desired Output

1 1 1
2 2 2
3    
  4 4
5 5 5
1 1
    2
3 3 3

Here's a more complicated example.

Input

000 000 000
002 002 001
006 006 006
008 008 007
009 009 009
011 012 010
013 013 013
015 015 014
016 016 016
018 019 017
020 020 020
021 021 022
024 024 024
026 025 025
028 026 026
118 028 027
119 118 118
032 119 117
036 032 032
037 033 033
039 034 034
040 037 037
042 039 038
043 040 040
045 042 041
046 043 043
048 045 044
    046 046
    049 047

Desired Output

000 000 000
        001
002 002
006 006 006
        007
008 008 
009 009 009
        010
011        
    012 
013 013 013
        014
015 015 
016 016 016
        017
018     
    019 
020 020 020
021 021 
        022
024 024 024
    025 025
026 026 026
        027
028 028 
118 118 118
        117
119 119 
032 032 032
    033 033
    034 034
036     
037 037 037
        038
039 039 
040 040 040
        041
042 042 
043 043 043
        044
045 045 
046 046 046
        047
048     
    049

Ideally, I'd like to use tools that are built in to Linux/Unix. I'd also like the output to remain a single document with three columns, e.g., > whatever.csv.

The closest I've been able to get is to run sdiff on the original text files, but although that correctly aligns elements that the files share in common, it does not handle the differences as I would like.

  • You say that the original files do not require sorting. Would it be okay to sort them, individually? – Kusalananda Dec 5 at 18:11
  • It would be all right as long as each file/column retains its original order. – MilesO'Brien Dec 5 at 18:17
BEGIN {
    # We assume the default input field separator (changeable with "-F")
    # Output will be tab delimited.
    OFS = "\t"
}
{
    # The number of output records that this input record results in.
    k=0

    # "seen" records which new record a field should be part of.
    # There may be NF new records for each input record if all
    # fields are unique.
    delete seen

    # "a" holds all data for the new output records.
    # It's basically a 2-dimensional NFxNF matrix
    # encodod in a 1-dimensional array.
    delete a

    # Iterate over the fields
    for (i=1; i<=NF; ++i) {
        if (!seen[$i]) {
            # This data has not been seen before (in this input record),
            # assign it to the next output line.

            seen[$i] = ++k
        }

        # Assign the input field to the right spot
        a[(seen[$i]-1)*NF + i] = $i
    }

    # Save NF as this is reset by emptying $0 later.
    nf = NF

    # Create and output new lines
    for (j = 1; j<=k; ++j) {
        $0 = ""

        # Create new output record
        for (i = 1; i<=nf; ++i)
            $i = a[(j-1)*nf + i]

        # Output record
        print
    }
}

Testing on the given data:

$ awk -f script.awk file
1       1       1
2       2       2
3
        4       4
5       5       5
1       1
                2
3       3       3

Testing on other data:

$ cat file
a b c e
1 2 1 1
2 1 1 1
1 1 1 2
$ awk -f script.awk file
a
        b
                c
                        e
1               1       1
        2
2
        1       1       1
1       1       1
                        2
  • This works on my example and on most of my data; consequently, it is the answer. It does not work with some data, doubtless because I have made an assumption of some kind. When I work out what that is, it is probably better to ask a new question that to edit the original post. – MilesO'Brien Dec 5 at 19:49
  • @MilesO'Brien It's either the handling of the input data, or your interpretation of the output data. It would be interesting to see a lite of data that you believe it does not handle. – Kusalananda Dec 5 at 21:08
  • To be clear, your script does what I described in the OP on all data that I feed it. It's just that I erred in my assumptions about the data and how to handle it. Now, I'm unsure whether I should start a new thread or add to the OP. – MilesO'Brien Dec 5 at 22:18
  • I have updated the OP with a more complicated example. On reflection, I have decided that the way in which I worded the OP is compatible with both examples. – MilesO'Brien Dec 6 at 13:35
  • @MilesO'Brien Ah, so it's the order in which the new lines are added that is wrong, I get it. I will mull over this for a while and make a change later today (I'm a bit busy with other work ATM). – Kusalananda Dec 6 at 14:45

Here's a "brute force" solution in a shell script using paste and read.

#!/bin/sh

paste a.txt b.txt c.txt |
while read -r a b c; do
    if [ "$a" = "$b" ] && [ "$b" = "$c" ]; then
        printf '%s\t%s\t%s\n' "$a" "$b" "$c"
    elif [ "$a" = "$b" ]; then
        printf '%s\t%s\n\t\t%s\n' "$a" "$b" "$c"
    elif [ "$a" = "$c" ]; then
        printf '%s\t\t%s\n\t%s\n' "$a" "$c" "$b"
    elif [ "$b" = "$c" ]; then
        printf '%s\n\t%s\t%s\n' "$a" "$b" "$c"
    else
        printf '%s\n\t%s\n\t\t%s\n' "$a" "$b" "$c"
    fi
done

There's probably a more elegant solution but I couldn't come up with a good one off the top of my head.

You could probably use awk instead if you prefer it -- I think the result would look very similar. (One advantage of using awk would be that it could potentially do the job of paste at the same time, if that's useful to you.)

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.