Here's a little sed
script that I actually wrote for myself not too long ago. I have just had a little fun updating it, though. It does the whole job by itself:
cdup() { _u= _d=
case "${1#-}" in (U) _u='\)\(';; (D) _d='\
'; _d="$_d\\2$_d";; (*) ! :;;esac && shift
sed 's/ */ /g;H;1h;1d;x;:t
s/ *\(.*\(\n\)\)\([^ ]\{1,\}\) */\2\3 \1/;tt
s/ / /g;h;$!d;s/.*/ & /;:n
/\( \([^ ]\{1,\}\) \)\(.*'"$_u\1${_d:+.*}\)/{
s//\3${_d:- }"'/;s/$\n*//;tn
}; s/.* \n\n*//;s/ *//;s// /g
s/\n\n/ /g;y/ \n/\n /' "$@"
unset -v _u _d
}
sed
works two lines at a time at rearranging the fields in its input to align by column - and stacks its work in the hold buffer between each line. It doesn't delimit on anything other than the original space delimiter in your sample (and I originally wrote it to handle $IFS
separated arg arrays) - and so, provided that delimiter is solid, fields of any reasonable length containing most any character but the delimiter should all work just as well.
So it does (L1COL1\nL2COL1) (L1COL2\nL2COL2)...((L[12]C1)\nL3COL1)...
for as long as it must until it encounters the last line. By the time it does it has already so neatly arranged all of the data in its memory that it is a trivial matter to check for duplicates - and so it prints columns only once no matter how many times they appear in input:
cdup <<\COLS
1 A 4 Z 1
2 B 3 Y 2
3 C 2 X 3
4 D 1 W 4
5 E 0 U 5
COLS
OUTPUT
A B C D E
4 3 2 1 0
Z Y X W U
1 2 3 4 5
But with the -U
flag it prints only unique items so...
cdup -U <<\COLS
1 A 4 Z 1
2 B 3 Y 2
3 C 2 X 3
4 D 1 W 4
5 E 0 U 5
COLS
...gets...
A B C D E
4 3 2 1 0
Z Y X W U
Or -D for duplicates only, with an extra record per duplicate column appearance.
Not so bad...
cdup -D <<\DATA
1 1 A A 4 Z 1
2 2 B B 3 Y 2
3 3 C C 2 X 3
4 4 D D 1 W 4
5 5 E E 0 U 5
DATA
1 2 3 4 5
1 2 3 4 5
A B C D E