1

I have following data in a unix flatfile and I want to suppress duplicate values in first two columns and convert it as shown below:

1:x:4:3:2:y
1:x:7:9:l:z
1:0:3:j:k:m
2:9:r:s:6:u
2:m:y:5:7:9
2:u:7:9:7:6
3:a:b:c:d:e
3:a:b:d:e:f
3:a:b:n:r:s

To:

1:x:4:3:2:y
 : :7:9:l:z
 :0:3:j:k:m
2:9:r:s:6:u
 :m:y:5:7:9
 :u:7:9:7:6
3:a:b:c:d:e
 : :b:d:e:f
 : :b:n:r:s
  • 1
    What have you tried so far and where are you stuck? – Freddy Oct 17 '19 at 5:56
  • I might help you if you are allowed to use Ruby programming language... Is that allowed? – S.Goswami Oct 17 '19 at 5:57
  • @s.goswami yes pls go ahead – Naga Oct 17 '19 at 6:02
  • bash-4.4$ which ruby /usr/bin/ruby bash-4.4$ ruby -v ruby 1.9.3p551 (2014-11-13 revision 48407) [sparcv9-solaris2.11] bash-4.4$ – Naga Oct 17 '19 at 6:03
  • 1
    Are you only concerned with consecutive duplicates, or would an x in column 2 in a line further down also be deleted? – Kusalananda Oct 17 '19 at 7:22
3

Straightforwardly with awk:

$ awk -F':' '++a[$1] > 1{ $1=" " }++b[$2] > 1{ $2=" " }1' OFS=':' inp_file
1:x:4:3:2:y
 : :7:9:l:z
 :0:3:j:k:m
2:9:r:s:6:u
 :m:y:5:7:9
 :u:7:9:7:6
3:a:b:c:d:e
 : :b:d:e:f
 : :b:n:r:s
  • bash-4.4$ awk -F':' '++a[$1] > 1{ $1=" " }++b[$2] > 1{ $2=" " }1' OFS=':' Array_e.log awk: syntax error near line 1 awk: bailing out near line 1 bash-4.4$ – Naga Oct 17 '19 at 6:35
  • @Naga, post your awk version – RomanPerekhrest Oct 17 '19 at 7:04
  • that will also show an extra line like 3:x:b:n:r:s at the end with x being a duplicate of the a above it. I'm not sure that's what's intended -- it's hard to say based on the OP. – mosvy Oct 17 '19 at 9:15
  • @RomanPerekhrest . how to find awk version ? awk --version hangs bash-4.4$ which awk /usr/bin/awk bash-4.4$ /usr/bin/awk --version – Naga Oct 17 '19 at 10:01
  • @Naga, and what's the output of /usr/bin/awk --version ? – RomanPerekhrest Oct 17 '19 at 10:03
0

As @Naga mentioned in the comments that a Ruby script is allowed. Here I have came up with a solution.

Ruby Script

#!/usr/bin/env ruby
# Frozen_String_Literal: false
$-v = nil
d, $-s, i, $,, $; = [], ?\s.freeze, '', ?:.freeze, ?:.freeze
::FILENAME = $*[0] || abort("Pass the filename as Argument.\nExample:\n\t#{File.basename($0)} hello.rb")

puts IO.readlines(File.join(Dir.pwd, FILENAME)).map! { |x|
    c = x.tap(&:strip!).split
    d.clear && i.replace(c[0].to_s) if i != c[0]
    a = c.first(2).map { |y| d.find { |z| z.include?(y) } ? $-s : y }.join << $, << c.drop(2).join unless c.empty?
    d << c.first(2).join
    a
} rescue exit!

Running

ruby scriptname.rb inputfile.txt

Output

[Assuming inputfile.txt contains the same data in the question]

1:x:4:3:2:y
 : :7:9:l:z
 :0:3:j:k:m
2:9:r:s:6:u
 :m:y:5:7:9
 :u:7:9:7:6
3:a:b:c:d:e
 : :b:d:e:f
 : :b:n:r:s

Hope this helps!

  • ruby script is working fine, but i have challenge 1:x,a:4:3:2:y 1:x,1:7:9:l:z 1:0,2:3:j:k:m 2:9:r:s:6:u 2:m:y:5:7:9 2:u,1:7:9:7:6 3:a:b:c:d:e 3:a,2:b:d:e:f 3:a,1:b:n:r:s – Naga Oct 17 '19 at 10:03
  • is that space splitted instead of new line characters? – S.Goswami Oct 17 '19 at 15:45
  • I have edited the answer so that if the given file doesn't exist (or for any other error), it will exit with return value 1. Also, the script will not raise errors for invalid files... – S.Goswami Oct 17 '19 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.