2

I have a list of one column that contain ~ 100 lines, in which some lines are repeated, and my purpose is to get rid of a specific duplicate lines and leave only one copy, while the other lines kept untouched.

An extract of the files that I'm working on :

V(Mn9)   
V(C1,H3) 
V(Mn6)   
V(Mn6)   
V(C4,H6) 
V(Mn9)   
V(Mn9)   
V(C1,Mn6)
V(C4,Mn9)
V(Mn6)   
V(C1,C4) 
C(Mn9)   
C(Mn6)   
C(C1)    
C(C4)    
C(Mn9)   
C(Mn6)   
V(C1,H2) 
V(Mn9)   
V(Mn6)   
V(C4,H5)

My purpose is to remove all the duplicate lines contain C(Xx0-9) and leave one of them and kept the V(Xxx..).

The result I seek :

V(Mn9)   
V(C1,H3) 
V(Mn6)   
V(Mn6)   
V(C4,H6) 
V(Mn9)   
V(Mn9)   
V(C1,Mn6)
V(C4,Mn9)
V(Mn6)   
V(C1,C4) 
C(C1)    
C(C4)    
C(Mn9)   
C(Mn6)   
V(C1,H2) 
V(Mn9)   
V(Mn6)   
V(C4,H5)

I used the command :

sed '0,/C(Mn9)/{/C(Mn9)/d}' inputfile.txt | sed '0,/C(Mn6)/{/C(Mn6)/d}'

and it's working, but it's not good enough for the whole file, because there is a lot of C(Xx1-50), I thinked to use regular expression, but I don't know how, that's why I need your help.

5
  • You want to remove the duplicates occurring first or last?
    – Inian
    Nov 3, 2021 at 6:37
  • Can you mix the order of lines? Can you sort the file before removing duplicates?
    – nobody
    Nov 3, 2021 at 7:16
  • It doesn't matter wich occurrence, the important thing is to keep one occurrence of each C(Mn1-50), without touching the others. Nov 3, 2021 at 8:58
  • double check your desired output and specify if you wand the first unique C(XxN) of if you want the second. ie.. when should it be printed? the first time it is seen? or when it is detected as duplicate? Nov 4, 2021 at 16:51
  • @ЯрославРахматуллин doesn't matter wich one, the important thing is to keep one match. i.e. one C(Cl8) and one C(Cl7), and of course the other duplicate keep it untouched. Nov 4, 2021 at 23:06

2 Answers 2

0
$ awk '!(/^C\(..[0-9])$/ && seen[$0]++)' file
V(Mn9)
V(C1,H3)
V(Mn6)
V(Mn6)
V(C4,H6)
V(Mn9)
V(Mn9)
V(C1,Mn6)
V(C4,Mn9)
V(Mn6)
V(C1,C4)
C(Mn9)
C(Mn6)
C(C1)
C(C4)
V(C1,H2)
V(Mn9)
V(Mn6)
V(C4,H5)

The above assumes you have no white space before/after the visible characters in your sample input. If you do then remove them, e.g.:

$ awk '{gsub(/^[[:space:]]+|[[:space:]]+$/,"")} !(/^C\(..[0-9])$/ && seen[$0]++)' file
V(Mn9)
V(C1,H3)
V(Mn6)
V(Mn6)
V(C4,H6)
V(Mn9)
V(Mn9)
V(C1,Mn6)
V(C4,Mn9)
V(Mn6)
V(C1,C4)
C(Mn9)
C(Mn6)
C(C1)
C(C4)
V(C1,H2)
V(Mn9)
V(Mn6)
V(C4,H5)
8
  • This solution does not take into account the order of the C(XxN) values, and produces output that is different from the desired result as specified in the question. It does not matter though, because the order is not important for the author of the question. Nov 4, 2021 at 17:49
  • The author already stated in a comment that the output order doesn't matter. You can't expect them to provide every possible variation of output order in the question and it's pointless commenting on or editing every answer to say it has the desired behavior. if you feel it's not clear from the question that the order of output lines doesn't matter then edit the question (or ask the question author to clarify).
    – Ed Morton
    Nov 4, 2021 at 17:51
  • Strictly speaking your answer provides a solution without any explanation of how it works. Nov 4, 2021 at 17:52
  • 1
    You win. I'm moving on. Nov 4, 2021 at 17:56
  • 1
    @ЯрославРахматуллин the solution given by Ed Morton is quite good. Nov 4, 2021 at 23:11
0

I suggest to use sed to collect lines in the hold space to check whether they appeared before:

 sed -n 'H;G;/^\(C([^)]*)\).*\1 *\n/!P'
  • H appends the current line to the hold space
  • G appends the hold space with all lines we ever saw to the pattern space
  • C([^)]*) is one of those C(…) patterns, the ^ anchors it to the beginning of the line and it's surrounded by \(…\), so it can be backreferenced as \1 later. We need \1 *\n as pattern, with the newline (after possible whitespaces) to avoid matching the freshly appended line at the end. So the whole pattern /^\(C([^)]*)\).*\1 *\n/ matches a line with a duplicate C(…), so only if this ! doesn't match,
  • Print everything before the first newline (= without the appended hold space), while default output is suppressed by the -n option

Note that depending on you sed version and file size, this may fail because over the time, all lines will be in memory.

2
  • Sorry, but your command doesn't give any result. Nov 3, 2021 at 9:18
  • I tested it on the file you provided … try by yourself by c&P ing your data to a file. Is it possible your actual file has some hidden characters like tabs or carriage returns (from DOS line endings)?
    – Philippos
    Nov 3, 2021 at 16:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.