-1

I have text file in which I want to search for duplicate occurrences of New Cell lines

New Cell
New Cell
3250,36, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,16.3, 0.0,16.4,20.1,21.8,23.3,25.2,26.4,27.4,26.9,26.4, 0.0, 0.0, 0.0, 0.0, 0.0,
3251,36, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,18.3,23.5,25.3,25.4,27.5,26.0,28.6,28.7,31.8,33.2,34.5,35.0,36.4,38.9,37.9,38.9,38.2,41.3, 0.0, 0.0, 0.0, 0.0, 0.0,
3251,37, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,19.1,21.9,18.0,22.7,21.2,25.3,26.4,29.8,30.3,30.9,31.4,32.3,32.4,31.2,31.3,30.0,27.6,28.1,28.4, 0.0, 0.0, 0.0, 0.0, 0.0,
3252,36, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,18.4,17.1,20.5,22.9,24.8,21.5,24.9,23.7,26.6,27.2,28.3,27.9,28.4,28.5,30.0,31.0,30.6,31.2,31.6,30.3, 0.0, 0.0, 0.0, 0.0, 0.0,
New Cell

I have to remove duplicate line while retaining the original. My output should look like

New Cell
3250,36, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,16.3, 0.0,16.4,20.1,21.8,23.3,25.2,26.4,27.4,26.9,26.4, 0.0, 0.0, 0.0, 0.0, 0.0,
3251,36, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,18.3,23.5,25.3,25.4,27.5,26.0,28.6,28.7,31.8,33.2,34.5,35.0,36.4,38.9,37.9,38.9,38.2,41.3, 0.0, 0.0, 0.0, 0.0, 0.0,
3251,37, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,19.1,21.9,18.0,22.7,21.2,25.3,26.4,29.8,30.3,30.9,31.4,32.3,32.4,31.2,31.3,30.0,27.6,28.1,28.4, 0.0, 0.0, 0.0, 0.0, 0.0,
3252,36, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,18.4,17.1,20.5,22.9,24.8,21.5,24.9,23.7,26.6,27.2,28.3,27.9,28.4,28.5,30.0,31.0,30.6,31.2,31.6,30.3, 0.0, 0.0, 0.0, 0.0, 0.0,
New Cell
1

You need the uniq command. It does exactly what you need. Type man uniq for help.

uniq «input-file» «output-file»

should do it. However this may occasionally clobber some of the numbers.

Using sed

With sed we can say remove duplicates, only if matching pattern.

sed -r -e '$!N; /^(«test to remove duplicates of»)\n\1$/!P; D'

Explanation:

  • run sed in extended regex mode -r, expression is next -e
  • single quotes ' tell shell not to interpret this, just pass it on (to sed).
  • $!N if not last line of file, then append it to pattern space.
  • `;' is sed's command separator
  • /^(«test to remove duplicates of»)\n\1$/
    • /regex/ says what to search for.
    • ^ starts with
    • (something) something to search for, and remember it.
    • \n a newline character.
    • \1 the first thing I asked you to remember.
    • $ End
    • ! negate the search.
    • P print, if search matched.
    • D Delete up to newline and repeat.
  • Yes i tried with uniq. It works well for New Cell. But I guess it will also change the numbers – geet Nov 20 '18 at 11:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.