5

I want to remove lines which exist already in the previous line from the command line in UNIX. I have the following data in a file.

 <xref id="gi_525506931_ref_NP_001266519.1__brain_aromatase"/>
                    <entry ac="IPR002401" desc="Cytochrome P450, E-class, group I" name="Cyt_P450_E_grp-I" type="FAMILY">
                    <entry ac="IPR001128" desc="Cytochrome P450" name="Cyt_P450" type="FAMILY">
                    <entry ac="IPR001128" desc="Cytochrome P450" name="Cyt_P450" type="FAMILY">
                    <entry ac="IPR001128" desc="Cytochrome P450" name="Cyt_P450" type="FAMILY">

So, it should look like this:

 <xref id="gi_525506931_ref_NP_001266519.1__brain_aromatase"/>
                    <entry ac="IPR002401" desc="Cytochrome P450, E-class, group I" name="Cyt_P450_E_grp-I" type="FAMILY">
                    <entry ac="IPR001128" desc="Cytochrome P450" name="Cyt_P450" type="FAMILY">
  • 2
    Welcome to U&L. We are (often) a pretty forgiving bunch, if you make mistakes we'll tell you or correct them. You can review the changes that I made to your post, just click on the link above my (or any follow up editor's) avatar. – Anthon Nov 5 '14 at 5:12
  • Hi Anthon, I am not able to accept two answers although cuonglm's answer also gave me required result. Also, I am not able to upvote the answers. – Prakki Rama Nov 5 '14 at 6:57
  • 1
    Upvoting comes with 15 reputations, wait for it ..., there you go, should have it – Anthon Nov 5 '14 at 6:58
8

If you can guarantee that the identical lines will be consecutive, you can use

uniq your_file

and cuonglm's answer will work even if they're not.

  • Yes. Thats what I wanted. – Prakki Rama Nov 5 '14 at 5:10
  • 1
    A line is normally consecutive with the previous line (as in the title of the question) ;-) – Anthon Nov 5 '14 at 5:15
  • @Anthon You're right. But, hey, completeness never hurts :) (Incompleteness, on the other hand,...) – Joseph R. Nov 5 '14 at 5:21
3

With awk:

awk '!a[$0]++' file

An explanation for this idiom you can read here.

  • 1
    This also removes doubles that are the same with entries that are not the previous line, the title states "same as the previous line" – Anthon Nov 5 '14 at 5:17
  • 2
    ...cool! Maybe adding a bit of explanation? ;-) – Rmano Nov 5 '14 at 10:03
  • I second that emotion, @Rmano. Currently indistinguishable from magic :p – Svish Nov 5 '14 at 13:37
  • @Rmano and Svish: I add link to explanation. – cuonglm Nov 5 '14 at 15:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.