Can anyone help how to compare two consecutive lines from a text file for the first character and if both the first characters are same ignore/delete the second line and print into new file only the first line.


I want to delete #1002.

  • Your last line also starts with the same First Char. What do you want to do with that?
    – neuron
    Jul 22 '15 at 10:07
  • next line after #1003 is not starts with # so i want next line to be as it is
    – keshav
    Jul 22 '15 at 10:11
  • What about when there are 3 (or more) matches in a row?
    – Peter.O
    Jul 22 '15 at 12:20

Using GNU uniq with the -w option1:

  -w, --check-chars=N
          compare no more than N characters in lines

you could omit consecutive duplicate lines, comparing just the first character:

uniq -w1 infile >outfile

1. This won't work properly for multi-byte characters. See Stéphane's note below

  • Note that in the case of GNU uniq (my version 8.23 at least), that help message is misleading as, in effect, it compares N bytes, not N characters. So that won't work properly for multi-byte characters. Jul 22 '15 at 11:31
  • AFAIK, BSD doesn't support -w. At least FreeBSD doesn't. That answer is still the best GNU answer. The fact that it doesn't work with GNU uniq now (for multi-byte characters which is not the case in the OP's sample) is a bug there. Hopefully, it will be fixed. May have been already in newer versions. I haven't checked. Jul 22 '15 at 12:00
  • @don_crissti I think this is a good enough answer that it would be a shame if you delete it. Just mentioning the restriction to single-byte characters in the answer itself makes it perfectly fine.
    – David Z
    Jul 22 '15 at 14:06
  • 1
    surely it works with two consecutive lines...
    – mikeserv
    Jul 22 '15 at 14:35
perl -C -ne '$c = substr($_,0,1);
             print unless $c eq $l;
             $l = $c;' < file.in > file.out
sed '$n;h;N;/^\(.\).*\n\1/g;/\n/P;//!G;D' <in >out

There's a sed script that will do it.

It works like this:

  1. If current line is the last, print it to standard out and end the script.
  2. If not, save a copy of current line to hold space.
  3. Then append the Next input line to pattern space.
  4. If the first char in pattern space is identical to the first char in the line just appended, overwrite pattern space with the saved hold-space copy.
  5. If not, then we'll still have a\newline in pattern space, in which case we should Print only up to that point.
  6. If not, Get another copy of our held line appended to pattern space.
  7. Regardless, Delete up to the first newline in pattern space and start over from the top of the script with what remains.

Essentially, it works two lines at a time, Printing only the first when the first char in the second line doesn't match that of the first line, and recursively overwrites those which do with the first occurrence in a series. And so it can quickly and capably handle input of any size with minimal buffering and squeeze a first char match series of any size to only its first occurrence. It will work with real-time input or else with a file, but it doesn't ever need to read or store any more than a copy of the current line and the next.

use warnings;
use strict;

my $first_line = <>;
my $second_line = <>;

if (substr($first_line, 0, 1) eq substr($second_line, 0, 1)) {
     print $first_line;
} else {
     # You didn't say what to do if the character are different.
perl -0777 -pe '1 while s/^(.)(.*)\n\1.*/$1$2/gm' file.in >file.out

That slurps the entire file, and loops until no more matches can be found.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.