Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Using sed, how can I remove duplicate letters from HEADERS within a text file?

NNAAMMEE
       nice - run a program with modified scheduling priority

SSYYNNOOPPSSIISS
       nice     [-n    adjustment]    [-adjustment]    [--adjustment=adjustment] [command [a$

Above is a an example. I want the output after parsing with sed to be:

NAME
       nice - run a program with modified scheduling priority

SYNOPSIS
       nice     [-n    adjustment]    [-adjustment]    [--adjustment=adjustment] [command [a$
share|improve this question
    
So only two and no more consecutive instances of a letter to be converted to one? –  1_CR Sep 10 '13 at 21:39
3  
The example looks like a "broken" man page. How did you come to that? –  michas Sep 10 '13 at 21:39
    
With perl you could do: perl -pe 's/([A-Z])\1/$1/g' –  michas Sep 10 '13 at 21:42
    
Thanks all for feedback. @1_CR Specifically just the headers, and not the rest of the text. –  user46865 Sep 10 '13 at 23:43
    
@michas It's just an example I found online. I'm trying to learn how to use sed effectively. –  user46865 Sep 10 '13 at 23:44

5 Answers 5

up vote 3 down vote accepted

Method #1

You can use this sed command to do it:

$ sed 's/\([A-Za-z]\)\1\+/\1/g' file.txt

Example

Using your above sample input I created a file, sample.txt.

$ sed 's/\([A-Za-z]\)\1\+/\1/g' sample.txt 
NAME
       nice - run a program with modified scheduling priority

       SYNOPSIS
              nice     [-n    adjustment]    [-adjustment] [--adjustment=adjustment] [comand [a$

Method #2

There is also this method which will remove all the duplicate characters:

$ sed 's/\(.\)\1/\1/g' file.txt 

Example

$ sed 's/\(.\)\1/\1/g' sample.txt 
NAME
    nice - run a program with modified scheduling priority

    SYNOPSIS
       nice   [-n  adjustment]  [-adjustment] [-adjustment=adjustment] [comand [a$

Method #3 (just the upper case)

The OP asked if you could modify it so that only the upper case characters would be removed, here's how using a modified method #1.

Example

$ sed 's/\([A-Z]\)\1\+/\1/g' sample.txt 
NAME
       nice - run a program with modified scheduling priority

       SYNOPSIS
              nice     [-n    adjustment]    [-adjustment] [--adjustment=adjustment] [command [a$

Details of the above methods

All the examples make use of a technique where when a character is first encountered that's in the set of characters A-Z or a-z that it's value is saved. Wrapping parens around characters tells sed to save them for later. That value is then stored in a temporary variable that you can access either immediately or later on. These variables are named \1 and \2.

So the trick we're using is we match the first letter.

\([A-Za-z]\)

Then we turn around and use the value that we just saved as a secondary character that must occur right after the first one above, hence:

\([A-Za-z]\)\1.

In sed we're also making use of the search and replace facility, s/../../g. The g means we're doing it globally.

So when we encounter a character, followed by another one, we substitute it out, and replace it with just one of the same character.

share|improve this answer
    
Thank you. Rremoving all duplicates isn't what I want to do; e.g., "command" should stay "command" but "NNAAMMEE" should become "NAME" is there a way to use sed with 's/(.)\1/\1/g' but only for double-capitals? –  user46865 Sep 10 '13 at 23:43
    
@JonSmith - you could take the first example and remove the a-z bit from the set of characters. sed 's/\([A-Z]\)\1\+/\1/g' –  slm Sep 10 '13 at 23:45
    
To do it in just headers, add /^[A-Z]/ - sed '/^[A-Z]/s/\(.\)\1/\1/g' –  Kevin Sep 11 '13 at 1:28
    
Can you explain what the parenthesis are for? How is [A-Z] different from ([A-Z]) Also, what is the + operator for? The \1 is for pattern matching, correct? –  user46865 Sep 11 '13 at 5:26
    
@slm I knew only basic of sed, this answer is amazing, but can you please explain. like what is meaning of \1 –  Rahul Patil Sep 12 '13 at 16:13

This command removes all double letters:

sed 's/\([[:alpha:]]\)\1/\1/g'

\1 stands for the text inside \(…\), so this command means: wherever there's an alphabetical character followed by itself, replace by that alphabetical character alone.

That will transform e.g. command into comand. It would be better to restrict the transformation to where it's needed: non-indented lines.

sed '/^[[:alpha:]]/ s/\([[:alpha:]]\)\1/\1/g'

This text is a man page rendered for terminals where bold is represented by overstrike: C\bC is rendered as bold, where \b is the backspace character (character number 8, also known as ^H). If the control characters are still there, forget about duplicate letters and instead remove the overstrike.

sed -e 's/.\b//g'

If you have a way to format the output, transofmr C\bC to bold and _\bC to underline.

sed -e 's/\(.\)\b\1/\e[1m\1\e[22m/g' -e 's/_\b\(.\)/\e[4m\1\e[24m/g' |
sed -e 's/\e[22m\e[1m//g' -e 's/\e[24m\e[4m//g'

If your sed doesn't understand backslash escapes, use the literal characters (Ctrl+H for \b and Ctrl+[ for \e).

share|improve this answer

This is by no means a trivial task. A simple substitution for letter doubles would be disastrous. Think of what it would do to words like "attention" or "forgetting" or (more relevant to your case) "command". The script below is a naive first try at a solution. It makes use of a dictionary to determine which words really have duplicate letters.

#!/usr/bin/perl

use strict;
use warnings;

my $input_file = shift//die "No file name given\n";
my $dictionary = shift//'/usr/share/dict/words';
open my $if,'<',$input_file or die "$input_file: $!\n";
open my $dict,'<',$dictionary or die "$dictionary: $!\n";
my %dictionary;
for(<$dict>){
    chomp;
    $dictionary{$_}++;
}
close $dictionary;

LINE: while(<$if>){
    chomp;

    WORD: for my $word ( split /\s+/ ){
            print "$word " and next WORD if exists $dictionary{lc $word};

            SUBSTITUTION: while($word=~ s{([A-Z])\1}{$1}i){
                exists $dictionary{lc $word} and last SUBSTITUTION;
            } #END SUBSTITUTION
            print "$word ";

     } #END WORD

     print "\n";

} #END LINE

Call it like

[user@host]./myscript.pl input_file optional_dictionary_file >output_file

If you don't supply a second argument, the dictionary file defaults to /usr/share/dict/words, which should be available on a decent GNU/Linux.

Disclaimer: This is untested.

Caveats:

  • It will break at least with hyphenated words (it uses spaces to decide what a "word" is).
  • It will only remove duplicated capitals to avoid messing with the contents of the man page themselves.
  • It will wreak havoc on hexadecimals like 0xFFFF.
  • Probably many more that I can't see.
share|improve this answer
    
I agree removing all duplicates isn't what I want to do; e.g., "commands" should stay "commands" but "NNAAMMEE" should become "NAME" –  user46865 Sep 10 '13 at 23:42
    
@JonSmith ... and what should the header SSHHEELLLL GGRRAAMMMMAARR (from a supposedly broken man bash page) become? –  Joseph R. Sep 11 '13 at 1:03

You can restrict the changes to the affected strings with something like this:

eval sed $(
for i in NAME SYNOPSIS DESCRIPTION "RETURN VALUE" ENVIRONMENT FILES EXAMPLES DIAGNOSTICS ERRORS "SEE ALSO" "CONFORMING TO" HISTORY AUTHORS BUGS; do
  dup=$(for j in $(seq 0 ${#i}); do printf "%s%s" "${i:j:1}" "${i:j:1}"; done)
  printf " -e \"s/%s/%s/\"" "$dup" "$i"
done)
share|improve this answer

In Python :

Method 1# Using Own Function :

#!/usr/bin/env python
from __future__ import print_function
import sys

def RemoveDupliChar(Word):
        NewWord = " "
        index = 0
        for char in Word:
               if char != NewWord[index]:
                       NewWord += char
                       index += 1
        print(NewWord.strip())

with open(sys.argv[1],'r') as InputFile:
        for line in InputFile:
                if line.isupper():
                        RemoveDupliChar(line)
                else:
                        print(line,end='')

Method 2# Using itertools.groupby:

Thanks to @falstretu

#!/usr/bin/env python
from __future__ import print_function
import itertools
import sys

with open(sys.argv[1],'r') as InputFile:
        for line in InputFile:
                if line.isupper():
                        print(''.join(ch for ch, _ in itertools.groupby(line)))
                else:
                        print(line,end='')

Execution :

root@ubuntu:~# python remove_duplicate_char.py Input.txt
NAME
       nice - run a program with modified scheduling priority

SYNOPSIS
       nice     [-n    adjustment]    [-adjustment]    [--adjustment=adjustment] [command [a$
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.