If I am having a line as:

There are seven pencil

I want to print this as:

Ther a svn pcil

What is the bash shell command for this?

Clarification: the goal is to remove all, at least twice occuring letters, except their first occurence.

  • 7
    FYI - bash is just your shell; it has some powerful features, but mainly it's for calling other utilities, such as text processors, etc. – Jeff Schaller Jan 24 '18 at 22:53
  • What is the rule that translates the input to the output? – Jeff Schaller Jan 24 '18 at 22:54
  • 5
    Why is the space repeated in the output? Which other characters can be repeated? – Jonathan Leffler Jan 25 '18 at 13:24

Based on sed classic synthax s/replace-this/with-that/g where g means global replace = all occurences , someone can use 2g instead of g which means global replacement but after second occurence (this is a gnu sed extension).

Example that removes only e:

$ echo $a
there are seven pencil

$ echo $a | sed 's/e//2g'
ther ar svn pncil

To remove all duplicate letter we can make a trick like this:

$ sed -f <(printf 's/%s//2g\n' {a..z}) <<<"$a"
ther a svn pcil

Unfortunatelly this will not work : sed 's/[a-z]//2g'

The above trick uses process substitution <( ) which can be used as file.

In my solution process substitution is treated like a sed script file, fed to sed by -f option = read sed commands from a file.

  • 1
    You don't need process substitution here. sed "$(printf ...)" will work just the same. You don't need zsh's (or compatible) <<< operator either. Using printf '%s\n' "$a" | sed... would work with any shell. – Stéphane Chazelas Jan 25 '18 at 9:50

Awk solution (for case-sensitive cases):

s="There are seven pencil"
printf '%s\n' "$s" | awk -v FS="" '{ 
           for(i=1; i<=NF; i++) 
               if ($i==" " || !a[$i]++) printf "%s", $i; print "" 
  • -v FS="" - set "empty" field separator so that each character becomes a separate field (not POSIX but a GNU extension supported by quite a few implementations)
  • for(i=1; i<=NF; i++) - iterating over characters
  • if ($i==" " || !a[$i]++) - if it's space char or having the 1st occurrence of a certain char

The output:

Ther a svn pcil

For case-insensitive, replace a[$i] with a[tolower($i)].


Here's Bash itself.

s="There are seven pencil"
declare -A A
while IFS= read -rn1 a; do
 [ -z "$a" ] || [ -n "${A[$a]}" ] && continue
 printf %s "$a"
 [ "$a" == " " ] || A[$a]=x
done <<<"$s"

Explanation line by line:

  1. Assign the string to a variable

    s="There are seven pencil"
  2. Declare an associative array A

    declare -A A
  3. This one's a bit complex. But without details, it reads the string by single characters and assigns the character just read to a. It's a while loop.

    while IFS= read -rn1 a; do
  4. Continue the loop (go to the next iteration, so read the next character) if the current character is empty ([ -z "$a" ]) or (||) if the value associated with this key (this character) has already been set (if it has then here it comes again, so we continue without printing it).

    [ -z "$a" ] || [ -n "${A[$a]}" ] && continue
  5. Print the current character.

    printf %s "$a"
  6. If the character is space - then don't execute A[$a]=x - that's what || means here. A[$a]=x is the association operation. To prevent all spaces no value should be assign to the key space in the A array. (See point 4.)

    [ "$a" == " " ] || A[$a]=x
  7. This ends the while loop. <<<"$s" is a here-string redirection. It feeds the loop with the string.

    done <<<"$s"
  8. This last echo prints the line separator. printf in point 5. printed only the character. Without this echo the output would appear on the same line as the following shell prompt. Remove it and see yourself.

  • I am struggling to figure this out. Would it be ok to have it explained a bit? I was wondering about it on your question about it. – Guy Jan 25 '18 at 1:59
  • 1
    @Guy Take a look now. As for my question, the code there is slightly different. Otherwise there would be no question. Compare line 6. – Tomasz Jan 25 '18 at 2:23
  • Cheers, that definitely makes a bit more sense! – Guy Jan 25 '18 at 2:25

Python solution:

remove_dups.py script:

import sys

s, res = set(), []
for c in sys.argv[1]:    # iterating over characters
    if c not in s:       # on the 1st occurrence of a character
        if not c.isspace(): s.add(c)
print(''.join(res))      # print unique chars


s="There are seven pencil"
python remove_dups.py "$s"

The output:

Ther a svn pcil

Another sed solution:

  • for single alphabetic character:

    $ echo 'here hear' | sed 's/\(\([[:alpha:]]\).*\)\2/\1'
    here ear
  • for all of them, the g flag won't help as characters already processed won't be checked again. So, use a loop - 't' will branch to the label as long as substitution succeeds

    $ echo There are seven pencils | sed -e :a -e 's/\(\([[:alpha:]]\).*\)\2/\1/; ta'
    Ther a svn pcil

    With GNU sed, you could shorten it to:

    sed -E ':a;s/(([[:alpha:]]).*)\2/\1;ta'

to ignore case (still with GNU sed):

$ echo 'There this That' | sed -E ':a; s/(([[:alpha:]]).*)\2/\1/i; ta'
Ther is a

with perl (here limited to ASCII letters only):

$ echo 'There are seven pencil' | perl -pe 'while(s/([a-zA-Z]).*?\K\1//g){}'
Ther a svn pcil
$ echo 'There this That' | perl -pe 'while(s/([a-z]).*?\K\1//gi){}'
Ther is a

To remove all duplicate characters (keeping only the first occurrence), in zsh:

$ s="There are seven pencils"
$ printf '%s\n' ${(j::)${(s::u)s}}
Ther asvnpcil

That approach can't be used if you want to exclude some characters from that de-duplication (like the space character in your example).

  • s:: split on the empty string to split the string into its character constituents as some awk implementations do with an empty FS
  • u: unique: remove duplicates in the resulting array
  • j::: join the elements of the array with empty strings in between.

An awk solution

Contents of input file file.txt:

There are seven pencil

For each line of file.txt, create a new string. Place characters from the original line into the new string, one character at a time, if they have not already occurred on that line. Always include space characters. Print the new string when the entire line has been processed:

awk '{
       printme="" ;
       for ( n=1 ; n<=length($0) ; n++ ) {
         char=substr($0,n,1) ;
         if ( printme !~ char || char == " " ) printme = printme char ;
       } ;
       print printme ;
     }' "file.txt"


Ther a svn pcil

We can do this by below simple sed command with for loop


There are seven pencil


 for i in {a..z}; do sed -i "s/$i//2g" input.txt; done


Ther a svn pcil
  • 1
    This is the same principle as George Vasiliou’s answer, but a worse implementation.  George’s answer handles arbitrary standard input and standard output.  Your answer requires that the text be in a file, and then it reads and writes that file 26 times!! – G-Man Jan 25 '18 at 22:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.