5

I have a string aaabefhhhhhthkkd from which I just need to extract non repetitive letters as output, preserving the order.

The string may contain upper case or lower case letters.

Input:

aaabefhhhhhthkkd

Output:

beftd

How this logic need to be defined so that I get the above required output?

I tried to use this command but it only partially worked for me:

echo "aaabefhhhhhthkkd" | sed 's/./&\n/g' | uniq

Output of above partially worked command:

a
b
e
f
h
t
h
k
d

Sample String to test:

String 1: aaabefhhhhhthkkd -> Output -> beftd

String 2: AAAbefhhhhhThkkD -> Output -> befTD 

String 3: AAAbefhMThkkD    -> Output -> befMTD 
1
  • 5
    I wish you had included in your examples a combination of upper and lower case instances of a single letter so we could see if you wanted the comparison to be case sensitive or not.
    – Ed Morton
    Nov 8 '20 at 13:52

11 Answers 11

5

uniq only works on adjacent duplicates - so if you want to use that, you'd need to sort your input first, for example:

fold -w1 | sort | uniq -u | paste -sd ''
  • fold -w1 does the same as your sed 's/./&\n/g' but without introducing an extra spurious newline
  • sort to make duplicate characters adjacent
  • uniq -u the -u is important to only print singletons
  • paste -sd '' joins the result back into a single line

Because of the sorting, you will not be able to get your desired output order in all cases ex.

$ echo 'AAAbefhMThkkD' | fold -w1 | sort | uniq -u | paste -sd ''
  DMTbef

If you don't want to roll your own solution, you could always use Perl's MoreUtils:

$ echo 'AAAbefhMThkkD' |
    perl -MList::MoreUtils=singleton -ne 'print singleton split //'
befMTD
2
  • 1
    Note that with the GNU implementation of fold, that won't work if the input has multi-byte characters. On GNU systems, you may prefer grep -o .. Also note that POSIX paste requires at least one filename argument (which can be - to specify stdin). Nov 8 '20 at 15:30
  • With the perl variant, you may want to add a -Mopen=locale to work with characters (as per the locale's encoding) instead of bytes. Nov 8 '20 at 15:46
3

Using any awk in any shell on every UNIX box:

$ echo 'aaabefhhhhhthkkd' |
    awk '{
        lgth = length()
        for (pos=1; pos<=lgth; pos++) {
            let = substr($0,pos,1)
            if ( gsub(let,"&") == 1 ) {
                printf "%s%s", let, (pos<lgth ? "" : ORS)
            }
        }
    }'
beftd
10
  • 1
    That is a nice variation, Ed. I think readers would benefit from a explanation from the most hairy parts ;)
    – Quasímodo
    Nov 8 '20 at 13:59
  • Anything in particular that's not obvious (and not already present/discussed in thousands of scripts all over the internet!)?
    – Ed Morton
    Nov 8 '20 at 14:03
  • 1
    All that is quite clear for both of us, but maybe gsub(let,"&") and what's the trick with (pos<lgth ? "" : ORS) could be enlightnening for others. (Yeah, surely it is areound the internet, but it can be a helping hand. For example "what does & mean in gsub" does not return me anything relevant.) But that is just a suggestion, of course.
    – Quasímodo
    Nov 8 '20 at 14:04
  • 1
    gsub(regexp,"&") is the idiomatic awk way to just count occurrences of strings matching a regexp and the ternary (i<max ? OFS ? ORS) is the idiomatic awk way when looping to print a separator between output strings followed by ORS at the end. Both are extremely common, we can't/shouldn't keep explaining the same common, idiomatic code everywhere it's used IMHO>
    – Ed Morton
    Nov 8 '20 at 14:07
  • 1
    Would be worth noting that it assumes that like in the OP's case, none of the characters in the input are regexp operators (like ., $...). Nov 8 '20 at 15:33
3
awk '
{
  n=split($0, a, "")
  for(i=1; i<=n; i++){
    if(gsub(a[i], "") == 1){ printf("%s", a[i]) }
  }
  print ""
}'
  • n=split($0, a, ""): a[1] becomes the 1st character of the string, a[2] the 2nd, etc. n is the total number of characters.
  • for(i=1; i<=n; i++): Let's loop over all the array a.
  • if(gsub(a[i], "") == 1): Delete all a[i] characters from the string. If only one character was deleted on the string,
    • printf("%s", a[i]) print that character.
  • print "" prints a newline character after all the line has been processed. This is optional if you have a single input line.

Example with condensed one-liner:

$ awk '{n=split($0,a,"");for(i=1;i<=n;i++)if(gsub(a[i],"")==1)printf("%s",a[i])}' <<< AAAbefhMThkkD
befMTD

Note: Splitting on a null string is not defined by POSIX. However, gawk (the GNU Awk), mawk and original-awk all implement the operation as desired.

5
  • Awesome it worked … I was also trying with logic like cut first character , loop in for loop check the character in string - cut character | grep and check the status and add it to an dynamic array Nov 8 '20 at 12:42
  • 1
    Not sure what you mean by "original-awk". oawk as in /bin/awk on Solaris? I'd be surprised if it behaved as intended here (not that anyone would care if it did or not, it'd just be an odd thing to mention)
    – Ed Morton
    Nov 8 '20 at 13:57
  • 1
    Np, thanks for providing the link. Sounds like that might be an implementation of "new awk", nawk. Can't imagine why they called it "original-awk" if that is the case.
    – Ed Morton
    Nov 8 '20 at 14:00
  • @EdMorton, that's the one that used to be maintained by Brian Kernighan (the K in awk), (now a community effort (github.com/onetrueawk/awk)) so descendant from the original implementation. bwk did take a few features from gawk. You'll see the gawk maintainer is now contributing to that awk. Nov 9 '20 at 14:51
  • @StéphaneChazelas Oh, I've heard of "one true awk" of course, just never heard of it being called "original awk". Thanks for the info. Too bad there's yet ANOTHER awk name for people to have to know, but at least it's not a different variant from the commonly known set. Yeah, I know Arnold's involved in that too.
    – Ed Morton
    Nov 9 '20 at 15:54
3

With sed, you could do something like:

sed '
  :1
  /\(.*\(.\).*\)\2/ { # while there is a duplicated char
    s//\2\1/; # move it to the front
    :2
      # remove characters that are the same as the first in a loop:
      s/^\(\(.\).*\)\2/\1/
    t2
    s/^.//
    b1
  }'

With the GNU implementation of sed, you can shorten it to:

sed -E ':1;s/(.*(.).*)\2/\2\1/;T;:2;s/^((.).*)\2/\1/;t2;s/^.//;t1'

If you want to do the check for duplicates case insensitively (for áÁbBcδΔ to become c for instance), you can add the i flag to the first 2 s commands in the GNU sed code above. Note however that it won't work for things like German ß vs SS.

And that would still not handle Unicode equivalence and work at character (not grapheme cluster) level, so for instance if you have aéá where those accented letters are expressed in their decomposed form, not only a U+00E9 é would not be considered the same as a U+0065 U+0301 é, but that aéá expressed as U+0061 U+0065 U+0301 U+0061 U+0301 would become e (U+0065), the only non-duplicated character in there, even if those 5 characters actually end up forming 3 distinct grapheme clusters. My first name in its decomposed form would become St́phan (with the combining acute accent landing on the t when both es are removed).

Using:

perl -Mopen=locale -lpe 's/\b{g}\Q$1\E\b{g}//gi while m/(\X)\X*\1/i'

here extending @sitaram's answer (using -Mopen=locale to treat input as characters instead of bytes, \X instead of . to match a grapheme cluster instead of character, and \b{g} for grapheme cluster boundary) would address some of those issues (not breaking down grapheme clusters in the middle, ß vs SS), but not the unicode equivalence:

$ echo $'groß KUSS. Ste\u0301phane, \ue9' |  perl -Mopen=locale -lpe 's/\b{g}\Q$1\E\b{g}//gi while m/(\X)\X*\1/i'
groKU.Stéphane,é

(ß spotted as duplicate of SS, the e in e\u0301 not associated with the standalone e, but the two variants of é not recognised as the same).

Also note that ß/SS would be turned to / as ß is processed first while SS/ß would turned to as the S is processed first.

It would also turn ßA/SAS into / as removing the duplicate As would reveal a SS, the uppercase version of ß. To avoid that, you could change it to:

perl -Mopen=locale -lpe 's/\b{g}\Q$1\E\b{g}/\n/gi while m/((?!\n)\X)\X*\1/i; s/\n//g'

That is, instead of removing the duplicate grapheme clusters, we change them to newline preventing characters on either side to be joined into a sequence of grapheme clusters that could the uppercase or lowercase variant of another grapheme cluster.

0
1

Variation on a theme

echo 'aaabefhhhhhthkkd' | 
 awk '{while (length()>0) {t=substr($0,1,1); printf (gsub( t ,"")==1)?t:""} print}'

beftd

Consume $0 by replacing the first character with "" until empty and print when only one replacement occurs.

1

With GNU awk assigning empty string to FS.

From GNU awk manual:

FS == ""

Each individual character in the record becomes a separate field. (This is a common extension; it is not specified by the POSIX standard.)

echo 'aaabefhhhhhthkkd' | awk -v FS= -v ORS='' '
{for (i=1; i<=NF;i++) if ( gsub($i,"&") == 1 ) print $i;print "\n"}'
beftd

1

This seems to work fine:

$ echo 'aaabefhhhhhthkkd' | perl -pe 's/$1//g while m/(.).*\1/'
beftd

Pretty easy to understand even if you don't know perl; it's just regex the way it's used in sed and any other such tools. Even the way while works is same as, say, bash/sh so that should be clear enough also.

I must admit I did not really understand all the solutions offered -- seemed like too much code for such a straightforward problem. I am guessing I missed something :-(

Also, if you want the comparison to be case-insensitive, add an i flag to the m// and the s///g:

$ echo 'aaabefhhHhhthkkd' | perl -pe 's/$1//gi while m/(.).*\1/i'
beftd
1
  • Like for the other perl-based variants, you may want to change $1 to \Q$1\E (to handle characters that happen to be regexp operators) and add -Mopen=locale to handle non-ASCII input. Nov 12 '20 at 19:00
0

Without awk:

string="aaabefhhhhhthkkd" && discard=$(echo $string|fold -w1|sort|uniq -d|tr -d '\n') && echo $string|sed "s/[$discard]//g"

Explanation:

string="aaabefhhhhhthkkd"

sets your string as variable 'string'

&&

separates the two commands, making sure second part runs after variable assignment.

discard=$(echo $string|fold -w1|sort|uniq -d|tr -d '\n')

will find all non unique characters in $string and save them in variable $discard. (unroll to characters, sort, identify non-uniques, roll back, save to variable)

echo $string|sed "s/[$discard]//g"

will remove characters that are non-unique from string

0

Here's a very simple Perl one-liner (with a little help from echo):

echo aaabefhhhhhthkkd | perl -ple 'my %z; for my $c (split q//){$z{$c}++} for my $k (keys %z){$_ =~ s/\Q$k\E//g if $z{$k} > 1}'

Explanation:

  1. -p: wrap a loop around the given program, each input line is in the default variable $_, and print each input line after processing
  2. -l: handle newlines so you don't have to
  3. -e: run the following as perl code
  4. my %z: initialise a hash to keep character counts in (for each line)
  5. for my $c (split q//): split the default variable, $_, into $c, looping over each character
  6. $z{$c}++: increment the count for character $c
  7. for my $k (keys %z): for each key in %z, the count of characters seen
  8. $_ =~ s/$k//g if $z{$k} > 1: remove all instances of character $k if its count is greater than 1
1
  • 1
    You'd want to use s/\Q$k\E//g to avoid problems with values of $k that happen to be regexp operators. Nov 11 '20 at 18:05
0

Using any awk in any shell on every UNIX box:

From Ed Morton answer, but replacing each character only once (much faster):

$ echo 'aaabefhhhhhthkkd' | 
    awk '{
           while ( length() ) {            # process all letters.
               let = substr($0,1,1)        # extract the first letter
               if ( gsub(let,"") == 1 ) {  # remove all letters (and count)
                   printf "%s", let        # if not repeated, print the letter
               }
           }
           printf "%s", ORS                # print "end of record" delimiter.
          }'

beftd
-2

It took so time to implement this but below approach as worked for me as expected and I am finally done !!!

Tested on below string

String 1: aaabefhhhhhthkkd -> Output -> beftd

String 2: AAAbefhhhhhThkkD -> Output -> befTD

String 3: AAAbefhMThkkD -> Output -> befMTD

#!/bin/bash

# String passed as an input 
str='AAAbefhMThkkD'

# Taking character count of provided string 
count=$(echo "$str" | tr -cd 'a-z|A-Z' | wc -c)

# Dynamic array created 
dynamic_array=()

# Looping through the for loop 
for (( i=1 ; i<=$((count)) ; i++ ))
{
    c=$(echo "$str" |  cut -c "$i")
    character_count=$(echo "$str" | tr -cd "$c" | wc -c) 
    echo "Character : $c  : $character_count" 

   if [ "$character_count" -eq 1 ]
   then  
        dynamic_array+=("$c")   
   fi           
}

str_array_value="${dynamic_array[*]}" ; echo "Output : ${str_array_value// /}" 

# Input :  aaabefhhhhhthkkd : Output : beftd 
# Input :  AAAbefhhhhhThkkD : Output : befTD
# Input :  AAAbefhMThkkD    : Output : befMTD  

Code corrected on https://www.shellcheck.net

3
  • 2
    You should copy/paste that into shellcheck.net and fix the issues it tells you about. There are other issues too that it won't tell you about because manipulating text just isn't what a shell is designed to do and so it's hard to code it correctly. Manipulating text is what awk was designed to do and shells were designed to call tools like awk to perform such tasks.
    – Ed Morton
    Nov 8 '20 at 13:55
  • 1
    Not sure what you mean. shellcheck.net is a web site where you copy/paste your shell script into it and it tells you about the basic problems, that's all. If I were you I'd use that as a learning opportunity for basic shell programming BUT at the end of the day I wouldn't use a shell script with loops like you have for this as it's just the wrong approach so it's not really worth trying to fix ALL of the issues (the ones shellcheck will tell you about and the rest).
    – Ed Morton
    Nov 8 '20 at 14:20
  • Trying to fix a looping shell script to manipulate text would be like trying to fix your pogo stick to do the 20 mile trip to work. Sure you CAN do it but there are far better alternatives. It's not a coincidence that there are no shell loops in anyone else's answer. If you don't understand something that shellcheck is telling you about after reading the references it provides then post a new question about that.
    – Ed Morton
    Nov 8 '20 at 14:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.