4

I have a random string with ugly chars in it: ÓˇÌ„ˇ™ÌÓ‹ÍÙ› ‹„ı

That chars have to be eliminated. The Whitelist consist of: a-zA-Z0-9 -_* + ß ä ü ö () % @ € & = . and Space

My first Script to do this:

regex="[^\-\_\*\+\ß\ä\ö\ü\(\)\%\@\€\&\=\.a-z0-9A-Z\ ]"
replaceChar="_"
echo "testflŒÆ˘ˆı››◊‹ıÓÌˇˆÁÓˆfl̈™ˇÏˆıÍÓÌıÓWÌtest" |sed -e "s/${regex}/${replaceChar}/g"

But this is my output:

test_ŒÆ__ı____ıÓÌ__ÁÓ__Ì___Ï_ıÍÓÌıÓWÌtest

My Output for $LANG

LANG=de_DE.UTF-8

echo "testflŒÆ˘ˆı››◊‹ıÓÌˇˆÁÓˆfl̈™ˇÏˆıÍÓÌıÓWÌtest" | od -c
0000000   t   e   s   t 357 254 202 305 222 303 206 313 230 313 206 304
0000020 261 342 200 272 342 200 272 342 227 212 342 200 271 304 261 303
0000040 223 303 214 313 207 313 206 303 201 303 223 313 206 357 254 202
0000060 303 214 313 206 342 204 242 313 207 303 217 313 206 304 261 303
0000100 215 303 223 303 214 304 261 303 223   W 303 214   t   e   s   t
0000120  \n
0000121
  • Intriguing, could it be something with codepages or unicode? I usually use iconv and strings to clean up strings. Modding this up, I am interested in knowing other people ideas too. – Rui F Ribeiro Dec 7 '15 at 16:30
  • what happens with sed "s/[^[:alpha:]]/_/g" ? – Rui F Ribeiro Dec 7 '15 at 16:38
  • This might be something to do with mismatched encodings. Can you edit your post to include your locale settings, and perhaps pipe your echo into od -c so we see the individual bytes? – dhag Dec 7 '15 at 16:39
  • 1
    a-z includes a lot more than the 26 English letters. You want to use abcdefghijklmnopqrstuvwxyz if you want specifically only the English letters and not the other characters that are between a and z. Same for A-Z. And you don't want to use backslash within [...]. – Stéphane Chazelas Dec 7 '15 at 17:28
  • Alternatively, set $LANG to a UTF-8 locale (as I assume those characters are in UTF-8. It probably is already), LC_COLLATE to C and all the other LC_* unset. That would work at least on GNU systems. (and regex="[^-_*+ßäöü()%@€&=.a-z0-9A-Z ]") – Stéphane Chazelas Dec 7 '15 at 17:34
2

This will create the correct regex:

a="$(printf '%s' {a..z} {A..Z} {0..9} - )"
b="_*+ßäöü()%@€&=."

regex="[^$b$a]"
replaceChar="_"

Then this will work:

line="testflŒÆ˘ˆı››◊‹ıÓÌˇˆÁÓˆfl̈™ˇÏˆıÍÓÌıÓWÌtest"
echo "$line" | sed -e "s/${regex}/${replaceChar}/g"

test_______________________________W_test

It is interesting to note that if LANG=C the command will fail. Even with a regex as simple as this:

$ (LANG=C; echo "testflŒÆtest" | sed -e "s/[^tesæ]/_/g")
test_____�_test

To see what character number that is:

$ (LANG=C; echo "testflŒÆtest" | sed -e "s/[^tesæ]/_/g")|od -An -tcx1
   t   e   s   t   _   _   _   _   _ 303   _   t   e   s   t  \n
  74  65  73  74  5f  5f  5f  5f  5f  c3  5f  74  65  73  74  0a

That is: 303. That repeats for longer strings as well. Maybe is what you saw.

  • An expression of the form {c1..c2}, where c1 and c2 are single characters (which may be multibyte characters), is expanded to every character in the range from c1 to c2 in whatever character sequence is used internally. < that's zsh, but i think all of the shells do the same with thing. man bash confuses me - it says using the default C locale - which I think would mean the C locale, but it just sounds weird to me - are there alternate C locales? – mikeserv Dec 8 '15 at 2:19
  • Character sequences are a tricky part of i18n (internationalization). There are systems for windows 1252, or codepage 8159-1 or (old) UCS-2, or many others. Each with it's particular number for the same character. A list of the particular numbers that some characters have will also be (very) different. In bash brace expansion, the characters are well defined as those that a C locale will make of them (mostly ascii). The brace expansion {a..z} will be the numbers between ascii a and ascii z. Which are very well defined. Was that your question? – user79743 Dec 8 '15 at 2:39
  • The zsh manual I have And this one here say: ` it is expanded to a list of the individual characters between the braces sorted into the order of the characters in the ASCII character set`. That leaves no space for interpretation. Or are you being contentious just to have fun? – user79743 Dec 8 '15 at 3:41
  • what? no...? are you having fun? what happened to chat? anyway, i copied just did man zshexpn and then found BRACE EXPANSION. it does also say: For characters with code points below 128 this is US ASCII (this is the only case most users will need). - but isnt that always the case in a UTF-8 locale? it seemed pretty clear to me that it used the set encoding when it said: the range from c1 to c2 in whatever character sequence is used internally. am i wrong? i easily could be. i will make an attempt to understand an argument you offer to the contrary. – mikeserv Dec 8 '15 at 3:47
  • hey...! i guess something has changed - i dont see a version tag for that link. if it matters, mine is: 5.1.1. the ksh manual is pretty definite about only expanding the C locale, for sure. – mikeserv Dec 8 '15 at 3:53
0

If I don't know the set very well (or its complement), I usually don't trust negation - especially of ranges. I don't know the first thing about most of those characters in your ugly string, or where they came from, or if my computer computer cares. I do know a few of the others, and I know how to remove anything but - so long as all of the ugly characters are at least valid characters.

alnum=0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ
sed -e's|.|&\n|g'     -e'# this opens up the string' \
    -e"s|\([-$alnum*_+ßäüö ()%&@=.$€]\)\{0,1\}.\{0,1\}\n|\1_|g" \
<<""
testflŒÆ˘ˆı››◊‹ıÓÌˇˆÁÓˆfl̈™ˇÏˆıÍÓÌıÓWÌtest

testWtest

^thats right, right?

So it splits the string out to a \newline per character, and then it scans the string a character at a time from left to right. As it does, it will do one of two things for each - either it will replace one of your whitelisted characters with 0 or 1 occurrences of itself, or it will remove 0 or 1 occurrences of some other character. In both cases it also removes the trailing newline delimiter.

I guess it is easier to see what it does with the _ underscores - (which is probably why you included them):

sed -e's|.|&\n|g'     -e'# this opens up the string' \
    -e"s|\([-$alnum*_+ßäüö ()%&@=.$€]\)\{0,1\}.\{0,1\}\n|\1_|g" \
<<""
    testflŒÆ˘ˆı››◊‹ıÓÌˇˆÁÓˆfl̈™ˇÏˆıÍÓÌıÓWÌtest

 _ _ _ _t_e_s_t________________________________W__t_e_s_t_

That's what sed will do with replacement of a possibly null-length string. Removing is fine - but sed can take it or leave it, and will. Oh, the spaces, right, well, I just copied and pasted into the terminal and so the four leading characters (for the Markdown code block indention) were spaces.

One reason I use the \newlines has to do with what happens for an invalid byte-sequence in pattern space. If it doesn't add up to a an actual character then . doesn't match, and /^.*$/ will fail. With the newlines, if any characters following the the bad byte sequence managed to match . in the first place, then:

sed    '/^.*$/!{/\n/D;}'

...will work past it (but not with a GNU sed - guess I should have checked first. I was using the AST sed earlier - which doesn't fuss). With GNU sed z will zap the whole pattern space.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.