Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am using following command to grep character set range for hexadecimal code 0900 (instead of अ) to 097F (instead of व). How I can use hexadecimal code in place of अ and व?

bzcat archive.bz2 | grep -v '<[अ-व]*\s' | tr '[:punct:][:blank:][:digit:]' '\n' | uniq | grep -o '^[अ-व]*$' | sort -f | uniq -c | sort -nr | head -50000 | awk '{print "<w f=\""$1"\">"$2"</w>"}' > hindi.xml

I get the following output...

    <w f="399651">और</w>
    <w f="264423">एक</w>
    <w f="213707">पर</w>
    <w f="74728">कर</w>
    <w f="44281">तक</w>
    <w f="35125">कई</w>
    <w f="26628">द</w>
    <w f="23981">इन</w>
    <w f="22861">जब</w> 
    ...

I just want to use hexadecimal code instead of अ and व in the above command. Any suggestion to do it will be greatly appreciated.

If using hexadecimal code is not at all possible , can I use unicode instead of hexadecimal code for character set ('अ-व') ?

I am using Ubuntu 10.04

share|improve this question
1  
What do you mean "not working"? Also -v inverts the match, from your question text it seems that is not what you want. –  Christian.K Aug 26 '11 at 6:14
    
@Christian.K Sorry for the delay... I have edited the question , please take a look. –  Dhrubo Bhattacharjee Aug 26 '11 at 10:56
    
I am still waiting for an appropriate answer . :( –  Dhrubo Bhattacharjee Aug 29 '11 at 7:03
add comment

migrated from stackoverflow.com Aug 26 '11 at 13:21

This question came from our site for professional and enthusiast programmers.

3 Answers

If shell escaping is enough you can use the $'\xHH' syntax like this:

grep -v "<["$'\x09\x00'"-"$'\x09\x7F'"]*\s"

Is that enough for your use case?

share|improve this answer
    
echo 'अ-व' | hd gives me e0 a4 85 - e0 a4 b5 –  enzotib Aug 26 '11 at 14:30
    
Indeed the OP gave unicode values, not hexadecimal dumps in UTF-8 encoding :-/ Since grep is not linked with any lib, I guess it's not possible to have the range conversion be performed by grep :-/ –  Stéphane Gimenez Aug 26 '11 at 14:48
1  
Btw, zsh is able to interpret "\u0900" and "\u097F", but the behavior will rely on the UTF-8 encoded range being continuous (probably it is). –  Stéphane Gimenez Aug 26 '11 at 14:49
    
I will try and let you know... tomorrow. –  Dhrubo Bhattacharjee Aug 26 '11 at 17:29
    
No grep -v "<["$'\x09\x00'"-"$'\x09\x7F'"]*\s" gives following output <w f="16929">x</w> <w f="10995">F</w> <w f="2548">FF</w> <w f="762">FFFFFF</w> <w f="655">FFFF</w> <w f="266">xx</w> <w f="215">FFF</w> <w f="117">xxx</w> .... This is not expected. :( , Can I use unicode instead of hexadecimal code or character set ('अ-व') ? –  Dhrubo Bhattacharjee Aug 28 '11 at 2:54
add comment

Look at this question.

Text is usually encoded in UTF-8; so you have to use the hex vales of the bytes used in utf-8 encoding.

grep "["$'\xe0\xa4\x85'"-"$'\xe0\xa4\xb5'"]"

and

grep '[अ-व]'

are equivalent, and they perform a locale-based matching (that is, matching is dependent on the sorting rules of devanagari script (that is, the matching is NOT "any char between \u0905 and \0935" but instead "anything sorting between devanagari A and devanagari VA"; there may be differences.

On the other hand, you have this (note -P):

grep -P "\xe0\xa4[\x85-\xb5]"

that will do a binary matching with those byte values.

share|improve this answer
add comment

we wanted to convert the non-ascii open double quote and close double quote to regular double quotes ("). Also the non-ascii single quote to regular single quote (').

to see them in the file (ubuntu bash shell):

$ grep -P "\x92" infile.txt  (single)
$ grep -P "\x93" infile.txt  (open double)
$ grep -P "\x94" infile.txt  (close double)

translate them:

$ /bin/sed "s/\x92/'/g" a.txt > b.txt
$ /bin/sed 's/\x93/"/g' b.txt > c.txt
$ /bin/sed 's/\x94/"/g' c.txt > d.txt
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.