Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I need to efficiently count every character of an arbitrary file by its character CLASS (as defined by the BASH man page); i.e.

[[:alnum:]], [[:alpha:]], [[:ascii:]], [[:blank:]], [[:cntrl:]], [[:digit:]], [[:graph:]], [[:lower:]], [[:print:]], [[:punct:]], [[:space:]], [[:upper:]], [[:word:]] and [[:xdigit:]]

Once the file is processed, display on a single line the resulting counts for each, even when zero.

Web searches have not been fruitful in finding something along these lines.

The arbitrary file (/tmp/f1.txt) will contain a variety of diverse text/data.

I am not looking to process ELF binaries nor UniCode (or any form of multi-byte) content.

I am not concerned about line count (CR and/or LF), only fixated on accumulating a count of each 'character' in the target file by the above classes.

I intend for this to end up as a standard function() in a larger bash script. Bash/sed/awk and the like are desired; while perl/python/ruby not so much.

Sample data files could be:

  • Zero bytes, i.e. no content at all.

  • A single character

  • A single word

  • Multiple words separated by whitespace

  • Multiple lines interspersed with whitespace and/or CarriageReturns and/or LineFeeds.

  • For multiline files there might not be a CR or LF to signify the end of the last line (yet all characters should still be counted).

share|improve this question
add comment

3 Answers 3

file=myfile
for class in alnum alpha blank cntrl digit graph lower print punct space upper xdigit
do
  printf '%7s: %d\n' "$class" "$(tr -Cd "[:${class}:]" < "$file" | wc -m)"
done

ascii and word are not standard character classes and are bash specific. word is alnum plus underscore, and ascii is characters 0 to 127, so you can do:

printf '%7s: %d\n' word "$(tr -Cd "_[:alnum:]" < "$file" | wc -m)"
printf '%7s: %d\n' ascii "$(LC_ALL=C tr -cd '\0-\177' < "$file" | wc -c)"

(note that the GNU implementation of tr, as of coreutils-8.22, won't work with multi-byte characters).

share|improve this answer
1  
word is not synonymous to alnum – “_” is word but is not alnum. pastebin.com/raw.php?i=czZcwix0 –  manatwork Jul 23 '13 at 7:47
    
@manatwork, you're right. Checking the bash code, it is indeed alnum+_, answer fixed accordingly. –  Stéphane Chazelas Jul 23 '13 at 8:44
add comment

Sounds like a fun class! What is it?

This will get you most of the way there; sed doesn't seem to support :ascii: or :word:, but:

for f in alnum alpha ascii blank cntrl digit graph lower print punct space upper word xdigit
do
  echo "$f: $(sed s/[^[:$f:]]//g b.txt | tr -d '\n' | wc -c)"
done

Where we use sed to strip out everything except the characters we care about, then delete all the blank lines, and just get the count of characters left over.

Should be relatively accurate, with the possible exception of +/-1 or off-by-a-factor-of-ten errors.

share|improve this answer
    
That doesn't work for cntrl or space since you're excluding the LF characters. Depending on the sed implementation, NUL characters may be an issue as well. –  Stéphane Chazelas Jul 23 '13 at 8:51
add comment

I'm not going to be looking up what characters fit into to which class - probably you can figure that out and/or just consult the other answers. But this will give you an unequivocal representation of your file character by character without missing any:

 _c2o() { od -A n -t o1 -w1 -v | tr -dc '0-9\n' ; } 
 _c2o <file
 163
 150
 072
 040
 167
 141
 162
 156
 151
 156
 147

That's a function I use in a lot of different ways. Each line is a single bite expressed in octal format - though od is very configurable. But in this way you can very easily just grep or sed for your target values and implement a line counter. It's really a piece of cake. And it's very fast.

Ok, so I went ahead and did the classes anyway:

_classes() { set -- ${classes=alnum alpha blank cntrl digit graph lower print punct space upper xdigit}
        while ${1+:} false ; do
                printf %b $(printf '\\%04o\n' $(seq 0 127)) |
                tr -dc "[:${1}:]" | {
                        printf "$1='"
                        _c2o
                        printf "'\n"
                } ; shift
        done
}

Run the above and you'll get output like:

xdigit='060
061
062
063
064
065
066
067
070
...
'

From there, I'd imagine something like:

eval "$(_classes)"
for class in $classes ; do
    eval "$class=\$(_c2o <file | grep -c -F "$class")"
done

I need to figure that out a little better - but that just about does the whole job.

share|improve this answer
    
That doesn't answer the question though. As that gives octal representation of bytes in the file, not characters and you end up being further away from getting character class information. –  Stéphane Chazelas May 7 at 8:39
    
Stephane, thats true - and i have an answer to that, or most of one, but i dozed off while finishing it up before i could post it. Ill have it up very soon. –  mikeserv May 7 at 8:55
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.