0

I have many files in a directory each like so:

AAA 
AA

AAAAAA
A


AAAA

I want to end up with this:

AAAAAAAAAAAAAAAA

So that when I run:

find ./ -name '*' -exec wc -m {} +

I get back 16, not 20+ depending on how many new line/spaces are counted.

Basically, I want to remove EVERYTHING from a file unless it is a letter.

4

Note that if you remove every newline character from a file, even the last one, then it's no longer a text file (unless the file ends up being empty) as a text file contains a sequence of text lines, text lines being delimited by newline characters.

Now, to remove all but alphabetical characters (any alphabet), as @Kusalanada said, POSIXly, you'd use tr -cd '[:alpha:]'.

Now, unfortunately, with some tr implementations, including GNU tr, that doesn't work for multi-byte characters. In UTF-8 locales, that means all characters but ASCII ones.

On GNU systems, you can use GNU awk or GNU sed which do support multibyte characters:

<file sed 's/[^[:alpha:]]//g' | tr -d '\n'

<file awk -v ORS= '{gsub(/[^[:alpha:]]/, ""); print}'

That syntax is not GNU-specific, but you'll find some non-GNU sed/awk implementations that don't support multibyte characters. Beware that GNU sed/awk at least will not remove sequences of bytes that don't form valid characters (like the output of printf 'à b \200\n' in a UTF-8 locale).

With uconv from the ICU project, you could do:

<file uconv -i -x '[^[:Letter:]]>;'

Where -i tells uconv to skip input it can't decode.

But that only works for UTF-8 data. Note that it uses Unicode character properties (some version of Unicode) as opposed to what your locale decides what's alphabetical or not.

With GNU grep, you could use:

<file grep -o '[:alpha:]' | tr -d '\n'

Or if built with PCRE support (using Unicode properties):

<file grep -Po '\pL' | tr -d '\n'

With GNU awk, another approach to skip the invalid input is to use RS:

<file gawk -v RS='[[:alpha:]]' -v ORS= '{print RT}'

To modify the files in-place, you can use gawk's inplace module:

gawk -i inplace gawk -v RS='[[:alpha:]]' -v ORS= '{print RT}' file
2

You don't need -name '*' as you want to process every file (* matches every file anyway, so it does not make any difference). You might however want -type f to only process regular files (not directories etc.)

To remove anything that is not a letter, you may use

tr -cd '[:alpha:]' <file

The -c complements the given set of characters, and [:alpha:] matches only alphabetical characters. The -d instructs tr to delete the matching characters.

The command you may want to execute is therefore

tr -cd '[:alpha:]' <file | wc -m

for each file.

Since this is too complex for find to execute directly, you will have to employ an in-line script:

find . -type f -exec sh -c '
    for pathname do
        tr -cd "[:alpha:]" <"$pathname" | wc -m
    done' sh {} +

Here, the in-line sh -c script will get batches of pathnames of files as arguments from find. The pipeline will be executed for each file.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.