How would I remove all non-ascii characters from one file? Would there be a specific command to perform this?

grep --colour='auto' -P -n'[^\x00-\x7]' /usr/local/...

I believe this finds the characters within the workflow, but how would I remove all the instances of the characters in question?

  • 2
    related: if you just want to avoid problems with control chars (instead of silently getting rid of them), you can simply use cat -v to show them in ASCII represantation for them. (eg. ^G for \007) – Matija Nalis Oct 15 at 15:04
  • 1
    When you say "non-ascii characters" do you also include accented characters? – Captain Man Oct 15 at 15:04
  • 1
    @MatijaNalis More info about the representation: en.wikipedia.org/wiki/Caret_notation – wjandrea Oct 15 at 18:09
  • 1
    What is the use case? Very often there are specific tools or different approaches which work much better than simply removing a bunch of special characters. Please note that ASCII does include several "special" characters like vertical tabs, bell, and NUL - are you sure you don't mean printable characters? – l0b0 Oct 15 at 19:43

ASCII characters are characters in the range from 0 to 177 (octal) inclusively.

To delete characters outside of this range in a file, use

tr -dc '\0-\177' <file >newfile

The tr command is a utility that works on single characters, either substituting them with other single characters (transliteration), deleting them, or compressing runs of the same character into a single character.

The command above would read from file and write the modified content to newfile. The -d option to tr makes the utility delete characters (instead of transliterating them), and -c makes it consider characters outside the given interval (instead of inside).


To replace the original file with the modified one, use

tr -dc '\0-\177' <file >newfile && mv newfile file

This renames the new file to the name of the old file after tr has completed successfully. If tr does not complete successfully, either because it could not read the original file or not write to the new file, the original file will be left unchanged.

With perl

perl -pi -e 's/[^[:ascii:]]//g'

If all you need is a regex: [\x00-\x7F] that you could apply to several utilities:

<file LC_ALL=C.UTF8   sed   's/[^\o0-\o177]//g'
<file   awk   '{print gensub(/[^\x0-\x7f]/,"","g")}'
<file   perl  -pe 's/[^[:ascii:]]//g;'
<file   tr    -dc '\0-\177'

Understand that sed, awk, and perl expect "text files" as defined in Unix. All work well in this case. But specifically, awk adds a trailing new line (whether it existed in the source file or not). The tr is designed to work with any file type.

However the NUL (\0) is not a valid character in a POSIX text file and should be avoided:

The lines do not contain NUL characters...

In fact, many control characters could generate problems in some cases.
So, probably you need [\x07-\x0d\x20-\x7e]

<file LC_ALL=C.UTF8   sed   's/[^\o007-\o015\o040-\o176]//g'
<file   awk   '{print gensub(/[^\x07-\x0d\x020-\x7e]/,"","g")}'
<file   perl  -pe 's/[^\x{7}-\x{d}\x{20}-\x{7e}]//g;'
<file   tr    -dc '\7-\15\40-\176'

The range 7-13 (in decimal) is \a\b\t\n\v\f\r (in order).
A similar (probably more portable) range could be written as [^[:space:][:print:]] (similar because it doesn't include\a\b` bell and backspace).

<file LC_ALL=C.UTF8   sed   's/[^[:space:][:print:]]//g'
<file   awk    '{print gensub(/[^\x07-\x0d\x020-\x7e]/,"","g")}'
<file   perl   -pe 's/[^[:space:][:print:]]//g;'
<file   tr     -dc '[:space:][:print:]'

Related:
Regex any ASCII character
Perl solution
Posix Text File

  • Note that the input to tr can be any type of file, not just text files. awk on the other hand, takes a text file. – Kusalananda Oct 16 at 8:10
  • It is quite hard for me to find anything else to call a file "only ascii characters" anything but a "text file" (yes, yes: in layman terms). @Kusalananda (note about awk added anyway). – Isaac Oct 17 at 15:19

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.