13

How would I remove all non-ascii characters from one file? Would there be a specific command to perform this?

grep --colour='auto' -P -n'[^\x00-\x7]' /usr/local/...

I believe this finds the characters within the workflow, but how would I remove all the instances of the characters in question?

  • 3
    related: if you just want to avoid problems with control chars (instead of silently getting rid of them), you can simply use cat -v to show them in ASCII represantation for them. (eg. ^G for \007) – Matija Nalis Oct 15 '18 at 15:04
  • 1
    When you say "non-ascii characters" do you also include accented characters? – Captain Man Oct 15 '18 at 15:04
  • 1
    @MatijaNalis More info about the representation: en.wikipedia.org/wiki/Caret_notation – wjandrea Oct 15 '18 at 18:09
  • 1
    What is the use case? Very often there are specific tools or different approaches which work much better than simply removing a bunch of special characters. Please note that ASCII does include several "special" characters like vertical tabs, bell, and NUL - are you sure you don't mean printable characters? – l0b0 Oct 15 '18 at 19:43
26

ASCII characters are characters in the range from 0 to 177 (octal) inclusively.

To delete characters outside of this range in a file, use

LC_ALL=C tr -dc '\0-\177' <file >newfile

The tr command is a utility that works on single characters, either substituting them with other single characters (transliteration), deleting them, or compressing runs of the same character into a single character.

The command above would read from file and write the modified content to newfile. The -d option to tr makes the utility delete characters (instead of transliterating them), and -c makes it consider characters outside the given interval (instead of inside).

LC_ALL=C makes sure that every byte value makes up a valid character. Without it, some tr implementations would abort if they found sequences of bytes that don't form valid characters in the locale's character encoding.


To replace the original file with the modified one, use

LC_ALL=C tr -dc '\0-\177' <file >newfile &&
mv newfile file

This renames the new file to the name of the old file after tr has completed successfully. If tr does not complete successfully, either because it could not read the original file or not write to the new file, the original file will be left unchanged.

Alternatively, to preserve as much as possible of the meta data (permissions etc.) of the original file, use

cp file tmpfile &&
LC_ALL=C tr -dc '\0-\177' <tmpfile >file &&
rm tmpfile
|improve this answer|||||
14

With perl

perl -pi -e 's/[^[:ascii:]]//g'
|improve this answer|||||
9

If all you need is a regex: [\x00-\x7F] that you could apply to several utilities:

<file LC_ALL=C   sed   's/[^\o0-\o177]//g'      # GNU sed without POSIXLY_CORRECT
<file LC_ALL=C   awk   '{gsub(/[^\0-\177]/,"");print}'
<file            perl  -pe 's/[^[:ascii:]]//g;'
<file LC_ALL=C   tr    -dc '\0-\177'

Understand that sed, awk, and perl expect "text files" as defined in Unix. All work well in this case. But specifically, awk adds a trailing new line (whether it existed in the source file or not) (using printf removes ALL newlines on the input). The tr is designed to work with any file type. However the NUL (\0) is not a valid character in a POSIX text file and should be avoided:

The lines do not contain NUL characters...

In fact, many control characters would generate other problems under some specific conditions.
So, probably you need [\x07-\x0d\x20-\x7e]

<file LC_ALL=C   sed   's/[^\o007-\o015\o040-\o176]//g'            # GNU sed without POSIXLY_CORRECT
<file LC_ALL=C   awk   '{gsub(/[^\0-\15\40-\176]/,"");print}'
<file            perl  -pe 's/[^\x{7}-\x{d}\x{20}-\x{7e}]//g;'
<file LC_ALL=C   tr    -dc '\7-\15\40-\176'

The range 7-13 (in decimal) is \a\b\t\n\v\f\r (in order).
A similar (probably more portable) range could be written as [^[:space:][:print:]] (similar because it doesn't include\a\b` --bell and backspace--).

<file LC_ALL=C   sed   's/[^[:space:][:print:]]//g'  # GNU sed without POSIXLY_CORRECT
<file LC_ALL=C   awk   '{gsub(/[^[:space:][:print:]]/,"");print}'
<file            perl   -pe 's/[^[:space:][:print:]]//g;'
<file LC_ALL=C   tr     -dc '[:space:][:print:]'

Related:
Regex any ASCII character
Perl solution
Posix Text File

|improve this answer|||||
  • Note that the input to tr can be any type of file, not just text files. awk on the other hand, takes a text file. – Kusalananda Oct 16 '18 at 8:10
  • It is quite hard for me to find anything else to call a file "only ascii characters" anything but a "text file" (yes, yes: in layman terms). @Kusalananda (note about awk added anyway). – Isaac Oct 17 '18 at 15:19
  • Note that gensub() is a gawk extension. You'd want gsub(...); print, and use octal instead of hex sequences (and LC_ALL=C) to be (more) portable. – Stéphane Chazelas Jul 31 '19 at 14:09
  • @StéphaneChazelas What is the limitation of GNU sed that makes the syntax GNU specific (I understand the POSIXLY_CORRECT problem). – Isaac Jul 31 '19 at 15:37
  • [^\o0] is to match on characters other than backslash, o and 0 in POSIX sed (in all implementations but GNU sed). That's not a limitation of GNU sed but a non-compliant extension, which is why it's disabled when POSIXLY_CORRECT is in the environment). – Stéphane Chazelas Jul 31 '19 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for?Browse other questions tagged or ask your own question.