1

Have a massive codebase (2Gb mostly code) where valid utf8 symbols in German were replaced by this symbol: "�"

Is there a way I can find all files that contain this reliably? Visual Studio Code does "sometimes" find files, but not consistently. It's very strange.

grep -rnw . -e '�'

only finds them in the few binary and .png files, but not in the text files (.c, .h, .m etc.) at all.

Any ideas? Huge thanks in advance!

7
  • 1
    You can check unprintable char hexadecimal value using od -c. Then search for value instead of char
    – Digvijay S
    Commented May 8, 2020 at 16:09
  • 6
    is a typical replacement character, so a character that some tools display in place of character or non-characters which otherwise they could not display (like control characters or bytes which don't form part of valid characters in the current locale). So that character may not be in those files at all. Commented May 8, 2020 at 16:21
  • 2
    In any case, you don't want to use -w here. Commented May 8, 2020 at 16:23
  • Does putting LC_ALL=C before grep help? Commented May 8, 2020 at 18:39
  • @JosephSible-ReinstateMonica In the C locale, all byte values are valid (no invalid values). Also, a C locale will not detect UTF-8 characters (what is being asked).
    – user232326
    Commented May 8, 2020 at 19:36

1 Answer 1

1

The answer should be based in:

grep -axv '.*' file.txt

To find (recursivelly) all filenames (-l) that have invalid characters, use:
(yes no filename needed)

grep -rlaxv '.*'

In

grep -axv '.*'
  • The -a is to open files that might appear as binary (not text) to grep.
  • The -x is to select whole lines (required by the next option).
  • The -v is to invert what is selected (works in convination with -x).
  • The regex .* will match any valid character. Inverted with -v, the whole command will match any whole line that contains any in-valid character.

However, understand that valid or invalid are definitions that depend on the encoding of the file being read. If your system is using UTF-8 encoding by default the command will detect invalid UTF-8 sequences. Change your locale settings to detect other encodings.

Additionally, the goal of the command is to find invalid characters. If the goal is to find actual characters '�', then just search for them:

grep -oP '�' file

Or, to avoid confusions about which character it is (in bash, ksh, zsh), to list filenames matching:

grep -rloP $'\Ufffd'

More details are given in this answer

Of course, this will not fix the files, just list them. To actually fix the files (if that is possible), please post an example of the problem. Two or three lines will suffice.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .