2

I want to find what control characters are in my file. I do not search for specific character, but it is possible \t or \n. I have a program that is telling me: Invalid control character. But when I open the file I do not see anything. How to show these characters? I suspect \n or \t or some characters that adds spaces.

I tried: grep '\n' myfile.txt But in the output it marked the n letter.

  • Hi! I think you can probably have some control characters in your file, is this page helpful somehow? – danieldeveloper001 May 15 '19 at 0:31
  • 1
    Is this question somehow different from your other question? How to find and replace seemingly a tab char using sed? It would help if you told us what program it is that is complaining (a "control character" could potentially be a character with some specific semantics when used with your program, not necessarily an ASCII control character). – Kusalananda May 15 '19 at 9:31
1

To look for a specific character

grep and sed don't support backslash notation for control characters. (sed does use backslash for regexp backreferences.) If you are using bash it can convert a backslash sequence to the actual control character before passing to these (or any) programs:

$ grep $'\t' file
$ sed -n /$'\t'/p file
$ # or change to l (ell) to visibly show the control character(s)
$ sed -n /$'\t'/l file

OTOH awk does portably support this notation:

$ awk '/\t/' file

To look for any control character

Both grep and sed can search for a complemented character class/range, which will find lines containing any character that is not a 'printable' (graphic or space) ASCII character. (The newline characters which separate lines in the file are not treated as in the lines, and thus do not satisfy this match. However, if you have a file with Windows or telnet/SMTP/etc style CRLF, the CR will match on every line, making this technique less useful.)

 $ export LANG=C # use an ASCII or at least single-byte locale; this is the simplest one
 $ grep '[^ -~]' file
 $ sed -n '/[^ -~]/p' file
 $ # or better (see below)
 $ sed -n '/[^ -~]/l' file 

To display which control character(s) is(are) present, in either the whole file or selected line(s), in addition to the options given by SHawarden you can also use:

 $ sed -n l [file] # that's ell not one; can merge into the selection as above
 $ cat -vT [file]
 $ # both read stdin if not given a filename 
 $ # and thus can be piped from a selection command above

Note some of these displays use backslash notation for at least some characters (\t = tab, \b = backspace) while others use 'caret' (which in ancient times was 'uparrow') notation (^I = tab, ^H = backspace). See any ASCII chart for the correspondences, noting that caret/uparrow represents either subtracting or adding hex 40 (equal to octal 100).

1

In case it's about json data (python's json package is known to report that error message), you could identify which json string has control characters in them with:

perl -Mcharnames=:full -C -l -0777 -ne '
  while (/"(?:\\.|[^"])*"/g) {
    my $offset = $-[0];
    my $string = $&;
    @ctrl = map {charnames::viacode(ord($_))} $string =~ /\p{PosixCntrl}/g;
    if (@ctrl) {
       print "Offset: $offset, String: $string, Ctrl: ". join "+", @ctrl
    }
  }' file.json

On an example file.json file here:

$ python -c 'import json; import os; print(json.load(file("file.json")))'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/json/__init__.py", line 291, in load
    **kw)
  File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 380, in raw_decode
    obj, end = self.scan_once(s, idx)
ValueError: Invalid control character at: line 1 column 22 (char 21)

And the perl code above returns:

Offset: 19, String: "a  b
c", Ctrl: CHARACTER TABULATION+LINE FEED

You can see the control character python complains about is that TAB one, 2 chars after the start of that "..." string. Note that perl reports the offsite in number of characters, while python reports them in number of bytes.

0

To see what that character is:

less sourcefile

or

od -c sourceFile

for a more verbose view.

  • less -r is the opposite of what is wanted here; it doesn't show any control chars. less without -r does show most -- but not tab (wanted here) and backspace. – dave_thompson_085 May 15 '19 at 8:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.