5

I need a script that checks and lists all text files for project conventions. With conventions I mean for example:

  • UTF-8 encoding
  • No trailing white spaces
  • Newline at the end of file
  • No not-ascii chars
  • LF for line endings

I do not want to reinvent the wheel. Maybe there is a tool doing this. Do you know some?

3
  • 1
    file -i or --mime
    – Hastur
    Dec 9, 2014 at 15:07
  • "Newline at the end of file": does that mean you want to be sure the implicit Unix newline is there for cases where you got the file from a Windows system? Also, with "checking", do you just mean "detect", or "convert such that it complies with what I need"?
    – ayekat
    Dec 9, 2014 at 16:27
  • In first step only checking. Dec 9, 2014 at 16:33

2 Answers 2

6

Detecting

  • UTF-8 encoding: file will usually give you the encoding:

    file --brief --mime-encoding myfile.txt

    Note that it may either be 'us-ascii' or 'utf-8', depending on whether it finds some UTF-8 characters, so you'll need to accept both.

The following points will mostly require you to pipe the output into wc -l (to count the number of lines of the output) and check whether it's 0 or not. Alternatively, they should usually have a return value of 0 if they found something, or 1 if not (in which case your requirements are fulfilled):

  • No trailing white spaces: That's a job for grep, I guess:

    grep -e '\s\+$' myfile.txt
  • Newline at the end of file: If the last character according to hexdump or xxd is 0a, there is a newline, and it's fine:

    xxd -ps myfile.txt | tail -n 1 | grep '0a$'

    (note that, unlike for the other points denoted here, you want this to find something)

  • No not-ascii chars: This is the same as "UTF-8 encoding", except maybe a little more strict.

    If you really want to be sure there are only ASCII characters in a file (see @Anthon's answer), you'll probably need something like

    xxd -g1 myfile.txt | cut -c 10-56 | grep '[a-f89][a-f0-9]'

    This searches for any characters outside the ASCII range (0x00-0x7F). It's not very elegant, though.

  • LF for line endings: file without any options will tell you something like

    ASCII text, with CRLF line terminators

    For a script, probably something like the following could do:

    xxd -g1 myfile.txt | cut -c 10-56 | grep '0d'

Fixing

  • UTF-8 encoding: There is iconv (1). Essentially it takes a "from" encoding (-f), a "to" encoding (-t), and the file. The "to" encoding is probably utf-8, whereas the "from" encoding can be obtained using file as described at the top of my post:

    file_encoding="$(file --brief --mime-encoding myfile.txt)"
    iconv -f "$file_encoding" -t 'utf-8' myfile.txt
  • No trailing white spaces: That's a job for sed, although I prefer the POSIX way (i.e. no -i switch) which means using printf + ex. Either way, the regex will look something like s/\s\+$//g. For the POSIX-compliant way:

    printf "%%s/\\s\\+\$//g\nwq\n" | ex -s myfile.txt

    For the non-POSIX-compliant way:

    sed -i 's/\s\+$//g' myfile.txt
  • Newline at the end of file: Unix applications usually append a missing newline at the end of file when they save it. To exploit that, this is bit of a hack:

    printf "wq\n" | ex -s myfile.txt

    (this will actually just open, save, quit)

  • No not-ascii chars: See "UTF-8 encoding" above.

  • LF for line endings: There is dos2unix (1). It should do exactly what you need.

0

You should try the file command, it already does a good job of determining information about the contents, although it doesn't analyse the full data file.

If you have a large file somefile where the first utf-8 character doesn't appear until the end, file somefile will not detect that as an utf-8 file.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .