What are some ways to compare file type and encodings to resolve buggy grep...?

Question

In this case I have a folder of .txt files I created using drag and copy and paste from Adobe Acrobat PDFs and Text Editor in OSX. I've been editing these files in vim. Now I'm using grep to find notes in these files. For example,

grep -c "\[t\]" Herbert*

[t] is the note and the directory has 22 files with the name-prefix Herbert* (as in Herbert-02-Transformation.txt, Herbert-14-Classification.txt). The trouble is grep only finds the search string in one file (Herbert-03-Square.txt), but each file has the same note many times.

The trouble files display converted in the status bar when I open them,

"Herbert-02-Transformation.txt" [converted] 276L, 57171C

A post at unix.stack suggests special characters from the original have survived the trip from PDF to vim to cause this trouble with grep. I've been deleting the borked characters and manually inserting the proper :digraphs. Some of the edited files are searchable by grep and other edited files are not. I attempted to use,

:e Herbert-02-Transformation.txt
:set encoding=utf-8` 
:w

but the file still opens as shown above. Now I'm thinking I need to figure out how to compare the encodings of those files where grep is working and those where it is not. There are enough help pages detailing how to set up file detection in .vimrc to avoid this problem that I've spent over 1.5 hours researching this, and I now think its a good question...

Answer 1

The easiest thing to do would be to pass a few lines from one of the problem files through od:

head Herbert-02-Transformation.txt | od -c

Or, if you see nothing strange in the first lines, look for your specific string and a few characters around it:

od -c Herbert-02-Transformation.txt | grep -C 5 "\["

For example:

$ echo "lorem ipsum [t] dolor sit amet" > foo.txt
$ od -c foo.txt 
0000000   l   o   r   e   m       i   p   s   u   m       [   t   ]    
0000020   d   o   l   o   r       s   i   t       a   m   e   t  \n
0000037

Any "weird" characters will be displayed by od.

---

Another useful tool is file which will return the file type and other information:

$ file --mime foo.txt 
foo.txt: text/plain; charset=us-ascii

Answer 2

If you have a text file of a known encoding, with invalid characters in between that do not belong to this encoding, you might be able to filter them out with iconv -c -f charset -t charset file. Careful not to overwrite your original file in the process as the result may be disastrous if you pick the wrong charset.

for example getting rid of umlauts:

$ echo Nähkästchen | iconv -c -f us-ascii -t us-ascii
Nhkstchen

A search&replace may be more appropriate though if those characters you have in there are in any way meaningful. Depends on what's actually in there.