Batch remove invisible characters from filenames

Question

I have hundreds of files on external drive that were probably copied a lot, and likely recovered at some time too. The filenames seem tho have acquired some weird characters over their lifespan and I need to fix them.

The problem is that I can list them with ls, but I can't do anything to them by referring to their names.

ls -l renders this result:

-rwx------  1 krzysztof  staff  322451 Jun 27  2009 file015.jpg
-rwx------  1 krzysztof  staff  331337 Jun 27  2009 file016.jpg
-rwx------  1 krzysztof  staff  303078 Jun 27  2009 file017.jpg
-rwx------  1 krzysztof  staff  312346 Jun 27  2009 file018.jpg

The files themselves are not corrupted, and if I manually change the name, the problem goes away. I can also create a file with seemingly exactly the same name (and both will be displayed at the same time with ls). This lead me to believe that there is something fishy going on with invisible characters, which I confirmed by running find . -name '*[! -~]*' - it returned a list of all problematic files.

I wouldn't care about these names normally, but some applications and tools apparently have problem with them too, and just straight up don't see them or ignore them.

One example would be a detox tool that I first attempted to fix the names with - it just pretended they don't exist.

exiftool throws No such file or directory. Similarly, piping the ls -1 to a loop, or saving the file names to a file and then piping them always results in various forms No such file or directory. Like if the invisible characters were not preserved when piping or saving?

$ while read p; do
  mv "$p" "$p"
done <`find . -name '*.jpg'`
zsh: no such file or directory: ./file015.jpg

Is there any way to clean the filenames in batch?

EDIT 1:

Per @terdon request I tried ls -v and ls -B - nothing shows up.

However, the ls | head -n1 | od -c is more interesting, and led me to a discovery of my own too. Here I owe you an apology, because in my original question I changed the names as I thought it didn't matter, but it does.

With that, here is the actual output, where file 014 was fixed manually, and the rest is broken:

❯ ls -1
Zdjęcie014.jpg
Zdjęcie015.jpg
Zdjęcie016.jpg
Zdjęcie017.jpg
Zdjęcie018.jpg

And here is ls | head -n2 | od -c - I used n2 to show more than first filename.

❯ ls | head -n2 | od -c
0000000    Z   d   j   e    ̨  **   c   i   e   0   1   4   .   j   p   g
0000020   \n   Z   d   j   e    ̨  **   c   i   e   0   1   5   .   j   p
0000040    g  \n
0000042

As you can see there is a ę character in both files, and it seems to be the same in ls | head -n2 | od -c. But! When I went back to edit and rename the files manually, in 014 (fixed file) when I position cursor after ę and hit backspace, the entire character disappears. This is not the case for other files - pressing backspace on ę changes it to e, and only second backspace removes it.

Thank you @terdon - now we know where the problem is. How can this be fixed in batch?

EDIT 2:

014 is fixed, 015 is broken:

❯ ls | head -n2 | LC_ALL=C sed -n l
Zdje\314\250cie014.jpg$
Zdje\314\250cie015.jpg$

EDIT 3:

As suggested in comments, I attempted to apply solution from How to rename filenames with accents on macOS? but again run into the No such file or directory

❯ rename $'s/\u119/e/g' ./*
Can't rename './Zdjęcie015.jpg' to './Zdjecie015.jpg': No such file or directory

rename $'s/\u0328//g' ./* does nothing both in bash and zsh

EDIT 4:

❯ PERL_UNICODE=ASD find . -depth -execdir rename -n 's/\pM//g' {} +
'Zdjęcie015.jpg' would be renamed to 'Zdjecie015.jpg'

and

❯ PERL_UNICODE=ASD find . -depth -execdir rename 's/\pM//g' {} +
Can't rename 'Zdjęcie015.jpg' to 'Zdjecie015.jpg': No such file or directory

Answer 1

I'm not sure if this really is an answer as I wasn't able to make it work from macOS, but it certainly is a workaround that solved my problem.

Since the external drive was a NAS, I ssh'ed into it and fixed the problem directly. Unfortunately fancy tools like rename were unavailable to me, so I had to craft something using find, mv and sed - and it worked just right!

Here is the command I used:

find ./ -maxdepth 1 -name "*.JPG" -exec sh -c 'mv "$1" "$(echo "$1" | sed s/Zdj.*cie/zdjecie/)"' _ {} \;

P.S. Thanks to the heroes from the comment section for helping along the way!