3

Multilingual environment and our file storage on linux of course allows all those international (Asian) characters in filenames. But this is causing problems syncing to other systems because the international characters all get treated by the other systems as ??? or something, and they are no longer unique filenames to those systems (culprit is MS Onedrive/Sharepoint that we're trying to sync to) -- two files in the same directory with different names (Asian characters) but having the same number of characters get treated as having identical file names, unfortunately, and we get a copy error. There seems to be no workaround except renaming files.

I want to simply append the files' own datetimes in ASCII to their filenames and this should solve it.

This command works beautifully for finding all the files that are 'problematic' for me, so to speak -- it locates all non-ASCII characters in file and folder names in /path/to/files/ and all subfolders:

find /path/to/files/ | grep -P "[\x80-\xFF]"

What I need to do with those files is append the base file name with the file's own datetime in ASCII (leaving file extension alone, as is).

Pretty much any datetime would work, either the birth, modify, change date as per stat. The information returned by these would work, for example:

stat -c '%y' filename

date -r filename

I can't figure out how to automatically append these datetimes for the files to the base filenames of the files listed by that first find command.

3
  • As far as I can see, both OneDrive and SharePoint handle Unicode filenames quite happily. We've got Cyrillic filenames, for example. I suspect your issue is something else
    – roaima
    Aug 23 '20 at 17:49
  • 1
    @roaima - Yes that could be (and I've asked Microsoft Support, given them all the information they need to reproduce it and my own HAR traces etc., but haven't heard any further solutions). I will post a question separately on sharepoint.stackexchange.com as this happens simply when trying to upload from a Windows computer to Sharepoint/OneDrive online and is not linux-specific.
    – SSP
    Aug 24 '20 at 1:34
  • 1
    @roaima sharepoint.stackexchange.com/questions/284075 --It works fine for Cyrillic and I'm sure many others. It does not work for Tibetan/Choekey/Dzongkha. Sample: མགོན་པོ་ཀླུ་སྒྲུབ་ཀྱི་རྣམ་ཐར། ༠༡.txt and མགོན་པོ་ཀླུ་སྒྲུབ་ཀྱི་རྣམ་ཐར། ༠༢.txt --filenames are different but treated 'equivalent' by SharePoint/OneDrive.
    – SSP
    Aug 24 '20 at 11:30
2

With zsh:

#! /bin/zsh -
zmodload zsh/stat
zmodload zsh/files # for its builtin mv to speed things up.
set +o multibyte -o extendedglob

# comment-out the line below once you're satisfied it does what you want.
mv() { printf 'Would rename %s to %s\n' ${(q+)2} ${(q+)3}; }

ts_format='-%FT%T.%3.%z'

# we build a $ts_pattern to be able to identify files that already have
# had a timestamp appended, by obtaining a sample timestamp for the /
# directory, and replacing all digits in it with [0-9]. That assumes you
# don't use wildcard characters nor day/month/timezone names, am/pm in your
# $ts_format.
stat -F $ts_format -A ts_sample +mtime /
ts_pattern=${ts_sample//[0-9]/[0-9]}

for file in **/(*[$'\x80'-$'\xff']*~*$~ts_pattern(.*|))(DNod); do
  stat -LF $ts_format -A ts +mtime -- $file || continue
  case $file:t in
    (?*.*) mv -- $file $file:r$ts.$file:e;; # insert ts before extension
    (*)    mv -- $file $file$ts;;
  esac
done

I hear some Microsoft OSes have issues with filenames that contain : characters so you may have to adapt the timestamp format (here in international standard format with millisecond precision 2020-08-23T08:14:38.318+0100). That's standard strftime() formatting directives, except for the %<precision>. subsecond part which is zsh-specific (there's no equivalent in strftime()).

7
  • Thank you! I assume this will take care of the file names with spaces: echo mv -- "'"$file"'" "'"$file:r-$ts.$file:e"'";; and echo mv -- "'"$file"'" "'"$file-$ts"'" (double quote, single quote, double quote on either end). Now I just have to figure out how to get the datetime in the format yyyy-mm-dd-hhmmss, maybe even milliseconds if available.
    – SSP
    Aug 23 '20 at 8:30
  • Unfortunately not quite working. The echo comes out great: mv -- 'test/༠༡ ༠༢.txt' 'test/༠༡ ༠༢-2020-08-23_14h57m49.149s.txt' but after removing 'echo' the command fails: ./addts.sh:mv:9: 'test/\M-\M-<\M- \M-\M-<\M-! \M-\M-<\M- \M-\M-<\M-".txt': no such file or directory
    – SSP
    Aug 23 '20 at 9:02
  • @SSP, no, adding those single quote characters don't make sense here, you'd need to remove them. Aug 23 '20 at 9:18
  • thank you, you are right and it does work as per your original answer. While testing I ran the script multiple times and I realized this will keep renaming the same file over and over again, so it just keeps getting appended with the datetime repeatedly if the script is run again. Is there an easy check that can be added in to prevent that?
    – SSP
    Aug 23 '20 at 9:29
  • 1
    @SSP Since it works perfectly, consider accepting the answer. This is how we mark the question as solved and say thanks at Stack.
    – Quasímodo
    Aug 23 '20 at 15:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.