Move files that have the same case-insensitive filename

Question

I have been asked by the data owner to copy a specific folder (and its large amount of subfolders and files) via FTPS to our cloud storage provider. I am using LFTP for that, and the upload worked well until I hit a snag.

There are several folders with multiple files that have the same filename except for case. For example, folder data has the following files: testfile1.txt, TestFile1.TXT

When I try to upload those via LFTP, I get an error that the file already exists. So for my purposes, I need the files to be case-insensitive before uploading. To address this issue, I would like to use a script that searches the current directory recursively, and moves any case-insensitive duplicates to a subfolder. In my example above, I would want the script to create a subfolder called Duplicates and then move TestFile1.TXT into it. I suppose it's possible that I could have multiple duplicated filenames, so the script should create a Duplicates2 folder for the second duplicated filename, and so on.

Also, I should note that for the few "duplicated" files that I checked, they had differing filesizes. I am not going to make any assumptions about the files being actual duplicates, which is why I want to move them rather than delete them.

Answer 1

The bash script below loops through the files in the current directory, looking for duplicate filenames case insensitively. If a match is found, it looks to create a "Duplicates" folder that doesn't exist already, then moves the duplicate file into that directory.

The outer loop is there in order to re-compute the file globs (*) for the loops, in case a file gets moved. The outer loop runs until no files are moved.

#!/bin/bash

changes=1
while [ $changes -gt 0 ]
do
  changes=0
  for one in *
  do
    for two in *
    do
      shopt -u nocasematch
      # if it's the exact same filename, skip
      [[ "$one" == "$two" ]] && continue
      shopt -s nocasematch
      # if the file name matches case-insensitively, then mv it
      if [[ "$one" == "$two" ]]
      then
        suffix=
        while [ -d Duplicates"${suffix}" ]
        do
          suffix=$((suffix + 1))
        done
        mkdir Duplicates"${suffix}"
        mv "$two" Duplicates"${suffix}"
        changes=1
        break
      fi
    done
  done
done

With these sample files:

afile.txt
TestFile1.TXT
TESTfile1.txT
testfile1.txt

A sample run of the script creates:

$ tree .
.
├── afile.txt
├── Duplicates
│   └── TestFile1.TXT
├── Duplicates1
│   └── testfile1.txt
└── TESTfile1.txT

Answer 2

Rather than creating a very complex script to sort out case sensitive you can actually make the FTP case sensitive so you don't need to do this. Refer the LFTP man page for specifics on options: Check if your LFTP command actually is has no case flag in it.

-i, --nocase case-insensitive pattern matching

You can transfer the folders as is file names should valid on the host. Adding more details about destination host would help.

Also I'd recommend to check if you can implement this solution base on file extension.

LFTP mget ignoring case

*.[cC][sS][vV]

I'm sure you don't have that issue as well.

If still the actual requirement is to have a script. You can create a python script using set.