1

I have been asked by the data owner to copy a specific folder (and its large amount of subfolders and files) via FTPS to our cloud storage provider. I am using LFTP for that, and the upload worked well until I hit a snag.

There are several folders with multiple files that have the same filename except for case. For example, folder data has the following files: testfile1.txt, TestFile1.TXT

When I try to upload those via LFTP, I get an error that the file already exists. So for my purposes, I need the files to be case-insensitive before uploading. To address this issue, I would like to use a script that searches the current directory recursively, and moves any case-insensitive duplicates to a subfolder. In my example above, I would want the script to create a subfolder called Duplicates and then move TestFile1.TXT into it. I suppose it's possible that I could have multiple duplicated filenames, so the script should create a Duplicates2 folder for the second duplicated filename, and so on.

Also, I should note that for the few "duplicated" files that I checked, they had differing filesizes. I am not going to make any assumptions about the files being actual duplicates, which is why I want to move them rather than delete them.

  • What is the filesystem you're copying to? Most native POSIX filesystems are entirely case-sensitive, so testfile1.txt and TestFile1.txt are in fact two different, nonconflicting filenames. – DopeGhoti Apr 9 '18 at 15:35
  • I just tested the cloud storage platform's case-sensitivity by uploading a file, changing the filename case, then uploading the original file again. It overwrote the renamed file. – Joe Apr 9 '18 at 17:58
0

The bash script below loops through the files in the current directory, looking for duplicate filenames case insensitively. If a match is found, it looks to create a "Duplicates" folder that doesn't exist already, then moves the duplicate file into that directory.

The outer loop is there in order to re-compute the file globs (*) for the loops, in case a file gets moved. The outer loop runs until no files are moved.

#!/bin/bash

changes=1
while [ $changes -gt 0 ]
do
  changes=0
  for one in *
  do
    for two in *
    do
      shopt -u nocasematch
      # if it's the exact same filename, skip
      [[ "$one" == "$two" ]] && continue
      shopt -s nocasematch
      # if the file name matches case-insensitively, then mv it
      if [[ "$one" == "$two" ]]
      then
        suffix=
        while [ -d Duplicates"${suffix}" ]
        do
          suffix=$((suffix + 1))
        done
        mkdir Duplicates"${suffix}"
        mv "$two" Duplicates"${suffix}"
        changes=1
        break
      fi
    done
  done
done

With these sample files:

afile.txt
TestFile1.TXT
TESTfile1.txT
testfile1.txt

A sample run of the script creates:

$ tree .
.
├── afile.txt
├── Duplicates
│   └── TestFile1.TXT
├── Duplicates1
│   └── testfile1.txt
└── TESTfile1.txT
  • Thank you for the script! Unfortunately I'm getting a syntax error, and being a Linux noob, I don't know how to resolve it. – Joe Apr 11 '18 at 15:07
  • Did you run it with bash, @Joe? What's the error you get? – Jeff Schaller Apr 11 '18 at 15:12
  • I created a .sh file with the code, and here's what I got: [username@text test]$ MvDuplicates.sh /home/username/scripts/MvDuplicates.sh: line 6: syntax error near unexpected toke' do 'home/username/scripts/MvDuplicates.sh: line 6: do – Joe Apr 11 '18 at 15:16
  • I then explicitly ran it with bash to be sure: "bash -x MvDuplicates.sh" – Joe Apr 11 '18 at 15:17
  • [username@text test]$ bash -x MvDuplicates.sh + LMOD_PKG=/usr/share/lmod/lmod + LMOD_DIR=/usr/share/lmod/lmod/libexec + LMOD_CMD=/usr/share/lmod/lmod/libexec/lmod + MODULESHOME=/usr/share/lmod/lmod + export LMOD_PKG + export LMOD_CMD + export LMOD_DIR + export MODULESHOME + '[' -z -XqeMRF ']' + '[' -z more ']' + LMOD_VERSION=6.0.15 + export LMOD_VERSION + '[' : '!=' : ']' + unalias ml + true ++ echo YES ++ tr '[:upper:]' '[:lower:]' + export_module=yes + '[' -n '4.1.2(1)-release' -a yes '!=' no ']' + export -f module + export -f ml – Joe Apr 11 '18 at 15:17
0

Rather than creating a very complex script to sort out case sensitive you can actually make the FTP case sensitive so you don't need to do this. Refer the LFTP man page for specifics on options: Check if your LFTP command actually is has no case flag in it.

-i, --nocase case-insensitive pattern matching

You can transfer the folders as is file names should valid on the host. Adding more details about destination host would help.

Also I'd recommend to check if you can implement this solution base on file extension.

LFTP mget ignoring case

*.[cC][sS][vV]

I'm sure you don't have that issue as well.

If still the actual requirement is to have a script. You can create a python script using set.

  • Thank you, but this won't help since my FTP destination filesystem is not case sensitive. – Joe Apr 11 '18 at 15:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.