Currently my mbsync is erroring out with a bunch of duplicated UIDs. This means I have to find all files that have duplicates of the sub-string "U=[0-9]+:" and change the most recent file to remove that sub-string.

For example, a simple case on one file would be:

$> fd ".*U=17:.*" --exec ls -la {} \;
-rw-------  1 djm  staff  95903 Mar  2 06:57 cur/1583291317.13980_115.DJM2,U=17:2,ST
-rw-------  1 djm  staff  13654 Sep 30  2015 cur/1580615936.64042_2698.DJM2,U=17:2,S
$> mv ./cur/1583291317.13980_115.DJM2,U=17:2,ST ./cur/1583291317.13980_115.DJM2

The problem is, I have about 1000 of these to do. So I'm trying to

  1. find all files that have that matching sub-string,
  2. sort them by the matching sub-string,
  3. show only the duplicate matching files
  4. Of those duplicates change the name of the latest file to remove that sub-string.

I'm having trouble finding all the duplicates while keeping the file-path intact to use later when changing the file name.

This is what I have so far:

fd ".*U=[0-9]+:.*" | sort -t , -k2.3n | xargs -I{} printf "%s\t${%s//.*,}\n" "{}" "{}"

Note, I'm using the rust utility fd similar to find.

I find all the files matching the pattern, I then numerically sort the output split on , starting at the third character.

This is where I'm having trouble filtering it down to duplicates as just piping into uniq -d gives me nothing because of the file-paths. So I'm thinking, I could printf the sub-string spaced away from the file path and then pipe that into uniq -d.

I'm not quite sure how to move forward. An answer would inform me how to filter my output down to duplicated file-paths based on the specified sub-string.

  • you could add -u to the sort invocation to pick the first of dupes but that's not guaranteed to be the latest file, it'll be the first among equals as reported by fd
    – iruvar
    Mar 7 '20 at 18:20
  • Step 3 and step 4 are getting me confused.. Here is how I would do the first 2 steps. (change date to whatever you need) find ./cur/ -type f -newerXY '2020-01-10' | egrep "U=[0-9]*:" | sed 's/,/ /' | awk '{print $2, $1}' | sort -n After that command, I would use the output and run it through a for a loop. for i in $(ls);do name=$(echo $i|sed 's/,.*//g'); mv $i $name;done Mar 7 '20 at 19:02

I found a solution by adding an extra step thanks to a suggestion by Heysus,

x=($(ls -lR ./cur | grep -o 'U=.*:' | sort -k1.3n | uniq -d))

for i in "${x[@]}"; do                                                                                        
  y=$(fd ".*${i}" --exec gstat -c '%X %n' \; | sort -nr | awk 'NR==1,NR==1 {print $2}')
  newy=$(echo "$y" | sed 's/,.*//g')
  mv "$y" "$newy"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.