Currently my mbsync is erroring out with a bunch of duplicated UIDs. This means I have to find all files that have duplicates of the sub-string "U=[0-9]+:" and change the most recent file to remove that sub-string.

For example, a simple case on one file would be:

$> fd ".*U=17:.*" --exec ls -la {} \;
-rw-------  1 djm  staff  95903 Mar  2 06:57 cur/1583291317.13980_115.DJM2,U=17:2,ST
-rw-------  1 djm  staff  13654 Sep 30  2015 cur/1580615936.64042_2698.DJM2,U=17:2,S
$> mv ./cur/1583291317.13980_115.DJM2,U=17:2,ST ./cur/1583291317.13980_115.DJM2

The problem is, I have about 1000 of these to do. So I'm trying to

  1. find all files that have that matching sub-string,
  2. sort them by the matching sub-string,
  3. show only the duplicate matching files
  4. Of those duplicates change the name of the latest file to remove that sub-string.

I'm having trouble finding all the duplicates while keeping the file-path intact to use later when changing the file name.

This is what I have so far:

fd ".*U=[0-9]+:.*" | sort -t , -k2.3n | xargs -I{} printf "%s\t${%s//.*,}\n" "{}" "{}"

Note, I'm using the rust utility fd similar to find.

I find all the files matching the pattern, I then numerically sort the output split on , starting at the third character.

This is where I'm having trouble filtering it down to duplicates as just piping into uniq -d gives me nothing because of the file-paths. So I'm thinking, I could printf the sub-string spaced away from the file path and then pipe that into uniq -d.

I'm not quite sure how to move forward. An answer would inform me how to filter my output down to duplicated file-paths based on the specified sub-string.

  • you could add -u to the sort invocation to pick the first of dupes but that's not guaranteed to be the latest file, it'll be the first among equals as reported by fd
    – iruvar
    Mar 7, 2020 at 18:20
  • Step 3 and step 4 are getting me confused.. Here is how I would do the first 2 steps. (change date to whatever you need) find ./cur/ -type f -newerXY '2020-01-10' | egrep "U=[0-9]*:" | sed 's/,/ /' | awk '{print $2, $1}' | sort -n After that command, I would use the output and run it through a for a loop. for i in $(ls);do name=$(echo $i|sed 's/,.*//g'); mv $i $name;done Mar 7, 2020 at 19:02

1 Answer 1


I found a solution by adding an extra step thanks to a suggestion by Heysus,

x=($(ls -lR ./cur | grep -o 'U=.*:' | sort -k1.3n | uniq -d))

for i in "${x[@]}"; do                                                                                        
  y=$(fd ".*${i}" --exec gstat -c '%X %n' \; | sort -nr | awk 'NR==1,NR==1 {print $2}')
  newy=$(echo "$y" | sed 's/,.*//g')
  mv "$y" "$newy"

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .