I have three files in two folders. The files are named, a.txt, b.txt and c.txt were located in A and B folders. I have used an app Full File Mini Comparer which compares the folder and saves to log to the A folder.

The log has some text as follows:

Different: A=/sdcard/A/a.txt B=/sdcard/B/a.txt
Same: A=/sdcard/A/b.txt B=/sdcard/B/b.txt
Different: A=/sdcard/A/c.txt B=/sdcard/B/c.txt

How can I use sed and rm or perhaps some other command to remove/delete the "Same" files permanently.


You have

$ tree
|-- A
|   |-- a.txt
|   |-- b.txt
|   `-- c.txt
`-- B
    |-- a.txt
    |-- b.txt
    `-- c.txt

2 directories, 6 files

Using fdupes:

$ fdupes -1 A B
A/b.txt B/b.txt

fdupes detects duplicates based on file contents. The -1 flag makes it output the filenames of each set of duplicates on a single line. Here, it detects that the b.txt files are identical.

You may use fdupes to delete duplicates:

$ fdupes --delete A B
[1] A/b.txt
[2] B/b.txt

Set 1 of 1, preserve files [1 - 2, all]: 1

   [+] A/b.txt
   [-] B/b.txt

It interactively asks which file to keep (or to keep both). I wrote 1 so the A/b.txt file was kept while B/b.txt was deleted.

See the manual for fdupes (man fdupes). If it's not installed on your system, then use a package manager to install it. It can also be made to automatically delete files without interactive prompting, but care must be taken when running it in this way. Always make a backup of your data before running a command that may delete files.

Note that fdupes will always keep at least one of the duplicates. If you want to delete all duplicates, then you may be interested in this patched version of fdupes mentioned in an answer to similar question over at SuperUser: https://superuser.com/a/947770/96962 (I have not tested this).

The reason I suggest using fdupes rather than parsing the log file that you have is that filenames embedded in a text document are difficult to parse correctly. It may not always be difficult (and is this particular example, it would be easy), but note that Unix allows for both spaces and newlines in the names of files and directories. It is technically possible to have a directory called

Same: A=

with a newline embedded in the name.

  • I did use fdupes. And also tried it's delete feature. But I want to delete both files. And have used fdupes, modified with an option none! to preserve none of them. Can't find it now. – PJ547 Oct 13 '18 at 9:19
  • @PJ547 You're thinking of this SuperUser answer: superuser.com/a/947770/96962 – Kusalananda Oct 13 '18 at 11:09
  • Yes, this is the one. Thank you @Kusalananda – PJ547 Oct 13 '18 at 13:29

With awk:

awk -F'[:]' '/Same:/{print $0}' logfile | xargs -n1 | awk -F'=' '{print $2}' | xargs rm -rf

awk looks for the line in the log file that contains the keyword "Same:", then xargs organize the variables and paths (i.e. A=***) one per line, after then awk captures the absolute path. In the final step, xargs calls for rm to delete the paths.

You should be aware that when xargs calls for rm to delete the paths, this will delete files definitely. The flag -I can be added to rm to remind the user to confirm deletion.

rm man

-I prompt once before removing more than three files, or when removing recursively; less intrusive than -i, while still giving protection against most mistakes

or with grep

grep "Same:" logfile | grep -oP '/[^ ]*' | xargs rm -rf

The first grep finds the line that contain the keyword Same.

The second grep fillers the logfile to get rid of everything except the paths relevant to the keyword. Finally, xargs calls for rm to delete the paths.

  • 2
    This would fail for files containing : or = in their names. – Kusalananda Oct 12 '18 at 15:53
  • 2
    I think it's perfectly fine to answer a question with those caveats, as long as they are actually mentioned. Having : and = in filenames is unusual (: is often part of individual messages in maildir mailboxes though), and it's even more unusual to have newlines in filenames. Spaces are fairly common though, for example on standard macOS systems, and I've never worked out what xargs do with space-delimited data (I tend to not use xargs). – Kusalananda Oct 12 '18 at 16:01
  • 1
    Calling rm -rf on the output of a script that essentially reformats an input file seems dangerous. In fact, this script is vulnerable to path traversal attacks: if a string given as a filename looks like /home/username for example, this could delete your home directory without any safety check or confirmation. I wouldn't feel safe running this command, even if I had written the input file myself. You can always make mistakes. – Malte Skoruppa Oct 12 '18 at 16:08
  • The first approach didn't do anything. However, the second one, gave me unrecognized option -- P. After looking on the web. Found it works but only before 10.x something. Still, thanks for this. – PJ547 Oct 13 '18 at 9:22
  • @PJ547 strange why first command did not do anything? I just run the command and i used paths similar to yours. What is your system? Did you get any error message? – user88036 Oct 13 '18 at 9:27

Do you REALLY want to delete all identical files, or just n-1 and keep one copy? Then, why not

awk  '/Same:/ {for (i=2; i<=NF; i++) {split ($i, T, "="); print "rm", T[2]}}' log 
rm /sdcard/A/b.txt
rm /sdcard/B/b.txt

and pipe into sh when happy with the result. If you want too keep one copy, start the loop from i=3.

Or, a different approach without awk:

echo rm $(md5sum path/to/files* | sort | uniq -Dw33 | cut -d" " -f3-)
rm file2 file4

Remove the echo if happy with the result. Should files have spaces in their names, additional steps needed to be taken.

  • Impressive. The first approach works perfectly. Can save it to .sh by adding "> a.sh" at the end. Thanks a lot. Also I wanted to know, how can I use the first approach, for fdupes output where the Same: isn't in the output. – PJ547 Oct 13 '18 at 8:59
  • You don't need to save it to a script file - you can either just pipe it | sh, or source it like . <(awk...) if your shell provides "process substitution". I don't have any experience with fdupes, so I just guess: use rm $(fdupes...), if you don't want to use fdupes's delete feature. – RudiC Oct 13 '18 at 9:11
  • I will try. Yes, totally forgot about the pipe function. Doing this after a long time. – PJ547 Oct 13 '18 at 9:17

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.