I have a bunch of gzipped text files. I am trying to remove duplicate files (using fdupes) but the problem is these files contain a line that has their timestamp in them, and the files are identical otherwise.

I'd like to be able to find duplicates, ignoring this line, but not remove the line from the files that I keep.

In my case the line is of the following form:

-- Dump completed on 2014-07-12 10:00:01

Right now I am using the following script, which works but removes the timestamp line from the kept files:


# Remove timestamp line from all gzipped text files by temporarily unzipping
# them, removing the line then rezipping. Preserve file system timestamp.
for a in *.sql.gz ; do
    gunzip -c $a | sed -e 's/^-- Dump completed.*//g' | gzip -c -9 > temp.gz
    touch -r $a temp.gz
    mv temp.gz $a

# Duplicates can now be removed.
fdupes . -dN

I want to find a solution that doesn't involve modifying the original files, leaving the timestamp line in.

Is there some way I can do this?

  • 1
    Why do you insist to keep the timestamp in the file, if the file name is also the timestamp? – Rabin Jul 13 '14 at 10:05
  • @Rabin I don't rely on file system information (names and timestamps) for important data, and it complicates processing in general as well (for example piping a file whose name is the sole source of a piece of information to another program loses the information). – Jason C Jul 13 '14 at 14:51
  • you can try a file system which support dedup. – Rabin Jul 13 '14 at 20:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.