I have a bunch of gzipped text files. I am trying to remove duplicate files (using
fdupes) but the problem is these files contain a line that has their timestamp in them, and the files are identical otherwise.
I'd like to be able to find duplicates, ignoring this line, but not remove the line from the files that I keep.
In my case the line is of the following form:
-- Dump completed on 2014-07-12 10:00:01
Right now I am using the following script, which works but removes the timestamp line from the kept files:
#!/bin/sh # Remove timestamp line from all gzipped text files by temporarily unzipping # them, removing the line then rezipping. Preserve file system timestamp. for a in *.sql.gz ; do gunzip -c $a | sed -e 's/^-- Dump completed.*//g' | gzip -c -9 > temp.gz touch -r $a temp.gz mv temp.gz $a done # Duplicates can now be removed. fdupes . -dN
I want to find a solution that doesn't involve modifying the original files, leaving the timestamp line in.
Is there some way I can do this?