Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have two files, one being a superset of the other. I want to remove the identical lines in the smaller files from the larger file.

One possible complication is that the lines contain backslashes.

How do I do this?

share|improve this question

Here is my snippet:

    # remove lines from a file 
    # $1 - source file with patterns of lines to be removed
    # $2 - destination file
    tmpfile=$(mktemp "$(dirname -- "$2")"/XXXXXXXX) &&
    grep -F -f "$1" -v -- "$2" >>"$tmpfile" &&
    mv -- "$tmpfile" "$2" &&

EDIT: I've just realized that there is no sed in it, but that wasn't critical, was it?

share|improve this answer
You might want to use mktemp for generating the temporary file name, instead of tmp-$(uuidgen) or similar hacks. – Riccardo Murri Jun 20 '11 at 13:06
OP presupposed that sed was necessary, but didn't specify why another solution would be unacceptable. +1. – Kevin M Jun 20 '11 at 13:40
I used the line in your script like so: grep -F -f "uniq_failing_specs.txt" -v -- "all_specs.txt" >>"passing_specs.txt" – thekingoftruth Oct 16 '12 at 11:36

Try the following script;

## $1 - Small File
## $2 - Large File

sed 's/^/\//; s/$/\/d/; s/\\/\\\\/g' $1 > $HOME/sed_scpt.txt
sed 's/\\/\\\\/g' $2 | sed -f $HOME/sed_scpt.txt > $HOME/desired_output.txt

## Alternatively, you could change the 2nd line with the following;
sed -f $HOME/sed_scpt.txt $2 > $HOME/desired_output.txt

Hope this helps.

NOTE: I've used GNU sed 4.2.1, if that matters as well.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.