File file1.txt contains lines like:


For example:


File file2.csv contains lines like:


For example:


I want to filter file2.csv removing all lines where the value of hash is present also in file1.txt. That's to say:

cat file1.txt | extract <hash> | sed '/<hash>/d' file2.csv

or something like this.

It should be straightforward, but I seem unable to make it work.

Can anyone please provide a working pipeline for this task?

6 Answers 6


cut -d / -f 4 file1.txt | paste -sd '|' | xargs -I{} grep -v -E {} file2.csv


cut -d / -f 4 file1.txt will select the hashes from the first file

paste -sd '|' will join all the hashes into a regular expression ex. H1|H2|H3

xargs -I{} grep -v -E {} file2.csv will invoke grep with the previous pattern as an argument, xargs will replace {} with the content of the STDIN

If you don't have paste you could replace it with tr "\\n" "|" | sed 's/|$//'

  • 3
    +1 but no need for cat, just cut -d / -f 4 file1.txt. Or if you prefer the sequential look, <file1.txt cut -d / -f 4
    – Sparhawk
    Commented Aug 18, 2015 at 6:22
  • @Sparhawk thanks! I didn't know ;-) solution updated :-) Commented Aug 19, 2015 at 7:54

Possible awk solution:

awk 'NR == FNR { x[$4] = 1; next; } { if (!($1 in x)) print $0; }' FS="/" file1.txt FS="," file2.txt

First we read file1.txt using FS (field separator) "/" and create array x with keys values from field $4 which is the hash you want. Next we read second file file2.txt setting FS to be , and check if value of field $1 does not exist as key in array x and if it does not we print it.
The same more idiomatic as proposed in comments could be:

awk 'NR == FNR { x[$4] = 1; next; } !($1 in x)' FS="/" file1.txt FS="," file2.txt
  • I appreciate your effort, but I'm afraid this flies way above my head. I keep hoping a solution based on some sed/grep/cat mixture will be possible. Commented Aug 17, 2015 at 12:12
  • 1
    I'll add an explanation, it is simple. And may be someone will propose a solution with tools you want.
    – taliezin
    Commented Aug 17, 2015 at 12:16
  • Why not just !($1 in x) instead of { if (!($1 in x)) print $0; }
    – iruvar
    Commented Aug 17, 2015 at 12:26
  • @1_CR it is my bad habit, I know it could be more idiomatic but I always think it will be simpler for explanation to OP.
    – taliezin
    Commented Aug 17, 2015 at 12:28
  • @Muzietto still, I think there's no harm starting to learn other tools such as this awk-based solution... in the long run, you'll learn to gravitate towards solutions that can be achieved using lesser pipes for simplicity... :)
    – h.j.k.
    Commented Aug 18, 2015 at 1:55

For GNU sed

sed -z 's%.*/\([^/]*\)/index.html\n%\1\\|%g;s%^%/%;s%\\|$%/d%' file1.csv |
sed -f - file2.csv

where first sed produce list of hashes in sed-command-format like /12ab09f46\|a77b3ff22\|..../d and transfer it to next sed-script which reads above command from input therefore -f - option.
Same with grep

grep -oP '[^/]*(?=/index.html$)' file1.csv | grep -Fvf - file2.csv

or without perl-expresions:

grep -o '[^/]*/index.html$' file1.csv | 
grep -o '^[^/]*' | 
grep -Fvf - file2.csv

or even better with cut:

cut -d/ -f4 file1.csv | grep -Fvf - file2.csv
  • This looks to me what I was looking for. Can you please illustrate it a bit? I cannot see how the second command will remove lines from file2.csv. Commented Aug 17, 2015 at 13:36
  • @Muzietto See updated
    – Costas
    Commented Aug 17, 2015 at 14:51
cut -d, -f1 file2 | while read key ; do 
   #check for appearance in file1 with successful grep:
   #exit status is 0 if pattern is found, only search for at least 1
   #appearance -> to speed it up
   if [[ $(grep -m 1 "/$key/" file1) ]] ; then
      sed "/^$key,/d" -i file2
      #note that we are gradually overwriting file2 (-i option),
      #so make a backup!

Note that the search stings are /$key/ and ^$key, to reduce results to be either between two slashes (file 1) or to be the first entry of a line and followed by a comma (file 2). This should make it safe if keys look like


in file 2, or like


in file 1


I've just tried the following one liner, and it seems to do the job:

 for i in `cat file1.txt  | awk -F"/" '{print $4}'`; do echo "\n $i" ; sed -ri "/^$i,/d" file2.csv ; done

Please replace first -ri with -re to test it. -re does a dry run, and if all is ok you can run it with -ri

  • mmmh, I've redirected the output of your code to a temporary file and it contains about 30k lines, whereas file2.csv has initially 240 and it is supposed to get filtered. Commented Aug 17, 2015 at 12:50
  • Well, I think that is because I print every hash in the first file, when I do the substitution(the echo "\n" $i part). Anyhow if you run it with -ri you don't have to redirect, because it does the substitution in place
    – primero
    Commented Aug 17, 2015 at 13:07
  • Also If you run with -re and redirect you will have file2 repeated for as many hashes you have in the first file. Basically for each hash in the first file it replaces it in the second file and prints the result, so that is why you have so many lines.
    – primero
    Commented Aug 17, 2015 at 13:10

In addition to Gabriele Lana's answer please note that BSD paste command needs dash to be specified to read content from standard input.

manual of paste command

If '-' is specified for one or more of the input files, the standard input is used; standard input is read one line at a time, circularly, for each instance of '-'.

So final need to be change like below

cut -d / -f 4 file1.txt | paste -sd '|' - | xargs -I{} grep -v -E {} file2.csv

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .