2

I have two files.  The first (emails) should be cleaned according to the second (domains). First is 15 GB, second is 160 MB.

dom=`cat file2.txt | xargs | sed -e "s/ /|/g"` ; sed -r "/$dom/d" file1.txt >> final_file.txt

This command gives me bash: /bin/sed: Argument list too long.

1
  • Posting sample contents of the two files, and at least some of dom, and the size of dom, would help. Being as you say file2 is 160mb and your first sed deletes nothing, I assume dom is also 160MB. And then you pass that as a single arg to sed. That's your issue -- one huge arg in the /../d command. Passing sed a multi-megabyte regex is not going to end well, either. Jan 9, 2021 at 13:23

1 Answer 1

4

Sounds like you just want:

grep -Fvf file2.txt file1.txt > final_file.txt

That is store in final_file.txt the lines of file1.txt that contain none of the lines of file2.txt.

Add the -x option if you want the lines of file1.txt that are not in file2.txt. Or -w to match on words (where bar.com would not match in foobar.com or bar.common, but would still match in foo.bar.com.us for instance).

But if we're talking gigabytes of data and megabytes of different strings to look for, even that is going to take ages.

A faster approach with a shell like ksh, zsh or bash with support for process substitution would be:

export LC_ALL=C
comm -23 <(sort file1.txt) <(sort file2.txt) > final_file.txt

Now if as you clarified in comments, file2.txt is meant to be a list of domains and you mean to filter out of file1.txt the lines that end in @ followed by any of those domains, then a more efficient approach would be to use a hash table:

awk -F@ '
  ! domains_processed {excluded[$0]; next}
  ! ($NF in excluded)
  ' file2.txt domains_processed=1 file1.txt > final_file.txt

Problems with your approach:

  • useless use of cat (UUOC). cat is to concatenate files. It makes little sense for a single file. You can use xargs < file or < file xargs for xargs stdin to be directly the file instead of a pipe from a cat process which just shoves the contents of the file.

  • xargs calls echo by default. While echo joins its arguments with space characters, which you want here, it also performs other things the list of which depends on the implementation. Also xargs expects the input in a very specific format. Here I'd expect you want each of the line of file2.txt to be passed as a separate argument to echo for which you'd need the GNU-specific xargs -rd '\n'. Also xargs will run echo as many times as necessary to avoid the limit of the size of arguments. So the output of xargs will have several lines for a 160MB input.

    To join the lines of a file with a specific character, the command is paste:

    paste -sd '|' file2.txt
    
  • Here, you're building a regex for sed -r (-r being a GNU extension) by joining those words with |, but you're not escaping the regexp operators found in those lines. If those are meant to be domain names, then note that . is a regexp operators which matches any character. You'd have bigger problems with other characters. That sed "/$dom/r" would be an arbitrary command execution vulnerability if you didn't have full control over the contents of file2.txt.

  • If file2.txt is 160MB large, then so will be $dom (more or less). Sizes of command lines is limited. On Linux, the size of a single argument is also limited (to 128KiB), so you can't pass the sed script via arguments. It would have to be passed with -f.

4
  • hello mate! yes, I want lines of file1.txt which are not in file2.txt, the problem is file1 contain emails(jonny@gmail.com) and file2 contain domains (gmail.com etc). Will it work? thx!
    – Jonny
    Jan 9, 2021 at 13:36
  • @Jonny, if that's a substring match, the comm / grep -x would not work. But then the question is if file2 contains gmail.com, should foo@bigmail.com, bar@foo.gmail.com, blah@foo.gmailling.uk be excluded? Jan 9, 2021 at 13:46
  • only @gmail.com should be excluded, without subdomains
    – Jonny
    Jan 9, 2021 at 14:36
  • @Jonny, see edit. Jan 9, 2021 at 14:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.