bash: /bin/sed: Argument list too long

Question

I have two files.  The first (emails) should be cleaned according to the second (domains). First is 15 GB, second is 160 MB.

dom=`cat file2.txt | xargs | sed -e "s/ /|/g"` ; sed -r "/$dom/d" file1.txt >> final_file.txt

This command gives me bash: /bin/sed: Argument list too long.

Answer 1

Sounds like you just want:

grep -Fvf file2.txt file1.txt > final_file.txt

That is store in final_file.txt the lines of file1.txt that contain none of the lines of file2.txt.

Add the -x option if you want the lines of file1.txt that are not in file2.txt. Or -w to match on words (where bar.com would not match in foobar.com or bar.common, but would still match in foo.bar.com.us for instance).

But if we're talking gigabytes of data and megabytes of different strings to look for, even that is going to take ages.

A faster approach with a shell like ksh, zsh or bash with support for process substitution would be:

export LC_ALL=C
comm -23 <(sort file1.txt) <(sort file2.txt) > final_file.txt

Now if as you clarified in comments, file2.txt is meant to be a list of domains and you mean to filter out of file1.txt the lines that end in @ followed by any of those domains, then a more efficient approach would be to use a hash table:

awk -F@ '
  ! domains_processed {excluded[$0]; next}
  ! ($NF in excluded)
  ' file2.txt domains_processed=1 file1.txt > final_file.txt

Problems with your approach:

• useless use of cat (UUOC). cat is to concatenate files. It makes little sense for a single file. You can use xargs < file or < file xargs for xargs stdin to be directly the file instead of a pipe from a cat process which just shoves the contents of the file.

• xargs calls echo by default. While echo joins its arguments with space characters, which you want here, it also performs other things the list of which depends on the implementation. Also xargs expects the input in a very specific format. Here I'd expect you want each of the line of file2.txt to be passed as a separate argument to echo for which you'd need the GNU-specific xargs -rd '\n'. Also xargs will run echo as many times as necessary to avoid the limit of the size of arguments. So the output of xargs will have several lines for a 160MB input.

  To join the lines of a file with a specific character, the command is paste:

  paste -sd '|' file2.txt
  

• Here, you're building a regex for sed -r (-r being a GNU extension) by joining those words with |, but you're not escaping the regexp operators found in those lines. If those are meant to be domain names, then note that . is a regexp operators which matches any character. You'd have bigger problems with other characters. That sed "/$dom/r" would be an arbitrary command execution vulnerability if you didn't have full control over the contents of file2.txt.

• If file2.txt is 160MB large, then so will be $dom (more or less). Sizes of command lines is limited. On Linux, the size of a single argument is also limited (to 128KiB), so you can't pass the sed script via arguments. It would have to be passed with -f.