The tool to select lines based on their content is grep, as long as the rule to select the content can be expressed as a regular expression.
A regular expression for “starts with ATOM
” is ^ATOM
. A regular expression for “ends with H
” is H$
. Since the two can't overlap, a regular expression for “starts with ATOM
, then contains anything, and ends with H
” is ^ATOM.*H$
.
To select lines that do not match the regular expression, use the -v
option.
grep -v '^ATOM.*H$' large_file.txt >not_atom_h.txt
For more general conditions, especially for a column-based format, you can use awk. Here is an awk program that's equivalent for your sample data: it prints lines where the first column is not ATOM
, or the last column is not H
. In this specific case, there is no advantage to awk, it would be slower and not simpler. I mention it because small variations on your problem, for example if a column was added after the one that may or may not be H
, would make it a lot more difficult to solve with grep.
awk '$1 != "ATOM" || $NF != "H"' large_file.txt >not_atom_h.txt
grep -v H$ very_large_file >results
, orsed '/H$/d' very_large_file >results
.