2

I have a file that looks like the following:

TITLE     Protein in water t=   0.00000
REMARK    THIS IS A SIMULATION BOX
ATOM      1  N   SER A 107      20.799  63.728  25.985  1.00  0.00           N
ATOM      2  H1  SER A 107      21.658  64.259  25.980  1.00  0.00           H

This is a very large file: 1.6G and a little over 20 million lines. I would like to get the lines that do not start with ATOM and end with H and save them into another file. What would be the most efficient way to do this?

6
  • 4
    grep -v H$ very_large_file >results, or sed '/H$/d' very_large_file >results.
    – Satō Katsura
    Jul 27, 2016 at 15:22
  • 2
    How do you measure efficiency? Time it takes to enter the command? Execute the command? Remember how it works 2 years from now?
    – Jeff Schaller ♦
    Jul 27, 2016 at 16:17
  • 1
    What if a "REMARK" line - or the TITLE line - ends with "H" ?
    – Jeff Schaller ♦
    Jul 27, 2016 at 16:18
  • @JeffSchaller There is no such line in my file, but that is a valid concern. To make it more general, maybe we can search lines that start with ATOM and end with H?
    – sodiumnitrate
    Jul 27, 2016 at 16:25
  • 1
    Since this is a PDB file it has fixed column (Fortran) assignments, so a rather 'efficient' equivalent approach would be (that's nine dots): grep -v '^ATOM.........H' myfile.pdb > hydrogen_suppressed_file.pdb
    – Theophrastus
    Jul 28, 2016 at 0:18

4 Answers 4

Reset to default
3

Based on the clarification from the comments,

sed -n '/^ATOM.*H$/!p' input > output

will remove (not print) lines that start with "ATOM" and end with "H" from the file named input and print the rest of the lines into the file named output. The sed syntax goes, from left to right:

  • -n -- don't print lines by default
  • /^ATOM.*H$/ -- look for lines that start with ATOM, followed by any number of characters, ending ($) with H
  • !p -- print lines that don't match the above pattern

A sample input file of:

TITLE     Protein in water t=   0.00000
REMARK    THIS IS A SIMULATION BOX
ATOM      1  N   SER A 107      20.799  63.728  25.985  1.00  0.00           N
ATOM      2  H1  SER A 107      21.658  64.259  25.980  1.00  0.00           H
TITLE     Protein in water t=   0.00000H
REMARK    THIS IS A SIMULATION BOXH
ATOM      1  N   SER A 107      20.799  63.728  25.985  1.00  0.00           N
ATOM      2  H1  SER A 107      21.658  64.259  25.980  1.00  0.00           H

results in:

TITLE     Protein in water t=   0.00000
REMARK    THIS IS A SIMULATION BOX
ATOM      1  N   SER A 107      20.799  63.728  25.985  1.00  0.00           N
TITLE     Protein in water t=   0.00000H
REMARK    THIS IS A SIMULATION BOXH
ATOM      1  N   SER A 107      20.799  63.728  25.985  1.00  0.00           N

A more direct sed syntax would be:

sed '/^ATOM.*H$/d' input > output

which says:

  • (print lines by default)
  • search for lines that start with ATOM and end with H
  • delete (don't print) those lines
2
  • or with grep: grep -v '^ATOM.*H$' input > output. grep prints matching lines, and -v inverts the match. Of course the latter sed in this answer is more efficient by one character. :)
    – ilkkachu
    Jul 27, 2016 at 16:55
  • @ikkachu, grep -xv ATOM.\*H wins.
    – Stéphane Chazelas
    Jul 28, 2016 at 17:30
2

The tool to select lines based on their content is grep, as long as the rule to select the content can be expressed as a regular expression.

A regular expression for “starts with ATOM” is ^ATOM. A regular expression for “ends with H” is H$. Since the two can't overlap, a regular expression for “starts with ATOM, then contains anything, and ends with H” is ^ATOM.*H$.

To select lines that do not match the regular expression, use the -v option.

grep -v '^ATOM.*H$' large_file.txt >not_atom_h.txt

For more general conditions, especially for a column-based format, you can use awk. Here is an awk program that's equivalent for your sample data: it prints lines where the first column is not ATOM, or the last column is not H. In this specific case, there is no advantage to awk, it would be slower and not simpler. I mention it because small variations on your problem, for example if a column was added after the one that may or may not be H, would make it a lot more difficult to solve with grep.

awk '$1 != "ATOM" || $NF != "H"' large_file.txt >not_atom_h.txt
0

In terms of length of the command line, the shortest I can think of is:

grep -vx ATOM.\*H

In terms of processing speed, on my system at least, the quickest I've found was:

mawk '!/^ATOM.*H$/'
-3

(NOT THE BEST SOLUTION, per comments below)

You could also do:

grep '[H$]' source-file > target-file

Placing "H$" (which means "ends with H") within square brackets negates it. So grep will flag entries that dont match "H$"

6
  • Could you explain how this works?
    – sodiumnitrate
    Jul 27, 2016 at 16:29
  • Edited my entry
    – Nwunye
    Jul 27, 2016 at 16:32
  • In what version of grep does placing characters inside square brackets indicate regex negation? It looks to me like your expression prints any lines that contain either H or (literal) $
    – steeldriver
    Jul 27, 2016 at 16:43
  • Um, no, the brackets will match any character listed (and like most others, $ has no special meaning inside a bracket expression), and because there's no anchor, the whole pattern will match any line with an H or a $ anywhere. OTOH [^H]$ would match anything except an H just before end-of-line, and grep -v 'H$' would match any lines that don't end in anH;
    – ilkkachu
    Jul 27, 2016 at 16:45
  • If I was to guess, you saw this sort of grep syntax while trying to look for a specific process while also trying to omit the grep process. The behavior is distinct from the meaning.
    – Jeff Schaller ♦
    Jul 27, 2016 at 16:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .