Remove all lines except D

Question

I have scenario where my three huge files Test.txt , Test1.txt and Test2.txt has following details.

H|||||||||||||||||||||||
D||||||||||||||||||||||||
D|||||||||||||||||||||||
H|||||||||||||||||||||
D||||||||||||||||||||||||
D||||||||||||||||||||||||
T||||||||||||||||||||||||

I have to delete all except D lines. It should look like below in all my three files.(more than 10 GB)

D||||||||||||||||||||||||
D|||||||||||||||||||||||
D||||||||||||||||||||||||
D||||||||||||||||||||||||

So after retaining only D's lines in Test.txt, Test2.txt and Test3.txt, I have to merge those into new file.

I have done the above operation using sed.

sed '/^\('D'\)|/!d' $Filename.txt >>  $NewFilename.txt

But because of huge file its taking very long time.

Can we do this operation using any other command in efficient way?

Answer 1

cat Test.txt Test2.txt Test3.txt | LC_ALL=C grep '^D' > newfile.txt

Or:

for file in Test.txt Test2.txt Test3.txt; do
  LC_ALL=C grep '^D' < "$file"
done > newfile.txt

Or if your grep like GNU grep supports the -h option (to avoid printing file names):

LC_ALL=C grep -h '^D' Test.txt Test2.txt Test3.txt > newfile.txt

By using LC_ALL=C we avoid grep trying to parse UTF-8 data. By using ^D, grep will only look at the first character of each line. grep, especially GNU grep is generally a lot faster than sed.

Answer 2

This will most likely be an I/O bound operation (as opposed to CPU-bound), so even if you don't use a regexp-engine like:

grep -F 'D|' Test.txt Test2.txt Test3.txt

it will take a long time, simply because the file has to be tokenised into lines and then scanned for the pattern.

If this is a one off thing and you don't mind coding, you could mmap(3) the whole file into memory and use memmem(3):

char *p;
if ((p = memmem(file, size, "\nD|", 3)) != NULL) {
        /* massage the line, i.e. find the next '\n'
         * and print the region between p+1 and the
         * next '\n' */
}

where file is the pointer to the mmapped buffer and size is the file size. (If this turns out to be helpful, I'm happy to elaborate more).

This approach will still take some time (as your problem is I/O-bound) but at least you'd save the time to tokenise the file into lines.