I have scenario where my three huge files Test.txt , Test1.txt and Test2.txt has following details.


I have to delete all except D lines. It should look like below in all my three files.(more than 10 GB)


So after retaining only D's lines in Test.txt, Test2.txt and Test3.txt, I have to merge those into new file.

I have done the above operation using sed.

sed '/^\('D'\)|/!d' $Filename.txt >>  $NewFilename.txt

But because of huge file its taking very long time.

Can we do this operation using any other command in efficient way?

  • 2
    Personally, I would've written it as sed -ne /^D/p', but by and large, if you don't know any other constraints on your file (fixed line length, for example), your task involves scanning the 10G for '\nD' no matter what. Aug 23, 2013 at 5:48
  • Of course you could try running this in parallel using split on the files. Just be patient, sit out the time (an hour or so?), and do something useful in the meantime. I have script/programs running for hours, days, weeks or even months. I don't wait until they're done :)
    – Bernhard
    Aug 23, 2013 at 6:06
  • Obligatory xkcd
    – msw
    Aug 23, 2013 at 10:57

2 Answers 2

cat Test.txt Test2.txt Test3.txt | LC_ALL=C grep '^D' > newfile.txt


for file in Test.txt Test2.txt Test3.txt; do
  LC_ALL=C grep '^D' < "$file"
done > newfile.txt

Or if your grep like GNU grep supports the -h option (to avoid printing file names):

LC_ALL=C grep -h '^D' Test.txt Test2.txt Test3.txt > newfile.txt

By using LC_ALL=C we avoid grep trying to parse UTF-8 data. By using ^D, grep will only look at the first character of each line. grep, especially GNU grep is generally a lot faster than sed.

  • I have tried above command , it works pretty good for huge files. But issue is when i run above command for multiple files like above LC_ALL=C grep '^D' Test.txt Test2.txt Test3.txt > newfile.txt Its removing every thing except D, but for each lines its adding respective file name. Test.txt: D||||||||||||||||||||||| Test.txt: D||||||||||||||||||||||| Test2.txt D|||||||||||||||||||||||| Test3.txt D||||||||||||||||||||||| So How to get rid of file names????
    – UNIXbest
    Aug 24, 2013 at 11:13
  • @UNIXbest. Oops, sorry, answer updated. Aug 24, 2013 at 11:43
  • Hello, IF just want to delete first and last lines of the files and merge into single file. How can we achieve this through LC_ALL=C grep command??
    – UNIXbest
    Sep 3, 2013 at 11:56
  • @UNIXbest, that's a different question, and the answer would not involve grep since grep is to print the lines matching given pattern. Sep 3, 2013 at 12:24
  • ok i got it. Do we have any other command to remove first and last line of multiple files and merge it into single file.
    – UNIXbest
    Sep 4, 2013 at 9:46

This will most likely be an I/O bound operation (as opposed to CPU-bound), so even if you don't use a regexp-engine like:

grep -F 'D|' Test.txt Test2.txt Test3.txt

it will take a long time, simply because the file has to be tokenised into lines and then scanned for the pattern.

If this is a one off thing and you don't mind coding, you could mmap(3) the whole file into memory and use memmem(3):

char *p;
if ((p = memmem(file, size, "\nD|", 3)) != NULL) {
        /* massage the line, i.e. find the next '\n'
         * and print the region between p+1 and the
         * next '\n' */

where file is the pointer to the mmapped buffer and size is the file size. (If this turns out to be helpful, I'm happy to elaborate more).

This approach will still take some time (as your problem is I/O-bound) but at least you'd save the time to tokenise the file into lines.

  • you are looking for "D|" anywhere in the line, not at the beginning. change grep -F 'D|' to grep -F '^D' (which will also speed things up, as grep knows it can just go to the next line without testing the rest of it) Aug 23, 2013 at 14:28
  • @OlivierDulac That won't work, grep -F allows only literal strings, and yes I'm aware of this caveat, I assumed the OP's files would all look like their example and not have a D| in the middle of a line.
    – hroptatyr
    Aug 23, 2013 at 14:38
  • then just grep '^D', which will speed things up still ^^ Aug 23, 2013 at 14:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .