Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have scenario where my three huge files Test.txt , Test1.txt and Test2.txt has following details.

H|||||||||||||||||||||||
D||||||||||||||||||||||||
D|||||||||||||||||||||||
H|||||||||||||||||||||
D||||||||||||||||||||||||
D||||||||||||||||||||||||
T||||||||||||||||||||||||

I have to delete all except D lines. It should look like below in all my three files.(more than 10 GB)

D||||||||||||||||||||||||
D|||||||||||||||||||||||
D||||||||||||||||||||||||
D||||||||||||||||||||||||

So after retaining only D's lines in Test.txt, Test2.txt and Test3.txt, I have to merge those into new file.

I have done the above operation using sed.

sed '/^\('D'\)|/!d' $Filename.txt >>  $NewFilename.txt

But because of huge file its taking very long time.

Can we do this operation using any other command in efficient way?

share|improve this question
2  
Personally, I would've written it as sed -ne /^D/p', but by and large, if you don't know any other constraints on your file (fixed line length, for example), your task involves scanning the 10G for '\nD' no matter what. –  Ulrich Schwarz Aug 23 '13 at 5:48
    
Of course you could try running this in parallel using split on the files. Just be patient, sit out the time (an hour or so?), and do something useful in the meantime. I have script/programs running for hours, days, weeks or even months. I don't wait until they're done :) –  Bernhard Aug 23 '13 at 6:06
    
Obligatory xkcd –  msw Aug 23 '13 at 10:57

2 Answers 2

up vote 13 down vote accepted
cat Test.txt Test2.txt Test3.txt | LC_ALL=C grep '^D' > newfile.txt

Or:

for file in Test.txt Test2.txt Test3.txt; do
  LC_ALL=C grep '^D' < "$file"
done > newfile.txt

Or if your grep like GNU grep supports the -h option (to avoid printing file names):

LC_ALL=C grep -h '^D' Test.txt Test2.txt Test3.txt > newfile.txt

By using LC_ALL=C we avoid grep trying to parse UTF-8 data. By using ^D, grep will only look at the first character of each line. grep, especially GNU grep is generally a lot faster than sed.

share|improve this answer
    
I have tried above command , it works pretty good for huge files. But issue is when i run above command for multiple files like above LC_ALL=C grep '^D' Test.txt Test2.txt Test3.txt > newfile.txt Its removing every thing except D, but for each lines its adding respective file name. Test.txt: D||||||||||||||||||||||| Test.txt: D||||||||||||||||||||||| Test2.txt D|||||||||||||||||||||||| Test3.txt D||||||||||||||||||||||| So How to get rid of file names???? –  UNIXbest Aug 24 '13 at 11:13
    
@UNIXbest. Oops, sorry, answer updated. –  Stéphane Chazelas Aug 24 '13 at 11:43
    
Hello, IF just want to delete first and last lines of the files and merge into single file. How can we achieve this through LC_ALL=C grep command?? –  UNIXbest Sep 3 '13 at 11:56
    
@UNIXbest, that's a different question, and the answer would not involve grep since grep is to print the lines matching given pattern. –  Stéphane Chazelas Sep 3 '13 at 12:24
    
ok i got it. Do we have any other command to remove first and last line of multiple files and merge it into single file. –  UNIXbest Sep 4 '13 at 9:46

This will most likely be an I/O bound operation (as opposed to CPU-bound), so even if you don't use a regexp-engine like:

grep -F 'D|' Test.txt Test2.txt Test3.txt

it will take a long time, simply because the file has to be tokenised into lines and then scanned for the pattern.

If this is a one off thing and you don't mind coding, you could mmap(3) the whole file into memory and use memmem(3):

char *p;
if ((p = memmem(file, size, "\nD|", 3)) != NULL) {
        /* massage the line, i.e. find the next '\n'
         * and print the region between p+1 and the
         * next '\n' */
}

where file is the pointer to the mmapped buffer and size is the file size. (If this turns out to be helpful, I'm happy to elaborate more).

This approach will still take some time (as your problem is I/O-bound) but at least you'd save the time to tokenise the file into lines.

share|improve this answer
    
you are looking for "D|" anywhere in the line, not at the beginning. change grep -F 'D|' to grep -F '^D' (which will also speed things up, as grep knows it can just go to the next line without testing the rest of it) –  Olivier Dulac Aug 23 '13 at 14:28
    
@OlivierDulac That won't work, grep -F allows only literal strings, and yes I'm aware of this caveat, I assumed the OP's files would all look like their example and not have a D| in the middle of a line. –  hroptatyr Aug 23 '13 at 14:38
    
then just grep '^D', which will speed things up still ^^ –  Olivier Dulac Aug 23 '13 at 14:43

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.