I have a .csv file around 30GB of size. I wanted to grep some lines which should follow multiple string matching condition. What could the correct way to do it using grep, awk or sed? I tried following command which is returning the result but its showing earlier dated data as well.

grep -w "for-outbound-sports\|2019-05-16" Master.csv

Is there any other way to do it faster using awk or sed or something else?


More specifically the sample Input:

"","22288","1990353330","for-outbound-STARZONE","22288","Local/1990353330@for-outbound-STARZONE-00042f49;2","DAHDI/i15/01990353330-c237","Dial","DAHDI/G0/01990353330,30","2019-01-17 13:45:05","2019-01-17 13:45:17","2019-01-17 13:45:32",27,15,"ANSWERED","DOCUMENTATION","1547732705.828852",""
"","22020","1990353330","for-outbound-sports","22020","Local/1990353330@for-outbound-sports-001b223f;2","DAHDI/i14/01990353330-553f8","Dial","DAHDI/G0/01990353330,30","2019-05-15 03:57:02","2019-05-15 03:57:10","2019-05-15 03:57:44",42,34,"ANSWERED","DOCUMENTATION","1557979022.5390225",""
"","22020","1990353330","for-outbound-sports","22020","Local/1990353330@for-outbound-sports-001b223f;2","DAHDI/i14/01990353330-553f8","Dial","DAHDI/G0/01990353330,30","2019-05-16 03:57:02","2019-05-16 03:57:10","2019-05-16 03:57:44",42,34,"ANSWERED","DOCUMENTATION","1557979022.5390225",""

Sample Output:

"","22020","1990353330","for-outbound-sports","22020","Local/1990353330@for-outbound-sports-001b223f;2","DAHDI/i14/01990353330-553f8","Dial","DAHDI/G0/01990353330,30","2019-05-16 03:57:02","2019-05-16 03:57:10","2019-05-16 03:57:44",42,34,"ANSWERED","DOCUMENTATION","1557979022.5390225",""

grep is already a really fast way to go through big files and find words or characters in lines, maybe the -w word-regexp makes it a bit slow. Often it is not the grep itself which is slow, it is mostly the output on the terminal. You can simply test it by direct the output to a file:

grep -w "for-outbound-sports\|2019-05-16" Master.csv > greped_master.csv

You have always the possibility to use the programm parallel to split the big file and make use of multithreading. e.g. parallel --pipe --block 2M grep foo < bigfile As you can find it here

  • I observed that the command grep -w "for-outbound-sports\|2019-05-16" Master.csv > greped_master.csv is not detecting the mentioned date, its detecting previous date as well. Do you have any idea about it? – Rezuan May 22 at 13:20
  • 2
    I think thats because you are looking for lines where is the word 'for-outbound-sports' OR '2019-05-16'. If in a row are one of the upper words, you will get the result. If you want to get only the lines where both words are matching you need to use an AND. grep -w "for-outbound-sports" Master.csv | grep -w "2019-05-16" or in one command grep -Ew "for-outbound-sports.*2019-05-16" Master.csv – Thorsten Beuter May 22 at 15:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.