Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm confused on grep/egrep. These tools match a single line I think? In some cases I have heard "logical" line.

Questions

  1. Is it correct to assume that essentially grep/egrep always processes a line at a time up to the new line character?
  2. I'm not really sure what is the "logical line"?
share|improve this question
add comment

2 Answers

up vote 3 down vote accepted

Most of the tools do not actually read a single line in from a file at a time, rather they use a buffer in memory to store chunks of lines. The tools operate a line at a time on the data in this buffer.

NOTE: By "line" I mean split by a \n, in grep's case, or whatever character is denoted as the "delimiter" when the "tool" is invoked.

Here's a example to illustrate this effect.

Sample data

Create a file with 100,000 lines in it. The file is called afile100k.

$ for i in $(seq 100000);do echo "file$i" >> afile100k; done

Using strace

We can utilize strace to peak inside a running process, in this case the grep command.

$ strace -s 2000 -o log100k grep 5 afile100k

This will log the output from strace, up to 2000 characters per line of output to the file log100k. The command we'll be tracing is grep 5 afile100k.

Here's some key output from the log:

ioctl(3, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7fff8bf73b20) = -1 ENOTTY (Inappropriate ioctl for device)

read(3, "file1\nfile2\nfile3\nfile4\nfile5\nfile6\nfile7\nfile8\nfile9\nfile10\nfile11\nfile12\nfile13\nfile14\nfile15\nfile16\nfile17\nfile18\nfile19\nfile20\nfile21\nfile22\nfile23\nfile24\nfile25\nfile26\nfile27\nfile28\nfile29\nfile30\nfile31\nfile32\nfile33\nfile34\nfile35\nfile36\nfile37\nfile38\nfile39\nfile40\nfile41\nfile42\nfile43\nfile44\nfile45\nfile46\nfile47\nfile48\nfile49\nfile50\nfile51\nfile52\nfile53\nfile54\nfile55\nfile56\nfile57\nfile58\nfile59\nfile60\nfile61\nfile62\nfile63\nfile64\nfile65\nfile66\nfile67\nfile68\nfile69\nfile70\nfile71\nfile72\nfile73\nfile74\nfile75\nfile76\nfile77\nfile78\nfile79\nfile80\nfile81\nfile82\nfile83\nfile84\nfile85\nfile86\nfile87\nfile88\nfile89\nfile90\nfile91\nfile92\nfile93\nfile94\nfile95\nfile96\nfile97\nfile98\nfile99\nfile100\nfile101\nfile102\nfile103\nfile104\nfile105\nfile106\nfile107\nfile108\nfile109\nfile110\nfile111\nfile112\nfile113\nfile114\nfile115\nfile116\nfile117\nfile118\nfile119\nfile120\nfile121\nfile122\nfile123\nfile124\nfile125\nfile126\nfile127\nfile128\nfile129\nfile130\nfile131\nfile132\nfile133\nfile134\nfile135\nfile136\nfile137\nfile138\nfile139\nfile140\nfile141\nfile142\nfile143\nfile144\nfile145\nfile146\nfile147\nfile148\nfile149\nfile150\nfile151\nfile152\nfile153\nfile154\nfile155\nfile156\nfile157\nfile158\nfile159\nfile160\nfile161\nfile162\nfile163\nfile164\nfile165\nfile166\nfile167\nfile168\nfile169\nfile170\nfile171\nfile172\nfile173\nfile174\nfile175\nfile176\nfile177\nfile178\nfile179\nfile180\nfile181\nfile182\nfile183\nfile184\nfile185\nfile186\nfile187\nfile188\nfile189\nfile190\nfile191\nfile192\nfile193\nfile194\nfile195\nfile196\nfile197\nfile198\nfile199\nfile200\nfile201\nfile202\nfile203\nfile204\nfile205\nfile206\nfile207\nfile208\nfile209\nfile210\nfile211\nfile212\nfile213\nfile214\nfile215\nfile216\nfile217\nfile218\nfile219\nfile220\nfile221\nfile222\nfile223\nfile224\nfile225\nfile226\nfile227\nfile228\nfile229\nfile230\nfile231\nfile232\nfile233\nfile234\nfile235\nfile236\nfile237\nfile238\nfile239\nfile240\nfile241\nfile242\nfile243\nfile244\nfile245\nfile246\nfile247\nfile248\nfile249\nfile250\nfile251\nfile252\nfile253\nfile254\nfile255\nfile256\nfile257\nfile258\nfile259\nfile260\nfile261\nfile262\nfile263\nfile"..., 32768) = 32768

lseek(3, 32768, SEEK_HOLE)              = 988895
lseek(3, 32768, SEEK_SET)               = 32768

Notice grep is reading in 32k (32768) at a time. NOTE: I've tried to break up the log a bit so that it's easier to read here on SE.

Now it starts writing out results:

fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 5), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcafcfff000
write(1, "file5\n", 6)                  = 6
write(1, "file15\n", 7)                 = 7
write(1, "file25\n", 7)                 = 7
write(1, "file35\n", 7)                 = 7
write(1, "file45\n", 7)                 = 7
write(1, "file50\n", 7)                 = 7

After exhausting out the contents of this buffer it will re-read the next 32k (32768) chunk from the file.

write(1, "file3759\n", 9)               = 9
read(3, "\nfile3765\nfile3766\nfile3767\nfile3768\nfile3769\nfile3770\nfile3771\nfile3772\nfile3773....\nfile3986\nf"..., 32768) = 32768

Followed by more writes:

write(1, "file3765\n", 9)               = 9
write(1, "file3775\n", 9)               = 9

Grep continues to do this until it's completely exhausted the contents of the file, at which point it ends.

write(1, "file99995\n", 10)             = 10
read(3, "", 24576)                      = 0
close(3)                                = 0
close(1)                                = 0
munmap(0x7fcafcfff000, 4096)            = 0
close(2)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++
share|improve this answer
    
Maybe I am misunderstanding your answer but I have noticed that grep/egrep is not able to match patterns that span different lines. So the fact that you say that it processes 32K bytes at a time makes me think that it "splits" this lines using the "\n" separator –  Jim Feb 1 at 23:47
    
@Jim - Correct, my 2nd sentence tried to convey this: "The tools operate a line at a time on the data in this buffer.". –  slm Feb 1 at 23:48
    
@Jim - added more details to that sentence, LMK if that clears things up. –  slm Feb 1 at 23:50
add comment

I'm not sure what you imply by a logical line, but grep (and egrep) work on lines of input.

Quoting from info grep:

`grep' searches input files for lines containing a match to a given pattern list. When it finds a match in a line, it copies the line to standard output (by default), or produces whatever other sort of output you have requested with options.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.