I want to locate a date which is somewhere in an 8 GB log (text).

Can I somewhat bypass a full sequential read, and first do binary splits of the file (size), or somehow navigating the filesystem inodes (which I know very little about), to start reading from each split point, until I find a suitable offset from where to start my text search for a line cotaining the date?

tail's read of the last line doesn't use a normal sequential read, so I wonder if this facility is somehow available in bash, or would I need to use Python or C/C++... but I am specifically interested in a bash option..


4 Answers 4


It sounds like you want:

tail -c +1048576

or whatever number of bytes you want to skip. The plus sign tells tail to measure from the start of the file instead of the end. If you're using the GNU version of tail you can write that as:

tail -c +1M

To get a fixed number of bytes after the cut, instead of all the rest of the file, just pipe it through head:

tail -c +1048576 | head -c 1024
  • Linux/bash flexibility is awsome (I definitely spent too long switching to Linux). I had just accepted akira's answer, but I've pulled that until I assess this more fully. dd jumps to a specific byte (as does tail), but it's a pain coding around unknown line lengths, and then a call to sed to strip off leading partial lines... It looks like tail|head can do that painlessly (as fast?). I don't understand how head can turn the tap off on tail, but it seems to :) It must be a case of: If head stops receiving, tail stops sending (and stops further reading). Must go.. back tomorrow.
    – Peter.O
    Mar 3, 2011 at 2:06
  • @fred.bear: tail/head are not able to blind guess the line-lengths as well. you have to jump to position x and then you can either look left or right of x for the next \n. it does not matter what the program is called. so, in both cases you jump to x and then use head to look to the right for the next end of line.
    – akira
    Mar 3, 2011 at 9:22
  • tail|head offers the ability to not be concerned at all about dd's count=val. With 'dd', if I don't grab enough data, it's "game over". The flexibility of arbitary line lengths is great. I've written a function for 'dd' which returns the "next nearest" full line and its offset, but I'd prefer to avoid the length issue. I've now tested tail|head, and it initially performs well (to offset=100MB), but slows down dramatically to take 2 min for one access at offset=8GB (I can awk it in 1 min) ... so it's great for smaller file's.. Thanks for making me aware of the tail/head combo :)
    – Peter.O
    Mar 3, 2011 at 12:57
for (( block = 0; block < 16; block += 1 ))
    echo $block; 
    dd if=INPUTFILE skip=$((block*512))MB bs=64 count=1 status=noxfer 2> /dev/null | \
        head -n 1

which .. creates no temp-split files, skips blocks * 512MB of data at each run, reads 64 bytes from that position and limits the output to the first line of that 64 bytes.

you might want to adjust 64 to whatever you think you need.

  • @akira.. This looks really good, but I want to look at it a bit more first.. (so, until tomorrow.....
    – Peter.O
    Mar 2, 2011 at 10:26
  • 1
    @akira.. 'dd' is awsome. It works in well with the binary split search... I can now extract a regex'd line (by its Date key), from a sorted 8G file in under 1 second... So it looks like I'll achieve my 3 second personal target for extracting a range of dates between two keys (inclusive).. excluding the output time, which varies depending on how much is being output.. I'll be using dd for that too... It is a great tool! :)
    – Peter.O
    Mar 6, 2011 at 15:59

I'd try something like this to split the log into 512MiB chunks for quicker parsing.

split <filename> -b 536870912

If you are looking for the file the following would work:

for file in x* ; do
  echo $file
  head -n 1 $file

Use that output to determine which file to grep for your date.

  • Thanks, but it is slower than a sequential search. Have a look at my comments here unix.stackexchange.com/questions/8121/… (rather than re-writing the same thing here)
    – Peter.O
    Mar 2, 2011 at 6:32
  • by using 'split' you touch every single byte once. if you do that, you could just grep the whole 8gb as well.
    – akira
    Mar 2, 2011 at 8:54
  • @sifusam.. I want to do a binary split search (not just split the files) en.wikipedia.org/wiki/Binary_search_algorithm ... so it was a good answer for a differnt question :).. Thanks for answering.. +1 to get you rolling ....
    – Peter.O
    Mar 2, 2011 at 10:32

Here is my script, I am looking for the first line were the first field is matching my number. The lines are sorted according to the first field. I use dd to check the first line of blocks of 128K, then I jump to the block and perform a search. It improve the efficiency is the file is over 1M.

Any comment or correction is appreciated !




max=$( echo $(du $f | cut -f1)" / $bs" | bc );
for i in $(seq 0 $max); do
 n=$(dd bs=${bs}K skip=$i if=$f 2> /dev/null| head -2 | tail -1 | cut -f1)
 if [ $n -gt $search ]; then
  block=`expr $i - 1` 
dd bs=${bs}K skip=$block if=$f 2> /dev/null| tail -n +2 | awk -v search="$search" '$1==search{print;exit 1;};$1>search{exit 1;};';

* EDIT *** grep is much faster and ack even better

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .