Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I want to locate a date which is somewhere in an 8 GB log (text).

Can I somewhat bypass a full sequential read, and first do binary splits of the file (size), or somehow navigating the filesystem inodes (which I know very little about), to start reading from each split point, until I find a suitable offset from where to start my text search for a line cotaining the date?

tail's read of the last line doesn't use a normal sequential read, so I wonder if this facility is somehow available in bash, or would I need to use Python or C/C++... but I am specifically interested in a bash option..

share|improve this question
add comment

4 Answers

up vote 6 down vote accepted
for (( block = 0; block < 16; block += 1 ))
do 
    echo $block; 
    dd if=INPUTFILE skip=$((block*512))MB bs=64 count=1 status=noxfer 2> /dev/null | \
        head -n 1
done

which .. creates no temp-split files, skips blocks * 512MB of data at each run, reads 64 bytes from that position and limits the output to the first line of that 64 bytes.

you might want to adjust 64 to whatever you think you need.

share|improve this answer
    
@akira.. This looks really good, but I want to look at it a bit more first.. (so, until tomorrow..... –  Peter.O Mar 2 '11 at 10:26
    
@akira.. 'dd' is awsome. It works in well with the binary split search... I can now extract a regex'd line (by its Date key), from a sorted 8G file in under 1 second... So it looks like I'll achieve my 3 second personal target for extracting a range of dates between two keys (inclusive).. excluding the output time, which varies depending on how much is being output.. I'll be using dd for that too... It is a great tool! :) –  Peter.O Mar 6 '11 at 15:59
add comment

It sounds like you want:

tail -c +1048576

or whatever number of bytes you want to skip. The plus sign tells tail to measure from the start of the file instead of the end. If you're using the GNU version of tail you can write that as:

tail -c +1M

To get a fixed number of bytes after the cut, instead of all the rest of the file, just pipe it through head:

tail -c +1048576 | head -c 1024
share|improve this answer
    
Linux/bash flexibility is awsome (I definitely spent too long switching to Linux). I had just accepted akira's answer, but I've pulled that until I assess this more fully. dd jumps to a specific byte (as does tail), but it's a pain coding around unknown line lengths, and then a call to sed to strip off leading partial lines... It looks like tail|head can do that painlessly (as fast?). I don't understand how head can turn the tap off on tail, but it seems to :) It must be a case of: If head stops receiving, tail stops sending (and stops further reading). Must go.. back tomorrow. –  Peter.O Mar 3 '11 at 2:06
    
@fred.bear: tail/head are not able to blind guess the line-lengths as well. you have to jump to position x and then you can either look left or right of x for the next \n. it does not matter what the program is called. so, in both cases you jump to x and then use head to look to the right for the next end of line. –  akira Mar 3 '11 at 9:22
    
tail|head offers the ability to not be concerned at all about dd's count=val. With 'dd', if I don't grab enough data, it's "game over". The flexibility of arbitary line lengths is great. I've written a function for 'dd' which returns the "next nearest" full line and its offset, but I'd prefer to avoid the length issue. I've now tested tail|head, and it initially performs well (to offset=100MB), but slows down dramatically to take 2 min for one access at offset=8GB (I can awk it in 1 min) ... so it's great for smaller file's.. Thanks for making me aware of the tail/head combo :) –  Peter.O Mar 3 '11 at 12:57
add comment

I'd try something like this to split the log into 512MiB chunks for quicker parsing.

split <filename> -b 536870912

If you are looking for the file the following would work:

for file in x* ; do
  echo $file
  head -n 1 $file
done

Use that output to determine which file to grep for your date.

share|improve this answer
    
Thanks, but it is slower than a sequential search. Have a look at my comments here unix.stackexchange.com/questions/8121/… (rather than re-writing the same thing here) –  Peter.O Mar 2 '11 at 6:32
    
by using 'split' you touch every single byte once. if you do that, you could just grep the whole 8gb as well. –  akira Mar 2 '11 at 8:54
    
@sifusam.. I want to do a binary split search (not just split the files) en.wikipedia.org/wiki/Binary_search_algorithm ... so it was a good answer for a differnt question :).. Thanks for answering.. +1 to get you rolling .... –  Peter.O Mar 2 '11 at 10:32
add comment

Here is my script, I am looking for the first line were the first field is matching my number. The lines are sorted according to the first field. I use dd to check the first line of blocks of 128K, then I jump to the block and perform a search. It improve the efficiency is the file is over 1M.

Any comment or correction is appreciated !

#!/bin/bash

search=$1;
f=$2;

bs=128;

max=$( echo $(du $f | cut -f1)" / $bs" | bc );
block=$max;
for i in $(seq 0 $max); do
 n=$(dd bs=${bs}K skip=$i if=$f 2> /dev/null| head -2 | tail -1 | cut -f1)
 if [ $n -gt $search ]; then
  block=`expr $i - 1` 
  break;
 fi
done; 
dd bs=${bs}K skip=$block if=$f 2> /dev/null| tail -n +2 | awk -v search="$search" '$1==search{print;exit 1;};$1>search{exit 1;};';

* EDIT *** grep is much faster and ack even better

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.