Is it possible in bash, to start reading a file from an arbitary byte count offset?

Question

I want to locate a date which is somewhere in an 8 GB log (text).

Can I somewhat bypass a full sequential read, and first do binary splits of the file (size), or somehow navigating the filesystem inodes (which I know very little about), to start reading from each split point, until I find a suitable offset from where to start my text search for a line cotaining the date?

tail's read of the last line doesn't use a normal sequential read, so I wonder if this facility is somehow available in bash, or would I need to use Python or C/C++... but I am specifically interested in a bash option..

Answer 1

It sounds like you want:

tail -c +1048576

or whatever number of bytes you want to skip. The plus sign tells tail to measure from the start of the file instead of the end. If you're using the GNU version of tail you can write that as:

tail -c +1M

To get a fixed number of bytes after the cut, instead of all the rest of the file, just pipe it through head:

tail -c +1048576 | head -c 1024

Answer 2

for (( block = 0; block < 16; block += 1 ))
do 
    echo $block; 
    dd if=INPUTFILE skip=$((block*512))MB bs=64 count=1 status=noxfer 2> /dev/null | \
        head -n 1
done

which .. creates no temp-split files, skips blocks * 512MB of data at each run, reads 64 bytes from that position and limits the output to the first line of that 64 bytes.

you might want to adjust 64 to whatever you think you need.

Answer 3

I'd try something like this to split the log into 512MiB chunks for quicker parsing.

split <filename> -b 536870912

If you are looking for the file the following would work:

for file in x* ; do
  echo $file
  head -n 1 $file
done

Use that output to determine which file to grep for your date.

Answer 4

Here is my script, I am looking for the first line were the first field is matching my number. The lines are sorted according to the first field. I use dd to check the first line of blocks of 128K, then I jump to the block and perform a search. It improve the efficiency is the file is over 1M.

Any comment or correction is appreciated !

#!/bin/bash

search=$1;
f=$2;

bs=128;

max=$( echo $(du $f | cut -f1)" / $bs" | bc );
block=$max;
for i in $(seq 0 $max); do
 n=$(dd bs=${bs}K skip=$i if=$f 2> /dev/null| head -2 | tail -1 | cut -f1)
 if [ $n -gt $search ]; then
  block=`expr $i - 1` 
  break;
 fi
done; 
dd bs=${bs}K skip=$block if=$f 2> /dev/null| tail -n +2 | awk -v search="$search" '$1==search{print;exit 1;};$1>search{exit 1;};';

* EDIT *** grep is much faster and ack even better