0

I'm searching backwards from the end of a file, it's probably easiest to explain by showing my code:

enough=0
numberOfLines=100
fileName="/path/to.file"
# Tail the last $numberOfLines lines; if the line is within the time range we want them double how far we go back
while [ $enough -eq 0 ]; do
        # Get the date of the first line
        dateOfFirstLine=`tail -$numberOfLines $fileName | sed -n 1p | grep -Po "^[0-9]{2}/[0-9]{2}-[0-9]{2}:[0-9]{2}:[0-9]{2}"`

        # See if it's still within the range we're interested in - if so then go back twice as far
        if [[ "$dateOfFirstLine" > "$startDate" ]] ; then
                numberOfLines=$((numberOfLines*2))
        elif [[ "$dateOfFirstLine" == "$startDate" ]] ; then
                numberOfLines=$((numberOfLines*2))
        else
                enough=1
                lines=`tail -$numberOfLines $fileName`
        fi
done

(I actually have an if in there that exits early if the number of lines we're tailing gets too high, but I've cut that out for simplicity)

Obviously I then go on to do some work with $lines.

My question is regarding this line:

dateOfFirstLine=`tail -$numberOfLines $fileName | sed -n 1p | grep -Po "^[0-9]{2}/[0-9]{2}-[0-9]{2}:[0-9]{2}:[0-9]{2}"`

will that be quick and negligible in terms of CPU and I/O, because it only ever reads one line, or will it actually read all the lines but only the first one will move through the pipe? I'm thinking tail still reads every line, right, in which case is there a better way of doing it?

On a related note, how is wc -l so fast even for big files? Is it not reading every character in the file and counting how many newline characters appear? It just seems really quick if that's what it's doing - does the filesystem keep data like that handy or something?

1
  • May be it have sence to find $startDate by sed initially?
    – Costas
    Mar 6 '15 at 18:49
2

Will tail -100 | sed -n 1p only read one line? No, sed will read 100 lines.

If your goal is to read the 100th line from the end of the file, do one of

tail -100 | head -1
tail -100 | sed 1q
6
  • What's the 1q do? I got the 1p from a Google search, but reading the man page for sed I couldn't figure out what it actually does. Thanks, by the way; I'll try out your suggestion on Monday, should make the whole think more efficient :)
    – mazz0
    Mar 6 '15 at 18:10
  • Out of curiosity, why tail -100 | sed -n 1p won't work?
    – heemayl
    Mar 6 '15 at 18:14
  • @Glenn and @mazz0, I think that should be sed '{p;q}'. The form you used seems not to work. @heemayl, that will also work, but sed might continue reading all data comming from the pipe before terminating, and the q commant instructs sed to terminate.
    – Janis
    Mar 6 '15 at 18:23
  • It will word, printing only the 1st line, but it does not quit sed, so the next 99 lines will be processed and not printed. Mar 6 '15 at 18:23
  • @Janis, what do you see with seq 10 | sed 1q? Mar 6 '15 at 18:25
0

The tail commands have some intelligence implemented (so I've heard and also experienced); if you test tail on very large files you will see that it takes no time to get to the last portion of the file. In your case you will only pass $numberOfLines lines through the first pipe and 1 line through the second pipe.

3
  • Ah, but it will pass $numberOfLines lines through the first pipe? I was kinda hoping it wouldn't, somehow, as it's just the first line I need to check at that point.
    – mazz0
    Mar 6 '15 at 18:08
  • With such small data volumes there's likely no difference at all since the pipe will be filled anyway with all passed data until the subsequent process will end. For huge amounts of data that pass the pipe see Glenn's optimization to terminate the subsequent process as early as possible.
    – Janis
    Mar 6 '15 at 18:13
  • Yeah, it won't make any appreciable difference in the case I'm using it, but I figured I'd get it as efficient as possible a) cos that's cooler and b) cos maybe in the future I'll use it on a file that has thousands of rows added per second and my while loop will therefore run many many times. :)
    – mazz0
    Mar 6 '15 at 18:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.