Print first and last line of all files in folder

Question

I have a bunch of log files that get overwritten (file.log.1, file.log.2 etc). When I copy them from the device making them onto my local machine I lose the original time stamps. So I'd like to put them in chronological order. The problem is that I don't necessarily know which is the newest and which is the oldest.

What I'd like to be able to do is, if all the logs are in a directory, print something like this:

file: file.log.1
first line: [first line that isn't whitespace]
last line: [last line that isn't whitespace]

I can just write a python script to do this, but I'd much rather do it with linux built-ins if possible. Is this a job for awk/sed? Or would this really be better off for a scripting language? If yes to awk/sed, how woul dyou go about doing it?

I found this awk command by searching, but it only accepts one file name and will print whatever the last line is (and there can be a variable number of empty lines at the end)

awk 'NR == 1 { print }END{ print }' filename

Answer 1

So I like sed the answer can be

for file in file.log.*
do
   echo "file: $file"
   echo -n "first line: "
   cat "$file" | sed -n '/^\s*$/!{p;q}'
   echo -n "last line: "
   tac "$file" | sed -n '/^\s*$/!{p;q}'
done

Answer 2

an awk command:

awk -v OFS=: '
    FNR==1 {
        # the last non-blank line from the previous file
        if (line) {print filename, fnr, line}
        filename=FILENAME
        line=""
        p=0
    } 
    /^[[:blank:]]*$/ {next} 
    !p {
        # the first non-blank line
        print FILENAME, FNR, $0; p=1
    }
    {fnr=FNR; line=$0} 
    END {print filename, fnr, line}
' *

for each file, print the filename, line number and line, colon-separated.

GNU awk v4 has BEGINFILE and ENDFILE patterns which simplify things quite a bit:

gawk -v OFS=: '
    BEGINFILE {p=0} 
    /^[[:blank:]]*$/ {next} 
    !p {print FILENAME, FNR, $0; p=1}
    {fnr=FNR; line=$0} 
    ENDFILE {print FILENAME, fnr, line}
' *

Answer 3

Try:

awk -F'\n' -vRS="" '
  {
    print "file: " FILENAME;
    gsub(/\n[[:blank:]]+|[[:blank:]]+\n/,"");
    print "first line: " $1;
    print "last line: " $NF;
  }
' file.log.*

Answer 4

Another approach would be to use head and tail:

EDIT (Thank you for the suggestion @don_crissti!)

for file in file.log.*
do
   echo "file: $file"
   echo -n "first line: "
   grep -v '^\s*$' "$file" | head -n1
   echo -n "last line: "
   grep -v '^\s*$' "$file" | tail -n1
done

Answer 5

What? No Perl?

for file in file.log.*; do 
    echo "FILE: $file"; 
    perl -ne 'if(/\S/){$k++; $l=$_}; 
              print "First line: $_" if $k==1; 
              END{print "Last line: $l\n"}' "$file";  
done

Explanation

• for file in file.log.* : iterate over all files whose names starts with file.log. in the current directory and save each of them as $file.
• echo "FILE: $file"; : print the file name.
• perl -ne : read the current input file line by line (-n), saving each line as the special Perl variable $_, and run the script given by -e on it.
• if(/\S/){$k++; $l=$_} : if the current line matches a non-whitespace character (\S), save the line as $l and increment the counter $k by one.
• print "First line: $_" if $k==1; : print the current line ($_) if $k is 1. This will print the 1st non-whitespace line.
• END{print "Last line: $l\n"} : this is executed after all input lines have been read. Since we save each non-whitespace line as $l, at the end of the file, $l will be the last non-whitespace line. Therefore, this will print the last line.

---

Another approach:

for file in file.log.*; do 
    printf "FILE: %s\nFirst line: %s\nLast line: %s\n\n" \
        "$file" \
        "$(grep -Em 1 '\S' "$file")" \
        "$(tac "$file" | grep -Em1 '\S' )"; 
done

Explanation

This is the same for loop only here we're using printf to print three strings. The file name, and the output of these two commands:

• grep -Pm 1 '\S' "$file" : The -E activates Extended Regular Expressions which let us use \S for "non-whitespace". The -m1 means "exit after the first match found".
• tac "$file" | grep -Em1 '\S' : tac is the inverse of cat. It will print the contents of a file, but from the last line to the first line. Therefore, this command will print the last non-whitespace line.

Answer 6

For completeness, here is a sed answer that only reads through each file once (hopefully faster than multiple sed, cat and tac invocations, if the files are large):

for file in file.log.*; do
    echo "file: $file"
    sed -n "
/[^[:space:]]/ {                # Match first non-whitespace line
    h                           # Copy to hold buffer
    s/^/first line: / p         # Add prefix and print
:loop
    $ {                         # Match last line
        g                       # Get contents of hold buffer
        s/^/last line: / p      # Add prefix and print
    }
    n                           # Load next line
    /[^[:space:]]/ h            # Copy non-whitespace line to hold buffer
    b loop                      # Jump back to process next line
}" "$file"
done

Answer 7

strm_1st_last() ( unset l i c n
l=1 i='i\\\\' c='c\\\\' n='\
';  eval "echo |sed -n \":b
    /[^[:blank:]]/h;$( wc -l "$@" | 
    sed -e '${1!d' -e "};s/[\"\/]/\\\\&/g
    s/^[ 0]*\([^ 0-9]\)/1$i$n empty: \1/;/\n/b
    s|^ *\([0-9]\{1,\}\) *\(.*\)|\\
    \$((1+(l+=\\1)-\\1)){$i$n\\2$n :l\\
         s/.*[^[:blank:]]/    \\&/p;//h;t\\
         n;\$((l-1)){ //bb$n//!$c$n\ALL BLANK$n};bl$n}\\
    \${l}{ x;s/^/    /p;s/.*//;x;d$n}|")"'" - "$@"'
)

That is a function which relies on one sed process to build a workable script for a second. Well, it does not just rely on a sub-shelled sed, but also wc to first get the total line numbers for each of its arguments, and the shell itself to tally usably the arithmetic expressions sed builds as line references for the second sed.

Basically it handles all arguments as a single stream. For example, if you do:

seq 10 > file1; seq 5 > file2; sed -n \$= file[12]

sed will print...

15

...because it automatically concatenates all filename arguments into a single data stream. So we just work with that.

First wc is asked for a report on all of their lengths, and then sed goes to work moulding that into a workable input script for a second sed.

If wc reports any of the files to have 0 lines sed will print empty :filename for it and all others before anything else. (probably just 0 lines would be more accurate, though).

If, while scanning a section of input lines that comprises an entire file sed encounters no non-blank characters at all, sed will report it ALL BLANK (which is the only type of line report that is not indented)

For all other cases, sed prints the first and last non-blank lines for each of its arguments and precedes each indented pair with the non-indented filename. This does not handle filenames containing newlines - though it probably wouldn't require much more to do it.

So if I do...

strm_1st_last file[12]

file1
    1
    10
file2
    1
    5

or...

: >file2; strm_1st_last file[12]

empty: file2
file1
    1
    10

...or...

strm_1st_last ~/*.sh


empty: /home/mikeserv/alleq.sh
empty: /home/mikeserv/mansed5.sh
/home/mikeserv/chrome.sh
    #!/bin/bash
    ) &
/home/mikeserv/crap.sh
ALL BLANK

/home/mikeserv/getopts.sh
    c=$# t=\     l='
    done; printf '%s\n' "'${str#?}'"; done
/home/mikeserv/mansedmaybe.sh
    mansed () {
    );}
/home/mikeserv/mansed.sh
    mansed () {
    );}
/home/mikeserv/pr_after.sh
    pr_after() if       _i= _o=D OPTIND=1 _n='\
        fi
/home/mikeserv/script.sh
    #!/usr/bin/sh
    echo done
/home/mikeserv/serial.sh
    sq() ( IFS=\' set -f
    #   a               b
/home/mikeserv/your_config.sh
    zifs() { eval "shift ${3+3}
    )
/home/mikeserv/ytplay.sh
    ytplay() ( 
    )
/home/mikeserv/zifs.sh
    zifs() { eval "shift ${3+3}
    )

Answer 8

I just had this same need. I did it like this:

for f in */*.log; do echo $f; head -1 $f; tail -1 $f; done

Answer 9

Try this:

for file in $(ls) ; do cat $file | sed -n '1p;$p' ; done

Answer 10

Using find you could try:

find . -type f -exec sed -n '1p;$p' {} \; -print