4

I have a bunch of log files that get overwritten (file.log.1, file.log.2 etc). When I copy them from the device making them onto my local machine I lose the original time stamps. So I'd like to put them in chronological order. The problem is that I don't necessarily know which is the newest and which is the oldest.

What I'd like to be able to do is, if all the logs are in a directory, print something like this:

file: file.log.1
first line: [first line that isn't whitespace]
last line: [last line that isn't whitespace]

I can just write a python script to do this, but I'd much rather do it with linux built-ins if possible. Is this a job for awk/sed? Or would this really be better off for a scripting language? If yes to awk/sed, how woul dyou go about doing it?

I found this awk command by searching, but it only accepts one file name and will print whatever the last line is (and there can be a variable number of empty lines at the end)

awk 'NR == 1 { print }END{ print }' filename
4
  • 3
    What are you using to copy the files? If you are copying to another linux file system then you should be able to preserve user, timestamp data. e.g. cp -p
    – bsd
    Dec 23, 2014 at 18:11
  • It's a windows CE system that I'm copying from to my linux host
    – Mitch
    Dec 23, 2014 at 19:03
  • 3
    Even when copying from a FAT filesystem, cp -a or cp -p should preserve the timestamp.
    – bsd
    Dec 23, 2014 at 19:36
  • 2
    Looks like classic example of XY Problem
    – jimmij
    Dec 23, 2014 at 22:29

10 Answers 10

6

So I like sed the answer can be

for file in file.log.*
do
   echo "file: $file"
   echo -n "first line: "
   cat "$file" | sed -n '/^\s*$/!{p;q}'
   echo -n "last line: "
   tac "$file" | sed -n '/^\s*$/!{p;q}'
done
3
  • 1
    No need to echo+cat and echo+tac, you could just sed -n '/^[[:space:]]*$/!{s/^/first line: /p;q}' "$file" and sed -n '/^[[:space:]]*$/!h;$!d;g;s/^/last line: /p' "$file" Dec 24, 2014 at 1:52
  • @don_crissti Nice. I like such decisions. But echo+tac much simple and, in case of big file, quicker.
    – Costas
    Dec 24, 2014 at 11:24
  • It works like a charm. But I really don't know what the thing is you put into the sed argument, especially the {p; g} part. Could you give any resouce on this. Much appreciated. Dec 20, 2021 at 8:25
5

an awk command:

awk -v OFS=: '
    FNR==1 {
        # the last non-blank line from the previous file
        if (line) {print filename, fnr, line}
        filename=FILENAME
        line=""
        p=0
    } 
    /^[[:blank:]]*$/ {next} 
    !p {
        # the first non-blank line
        print FILENAME, FNR, $0; p=1
    }
    {fnr=FNR; line=$0} 
    END {print filename, fnr, line}
' *

for each file, print the filename, line number and line, colon-separated.

GNU awk v4 has BEGINFILE and ENDFILE patterns which simplify things quite a bit:

gawk -v OFS=: '
    BEGINFILE {p=0} 
    /^[[:blank:]]*$/ {next} 
    !p {print FILENAME, FNR, $0; p=1}
    {fnr=FNR; line=$0} 
    ENDFILE {print FILENAME, fnr, line}
' *
2

Try:

awk -F'\n' -vRS="" '
  {
    print "file: " FILENAME;
    gsub(/\n[[:blank:]]+|[[:blank:]]+\n/,"");
    print "first line: " $1;
    print "last line: " $NF;
  }
' file.log.*
0
2

Another approach would be to use head and tail:

EDIT (Thank you for the suggestion @don_crissti!)

for file in file.log.*
do
   echo "file: $file"
   echo -n "first line: "
   grep -v '^\s*$' "$file" | head -n1
   echo -n "last line: "
   grep -v '^\s*$' "$file" | tail -n1
done
2
  • As per the question, OP wants to print the "first & last lines that aren't whitespace". So you could do something like grep -v '^\s*$' "$file" | head -n 1 and grep -v '^\s*$' "$file" | tail -n 1 to get the expected result. Dec 23, 2014 at 23:20
  • As a variant you can strip head command if use option -m1 for grep
    – Costas
    Dec 24, 2014 at 11:40
2

What? No Perl?

for file in file.log.*; do 
    echo "FILE: $file"; 
    perl -ne 'if(/\S/){$k++; $l=$_}; 
              print "First line: $_" if $k==1; 
              END{print "Last line: $l\n"}' "$file";  
done

Explanation

  • for file in file.log.* : iterate over all files whose names starts with file.log. in the current directory and save each of them as $file.
  • echo "FILE: $file"; : print the file name.
  • perl -ne : read the current input file line by line (-n), saving each line as the special Perl variable $_, and run the script given by -e on it.
  • if(/\S/){$k++; $l=$_} : if the current line matches a non-whitespace character (\S), save the line as $l and increment the counter $k by one.
  • print "First line: $_" if $k==1; : print the current line ($_) if $k is 1. This will print the 1st non-whitespace line.
  • END{print "Last line: $l\n"} : this is executed after all input lines have been read. Since we save each non-whitespace line as $l, at the end of the file, $l will be the last non-whitespace line. Therefore, this will print the last line.

Another approach:

for file in file.log.*; do 
    printf "FILE: %s\nFirst line: %s\nLast line: %s\n\n" \
        "$file" \
        "$(grep -Em 1 '\S' "$file")" \
        "$(tac "$file" | grep -Em1 '\S' )"; 
done

Explanation

This is the same for loop only here we're using printf to print three strings. The file name, and the output of these two commands:

  • grep -Pm 1 '\S' "$file" : The -E activates Extended Regular Expressions which let us use \S for "non-whitespace". The -m1 means "exit after the first match found".
  • tac "$file" | grep -Em1 '\S' : tac is the inverse of cat. It will print the contents of a file, but from the last line to the first line. Therefore, this command will print the last non-whitespace line.
1

For completeness, here is a sed answer that only reads through each file once (hopefully faster than multiple sed, cat and tac invocations, if the files are large):

for file in file.log.*; do
    echo "file: $file"
    sed -n "
/[^[:space:]]/ {                # Match first non-whitespace line
    h                           # Copy to hold buffer
    s/^/first line: / p         # Add prefix and print
:loop
    $ {                         # Match last line
        g                       # Get contents of hold buffer
        s/^/last line: / p      # Add prefix and print
    }
    n                           # Load next line
    /[^[:space:]]/ h            # Copy non-whitespace line to hold buffer
    b loop                      # Jump back to process next line
}" "$file"
done
3
  • I think you could use a one-liner... actually a two-liner as there's a new line after the backslash: for file in file.log.*; do sed -n '/[^[:space:]]/ {h;s/^/file: '"$file"'\ enter a new line here first line: /p;:loop;${g;s/^/last line: /p};n;/[^[:space:]]/h; b loop}' "$file"; done Dec 23, 2014 at 22:46
  • @don_crissti sure - pretty much replace newlines with ;. I find the indented, commented version easier to read though :) Dec 23, 2014 at 22:51
  • 1
    I know that. I meant without the "echo file: $file" part, if you read the code in my comment. Also, don't forget to quote the last $file otherwise your solution will fail for file names with spaces and stuff. Dec 23, 2014 at 22:54
0
strm_1st_last() ( unset l i c n
l=1 i='i\\\\' c='c\\\\' n='\
';  eval "echo |sed -n \":b
    /[^[:blank:]]/h;$( wc -l "$@" | 
    sed -e '${1!d' -e "};s/[\"\/]/\\\\&/g
    s/^[ 0]*\([^ 0-9]\)/1$i$n empty: \1/;/\n/b
    s|^ *\([0-9]\{1,\}\) *\(.*\)|\\
    \$((1+(l+=\\1)-\\1)){$i$n\\2$n :l\\
         s/.*[^[:blank:]]/    \\&/p;//h;t\\
         n;\$((l-1)){ //bb$n//!$c$n\ALL BLANK$n};bl$n}\\
    \${l}{ x;s/^/    /p;s/.*//;x;d$n}|")"'" - "$@"'
)

That is a function which relies on one sed process to build a workable script for a second. Well, it does not just rely on a sub-shelled sed, but also wc to first get the total line numbers for each of its arguments, and the shell itself to tally usably the arithmetic expressions sed builds as line references for the second sed.

Basically it handles all arguments as a single stream. For example, if you do:

seq 10 > file1; seq 5 > file2; sed -n \$= file[12]

sed will print...

15

...because it automatically concatenates all filename arguments into a single data stream. So we just work with that.

First wc is asked for a report on all of their lengths, and then sed goes to work moulding that into a workable input script for a second sed.

If wc reports any of the files to have 0 lines sed will print empty :filename for it and all others before anything else. (probably just 0 lines would be more accurate, though).

If, while scanning a section of input lines that comprises an entire file sed encounters no non-blank characters at all, sed will report it ALL BLANK (which is the only type of line report that is not indented)

For all other cases, sed prints the first and last non-blank lines for each of its arguments and precedes each indented pair with the non-indented filename. This does not handle filenames containing newlines - though it probably wouldn't require much more to do it.

So if I do...

strm_1st_last file[12]

file1
    1
    10
file2
    1
    5

or...

: >file2; strm_1st_last file[12]

empty: file2
file1
    1
    10

...or...

strm_1st_last ~/*.sh


empty: /home/mikeserv/alleq.sh
empty: /home/mikeserv/mansed5.sh
/home/mikeserv/chrome.sh
    #!/bin/bash
    ) &
/home/mikeserv/crap.sh
ALL BLANK

/home/mikeserv/getopts.sh
    c=$# t=\     l='
    done; printf '%s\n' "'${str#?}'"; done
/home/mikeserv/mansedmaybe.sh
    mansed () {
    );}
/home/mikeserv/mansed.sh
    mansed () {
    );}
/home/mikeserv/pr_after.sh
    pr_after() if       _i= _o=D OPTIND=1 _n='\
        fi
/home/mikeserv/script.sh
    #!/usr/bin/sh
    echo done
/home/mikeserv/serial.sh
    sq() ( IFS=\' set -f
    #   a               b
/home/mikeserv/your_config.sh
    zifs() { eval "shift ${3+3}
    )
/home/mikeserv/ytplay.sh
    ytplay() ( 
    )
/home/mikeserv/zifs.sh
    zifs() { eval "shift ${3+3}
    )
0

I just had this same need. I did it like this:

for f in */*.log; do echo $f; head -1 $f; tail -1 $f; done

-1

Try this:

for file in $(ls) ; do cat $file | sed -n '1p;$p' ; done
-2

Using find you could try:

find . -type f -exec sed -n '1p;$p' {} \; -print
1
  • Did you notice that the question says that the file(s) can contain lines that are entirely whitespace, and the OP wants the first and last lines that have non-blank content?  No, I guess you didn’t. Oct 2, 2017 at 22:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .