sort a list based on the datetime portion of records where datetime position in record may vary

Question

I want to sort a list on the datetime portion of its name.

Is this possible using sort? I cannot specify the sort column as the column may vary as shown in sample input below.

swid_ds_install_user_20171227172654_20425.log
package_user_20171227172949_5627.log
swid_state_definition_user_20171227162839_6515.log
swid_ds_install_user_20171227172732_23839.log
swid_appsrv_stop_user_20171227172258_27116.log
package_user_20171227172610_16198.log
swid_state_definition_user_20171227172344_322.log
package_user_20171227233634_23845.log
package_user_20171227162858_7082.log

I can reverse the order of fields through e.g.

awk -F_ '{for (i=NF;i>0;i--){printf $i"_"};printf "\n"}'

then sort with -d_ -k2,2 then reverse order of fields back to retain the original file name - purging residual delimiters with e.g. sed - but this gets awkward.

awk -F_ '{for (i=NF;i>0;i--){printf $i"_"};printf "\n"}' | sort -t'_' -k2,2 \
| awk -F_ '{for (i=NF;i>0;i--){printf $i"_"};printf "\n"}' | sed 's/^_//' \ 
| sed 's/_$//'

How would you approach this?

I was thinking along the lines of using sed to break out the datetime portion via regex and pipe that into sort and then use some built-in to recover the full filename and not only the matched regex, when printing the output.

Hoping I didn't produce another duplicate, I cannot really summarise the problem statement

Answer 1

awk -F_ '{print $(NF-1), $0}' | sort -k1,1 -n | cut -d' ' -f2-

This uses awk with _ as the field separator to copy the second-last field (datetime) to the beginning of the line, then uses sort to sort the input numerically on that field only, followed by cut to remove the extra field.

Example output with your sample input saved to a file called file:

$ awk -F_ '{print $(NF-1), $0}' file  | sort -k1,1 -n | cut -d' ' -f2-
swid_state_definition_user_20171227162839_6515.log
package_user_20171227162858_7082.log
swid_appsrv_stop_user_20171227172258_27116.log
swid_state_definition_user_20171227172344_322.log
package_user_20171227172610_16198.log
swid_ds_install_user_20171227172654_20425.log
swid_ds_install_user_20171227172732_23839.log
package_user_20171227172949_5627.log
package_user_20171227233634_23845.log

This assumes that the datetime is always going to be in the second-last field. If that is not the case, then if you are using GNU awk you could capture the pattern which looks like a datetime, and prepend that to the beginning of the line:

$ awk -F_ '{match($0,"_(20[0-9]{12})_",dt); print dt[1], $0}' file |
    sort -k1,1 -n | cut -d' ' -f2-

but I'd be inclined to use perl in this case.

GNU awk's match() function takes an optional third arrgument, the name of an array variable to store any captured matches in. In this case, there's only going to be one capture, so that'll be stored in the first element of the array, e.g. dt[1]. IIRC, POSIX awk still doesn't have any way to capture regex matches.

BTW, the assumption now is that the year is >= 2000. Adjust the regex to suit if that isn't always going to be true for your input data.

Answer 2

You could use zsh globs here like:

printf '%s\n' *_user_*.log(oe:'REPLY=${REPLY##*user_}':)

where oe:...: defines a sorting order based on the given expression. Here where we select the part of the file name to the right of "user_".

To sort on the last 2 _*:

printf '%s\n' *_*_*.log(oe:'REPLY=${(M)REPLY%_*_*}':)

Answer 3

This looks like it should work:

$ perl -e 'sub key($) { $_[0] =~ /(\d+)_\d+\.log$/; return $1; }; 
     @lines = <>; print sort {key($a) cmp key($b)}  @lines;'  < files
swid_state_definition_user_20171227162839_6515.log
package_user_20171227162858_7082.log
swid_appsrv_stop_user_20171227172258_27116.log
swid_state_definition_user_20171227172344_322.log
package_user_20171227172610_16198.log
swid_ds_install_user_20171227172654_20425.log
swid_ds_install_user_20171227172732_23839.log
package_user_20171227172949_5627.log
package_user_20171227233634_23845.log

The subroutine key picks the string of digits based on the fact that the datetime appears to always be the second-to-last part of the filename, before the .log and the other number field. Then we read the input lines in, and print them sorted using the output of key() as the sort key.

Perl's sort can take an inline code block that gets the values to be compared as $a and $b, and cmp returns less-then, equal, or greater-than (comparing as strings).

If the location of the timestamp could actually vary more, we could change the sub to pick a string of 14 digits anywhere in the string, e.g. separated by underscores here:

sub key($) { $_[0] =~ /_(\d{14})_/; return $1; }