Exclude last N lines that has been last modified from list and with file extension ending in .gz

Question

I want to use the find command on AIX to exclude files ending in .gz, and it must also exclude the last 2 lines from the list. For example, inside the directory, I have:

shop14_0_Log0002019754.gz
shop14_0_Log0002019755.gz
shop14_0_Log0002019756.gz
shop14_0_Log0002019757
shop14_0_Log0002019758.gz
shop14_0_Log0002019759.gz
shop14_0_Log0002019760.gz
shop14_0_Log0002019761.gz
shop14_0_Log0002019762

I want to get the output shown below by retrieving only the uncompressed files, but excluding the last 2 files from the bottom:

Output command must achieve:

shop14_0_Log0002019757

I am able to exclude the last 2 lines using the ls command, but how can I do this by excluding files whose names end with .gz? I am struggling to find a way in AIX/UNIX:

ls -ltr | awk '{print $9} |  sed '$d' |  sed '$d'

Using find, I am able to achieve the list of files that are not compressed by excluding .gz from the list, but it includes the last 2 files, which I do not want:

find . -type f ! -name '*\.gz'  -print 

The above find command returns:

./shop14_0_Log0002019757
./shop14_0_Log0002019762

The file shop14_0_Log0002019762 should be excluded from the list and if shop14_0_Log0002019761 was also uncompressed, then it must also be excluded from the list.

The "last 2" entries to be excluded are sorted based on file modification time. My eventual goal is to compress the uncompressed files.

How can I do this?

Answer 1

If you are using bash, and if you are 100% sure your file names will never contain newlines or spaces, you can do:

shopt -s extglob
ls -t !(*gz) | tail -n +3 | while IFS= read -r file; do gzip "$file"; done

The shopt -s extglob enables extended globbing which gives us !(*gz) for "not ending in gz". Then, by using ls -t, we sort by modification time with the neweest first, and tail -n +3 means "print everything starting from the 3rd line", so will skip the first two files. Finally, you pipe that to a while loop to gzip the files. Alternatively, you could also do:

gzip $(ls -t !(*gz) | tail -n +3)"

or

ls -t !(*gz) | tail -n +3 | xargs gzip

Note that this will work only because you can be sure your file names are sane. See https://mywiki.wooledge.org/ParsingLs for why parsing the output of ls is not recommended.

Answer 2

It is unclear should .gz files be excluded before stripping last 2 files or after. Because if before, then only two files stays in your example and both should be omitted, so output will be empty but your contains shop14_0_Log0002019757. I solve task with assumption that "after".

Solution 1, Bash:

#!/bin/bash

files=(*)
newest_1=${files[0]}
newest_2=${files[0]}

for f in "${files[@]}"; do
  if [[ $f -nt $newest_1 ]]; then
    newest_2=$newest_1
    newest_1=$f
  elif [[ $f -nt $newest_2 ]]; then
    newest_2=$f
  fi  
done

if [[ $newest_1 == "$newest_2" ]]; then
    filenames=$newest_1
else
    filenames="${newest_2}\n${newest_1}"
fi
   
echo -e "$filenames" | sed '/\.gz$/d'

Solution 2: uses GNU utilities which AIX doesn't have by default. But it can be useful as demonstration of approach.

find . -maxdepth 1 ! -name '.' -printf "%A@ %f\n" | sort -g | head -n -2 | cut -d' ' -f2 | sed '/\.gz/d'

Explanation

• find . - finds in current directory
• -maxdepth 1 - not recursive, 1 level only
• ! -name '.' - excludes current directory entry (.)
• -printf "%A@ %f\n" | - outputs file's last access time in seconds from 1970 with fractional part and filename
• sort -g | - sorts by float numbers
• head -n -2 | - outputs all lines except of last 2
• cut -d' ' -f2 | - leaves filenames only by cutting first column
• sed '/\.gz$/d' - removes .gz files.

Answer 3

Assuming that you like to use the find command this would probably solve your problem:

find . -type f ! -name '*\.gz' -print | awk '{Q[N++]=$0; N=N%3; if (Q[N]!="") {print Q[N]}}

The awk command fills an n-th length array Q

• Q[N++]

advances through the array modul 3 (longer arrays to skip more elements)

• N=N%3

And print the entry filled 2 inserts before (therefor ignoring the last two). The if skips the first to inserts, when the array is not yet fully initialized

123   123   123   123   123  ...
^^     ^^   ^ ^   ^^     ^^
||     ||   | |   ||     ||
io     io   O i   iO     iO
nu     nu   U n   nU     nU
 t      t   T      T      T

If you prefer to use 'ls' you can change the $0 to the n-th element (in this case there can't be spaces in the file name). Please check as I'm not sure if I fully understand the corner cases for your question.

Answer 4

Using the ksh93 shell (which is standard on newer AIX versions) for the array support, and assuming that the files are expanded in modification-time order (i.e., that the order of the filenames represents the same sequence as modification timestamps), you could use one array to get the entire list of files (useful for extracting the "last two" later), then use another array to find all of the uncompressed files. Gather the final list of files by looping through the uncompressed file list and removing any files that match the "last two" from earlier.

#!/usr/bin/ksh93

files=(*)
uncompressed=( !(*.gz) )

for index in "${!uncompressed[@]}"
do
  if [[ "${uncompressed[index]}" == "${files[-1]}" ]] || \
     [[ "${uncompressed[index]}" == "${files[-2]}" ]]
  then
    unset -v 'uncompressed[index]'
  fi
done

# echo gzip "${uncompressed[@]}"

At the end you have an array named "uncompressed"; you could compress those filenames as demonstrated (remove the echo).