2

I want to use the find command on AIX to exclude files ending in .gz, and it must also exclude the last 2 lines from the list. For example, inside the directory, I have:

shop14_0_Log0002019754.gz
shop14_0_Log0002019755.gz
shop14_0_Log0002019756.gz
shop14_0_Log0002019757
shop14_0_Log0002019758.gz
shop14_0_Log0002019759.gz
shop14_0_Log0002019760.gz
shop14_0_Log0002019761.gz
shop14_0_Log0002019762

I want to get the output shown below by retrieving only the uncompressed files, but excluding the last 2 files from the bottom:

Output command must achieve:

shop14_0_Log0002019757

I am able to exclude the last 2 lines using the ls command, but how can I do this by excluding files whose names end with .gz? I am struggling to find a way in AIX/UNIX:

ls -ltr | awk '{print $9} |  sed '$d' |  sed '$d'

Using find, I am able to achieve the list of files that are not compressed by excluding .gz from the list, but it includes the last 2 files, which I do not want:

find . -type f ! -name '*\.gz'  -print 

The above find command returns:

./shop14_0_Log0002019757
./shop14_0_Log0002019762

The file shop14_0_Log0002019762 should be excluded from the list and if shop14_0_Log0002019761 was also uncompressed, then it must also be excluded from the list.

The "last 2" entries to be excluded are sorted based on file modification time. My eventual goal is to compress the uncompressed files.

How can I do this?

2
  • 2
    Why not simply ls -tr | sed '$d' | sed '$d' | sed '/gz$/d'? (head version allowing negative numbers required and no filenames with linebreak allowed)
    – Philippos
    Jul 18, 2022 at 10:37
  • 1
    Do you want to use the file's modification time, or do you want to parse the date from the file name? Also, please edit your question and explain the final objective here. Do you simply need to make sure that all files except the two most recent ones are compressed?
    – terdon
    Jul 18, 2022 at 10:47

4 Answers 4

1

If you are using bash, and if you are 100% sure your file names will never contain newlines or spaces, you can do:

shopt -s extglob
ls -t !(*gz) | tail -n +3 | while IFS= read -r file; do gzip "$file"; done

The shopt -s extglob enables extended globbing which gives us !(*gz) for "not ending in gz". Then, by using ls -t, we sort by modification time with the neweest first, and tail -n +3 means "print everything starting from the 3rd line", so will skip the first two files. Finally, you pipe that to a while loop to gzip the files. Alternatively, you could also do:

gzip $(ls -t !(*gz) | tail -n +3)"

or

ls -t !(*gz) | tail -n +3 | xargs gzip

Note that this will work only because you can be sure your file names are sane. See https://mywiki.wooledge.org/ParsingLs for why parsing the output of ls is not recommended.

2
  • This works well. I changed it in a way that I use the: ls -ltr !(*gz) | awk '{print $9}' | sed '$d' | sed '$d'. From this I can do what I need Jul 18, 2022 at 11:52
  • 1
    @ChristopherKarsten that only makes it worse: that way it will break if the date format changes (which will happen for older files), or anything else. Adding -l only complicates things and provides absolutely nothing. Using the two sed calls is inefficient, but won't make a difference apart from that, but the -l means you need the extra awk which means that anything can break it. Please at least use ls -tr !(*gz) | sed '$d' | sed '$d'.
    – terdon
    Jul 18, 2022 at 12:29
0

It is unclear should .gz files be excluded before stripping last 2 files or after. Because if before, then only two files stays in your example and both should be omitted, so output will be empty but your contains shop14_0_Log0002019757. I solve task with assumption that "after".

Solution 1, Bash:

#!/bin/bash

files=(*)
newest_1=${files[0]}
newest_2=${files[0]}

for f in "${files[@]}"; do
  if [[ $f -nt $newest_1 ]]; then
    newest_2=$newest_1
    newest_1=$f
  elif [[ $f -nt $newest_2 ]]; then
    newest_2=$f
  fi  
done

if [[ $newest_1 == "$newest_2" ]]; then
    filenames=$newest_1
else
    filenames="${newest_2}\n${newest_1}"
fi
   
echo -e "$filenames" | sed '/\.gz$/d'

Solution 2: uses GNU utilities which AIX doesn't have by default. But it can be useful as demonstration of approach.

find . -maxdepth 1 ! -name '.' -printf "%A@ %f\n" | sort -g | head -n -2 | cut -d' ' -f2 | sed '/\.gz/d'

Explanation

  • find . - finds in current directory
  • -maxdepth 1 - not recursive, 1 level only
  • ! -name '.' - excludes current directory entry (.)
  • -printf "%A@ %f\n" | - outputs file's last access time in seconds from 1970 with fractional part and filename
  • sort -g | - sorts by float numbers
  • head -n -2 | - outputs all lines except of last 2
  • cut -d' ' -f2 | - leaves filenames only by cutting first column
  • sed '/\.gz$/d' - removes .gz files.
5
  • head -n -2 will probably not work with AIX head
    – Philippos
    Jul 18, 2022 at 10:41
  • 1
    And -maxdepth is a GNU find extension, as far as I know, and -printf is also not portable, and neither is sort -g. I really don't think this will work on AIX.
    – terdon
    Jul 18, 2022 at 10:49
  • @terdon -maxdepth 1 can be omitted if directory contains files only. I have used it just in case.
    – MiniMax
    Jul 18, 2022 at 11:12
  • @terdon AIX can't have GNU utilities?
    – MiniMax
    Jul 18, 2022 at 11:13
  • 1
    It can, but not by default, so please mention that and either explain how to install them, or at least that they are required for this to run. Also, your sed won't remove just files ending in .gz, but any file ending in gz, you might want to change that to sed '/\.gz$/d' to avoid removing files like thisfileisnotgz.
    – terdon
    Jul 18, 2022 at 11:27
0

Assuming that you like to use the find command this would probably solve your problem:

find . -type f ! -name '*\.gz' -print | awk '{Q[N++]=$0; N=N%3; if (Q[N]!="") {print Q[N]}}

The awk command fills an n-th length array Q

  • Q[N++]

advances through the array modul 3 (longer arrays to skip more elements)

  • N=N%3

And print the entry filled 2 inserts before (therefor ignoring the last two). The if skips the first to inserts, when the array is not yet fully initialized

123   123   123   123   123  ...
^^     ^^   ^ ^   ^^     ^^
||     ||   | |   ||     ||
io     io   O i   iO     iO
nu     nu   U n   nU     nU
 t      t   T      T      T

If you prefer to use 'ls' you can change the $0 to the n-th element (in this case there can't be spaces in the file name). Please check as I'm not sure if I fully understand the corner cases for your question.

0

Using the ksh93 shell (which is standard on newer AIX versions) for the array support, and assuming that the files are expanded in modification-time order (i.e., that the order of the filenames represents the same sequence as modification timestamps), you could use one array to get the entire list of files (useful for extracting the "last two" later), then use another array to find all of the uncompressed files. Gather the final list of files by looping through the uncompressed file list and removing any files that match the "last two" from earlier.

#!/usr/bin/ksh93

files=(*)
uncompressed=( !(*.gz) )

for index in "${!uncompressed[@]}"
do
  if [[ "${uncompressed[index]}" == "${files[-1]}" ]] || \
     [[ "${uncompressed[index]}" == "${files[-2]}" ]]
  then
    unset -v 'uncompressed[index]'
  fi
done

# echo gzip "${uncompressed[@]}"

At the end you have an array named "uncompressed"; you could compress those filenames as demonstrated (remove the echo).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .