I'm absolute new to bash scripting but I have to work with it.

I want to loop through a directory and upload them all to a remote server. Then I have to delete them.

Now the problem I have is following: tcpdump is writing pcap data into this dir (with rotating files), so I don't want to upload the latest file. How can I make sure that when iterating through the dir, that I'll 100-percent skip the latest one? Since the processes for creating the files and uploading them are running parallel, I in the moment I start the upload, there may be a "new" latest file. It wouldn't matter if I would skip uploading of the two newest files, but I don't want to upload the now-latest file.

I tried to learn bash scirpting by reading the solution for "normal" looping, but it's very confusing syntax so I'd be glad if you could help me with a starting point or point me to partial solutions which are understandable for beginners.

Thank you!

PS: The latest file has also the latest timestamp.

  • 2
    Is the "latest file" identified by the most recent timestamp or by a particular file name?
    – Jeff Schaller
    Nov 8 '19 at 12:51
  • it has the most recent timestamp, because only tcpdump is writing the data in there.
    – Standard
    Nov 8 '19 at 12:54

Using the zsh shell and assuming your upload command is called upload and takes all files as a list on its command line:

upload ./*(.Dom[2,-1])

The glob qualifier (.Dom[2,-1])) makes the preceding globbing pattern (*) match only regular files (using the . glob qualifier, and including hidden such files, using the D glob qualifier), and the expansion of the pattern would not contain the most recently modified file (om instructs the shell to sort the files by modification timestamp, and [2,-1] would pick out all but the first in the resulting list).

You could then use the same pattern to delete the transferred files, or you could use a loop:

for name in ./*(.Dom[2,-1]); do
    upload $name
    rm -f $name

This should handle all types of filenames, including filenames with embedded spaces, tabs or newlines.

  • I get syntax error, unexpected ( in for name in ./*(.Dom[2,-1]); do'
    – Standard
    Nov 8 '19 at 13:44
  • @wernersbacher Probably because you aren't using zsh which I mentioned as a prerequisite in my first sentence. The bash shell can't use these glob qualifiers. If you're writing a script, it would be easy to make the script executable by zsh via the #!-line.
    – Kusalananda
    Nov 8 '19 at 13:46
  • whoops, I see. Will look into it
    – Standard
    Nov 8 '19 at 13:49

You can use the -t option of ls to sort the files by modification. In combination with the tail command you can print all files except the first one. Something like this:

ls -t | tail -n +2

In a loop it could look like this:

for my_file in $(ls -t | tail -n +2); do upload $my_file; rm $my_file; done
  • That looks quite promising. Is the return value a multiine string? if so, how do I loop through every line/file to process them?
    – Standard
    Nov 8 '19 at 13:06
  • I updated my Comment.
    – roohan
    Nov 8 '19 at 13:08
  • 3
    Please be very very careful about processing the output of ls. It will break your code in horrible and unexpected ways if your file names contain characters that aresn't strict alphanumerics. Spaces for one.
    – roaima
    Nov 8 '19 at 13:08
  • 1
    @roaima thanks for your concern. I'm only using this script in a directory where I know which file names are there. But for other cases, how would you solve the problem without ls?
    – Standard
    Nov 8 '19 at 13:10
  • 1
    The non-quoting of the $my_file expansion would furthermore use the value as a globbing pattern.
    – Kusalananda
    Nov 8 '19 at 13:36
# find the latest file

latestf="$(ls -t /directory/*|head -1)";

# loop over files now

for f in /directory/* ; do 
    if [[ "${f}" == "${latestf}" ]];then
        echo "skipping file [${f}]";
        echo "processing file [${f}]";
        # your upload cmd here

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.