2

I have a ton of scripts which do this:

command | while read something; do
    a
    long
    list
    of
    commands
done

Has anyone ever thought of how to run all the commands fed through the pipe using parallel? There are ad-hoc solutions to this issue but I'm looking for something general which requires only minimal changes to my scripts and, if possible, also allows to run even when parallel is not installed.

command above can be pretty much everything, not limited to a single command. a long list of commands can be absolutely different as well.

For instance, consider this one liner which changes the modification dates of checked-out files in a git repository to the date they were last changed:

git ls-tree -r --name-only HEAD | 
while read filename; do
   unixtime=$(git log -1 --format="%at" -- "${filename}");
   touchtime=$(date -d @$unixtime +'%Y%m%d%H%M.%S');
   touch -t ${touchtime} "${filename}";
done

By default it's horribly slow because git log and touch are both quite slow commands. But it's just one example and a simple one at that.

6
  • But what would you want to parallelize? You can't run the commands in parallel, one depends on the output of the other. Do you mean parallelize on the filename? So that you run the set of commands in parallel on different files? – terdon Jun 15 at 13:43
  • Yes, I want to to parallelize on | (pipe) lines in case of e.g. while read filename. In a perfect world it would be great to have something which works on input separated by NULLs as well. – Artem S. Tashkinov Jun 15 at 14:00
  • So you mean parallel script.sh ::: < <(git ls-tree -r --name-only HEAD)? That would launch script.sh with each of the results of the git-ls command. – terdon Jun 15 at 14:03
  • Yeah, exactly, only this solution will only work if parallel is installed. – Artem S. Tashkinov Jun 15 at 14:04
  • Instead of using while read, you should write the scripts to loop over "$@". Then you can a) use xargs -P or parallel to parallelise them and b) use whichever situational method is appropriate to provide filenames and other args to the scripts. And use NUL as the arg/filename separator wherever possible. Note, though, that it's generally better to work smarter than to work harder - parallelisation has its uses, but so does better algorithms in the scripts. Both together are even better. – cas Jun 17 at 4:57
1

I would use a bash function and call that:

myfunc() {
   filename="$1"
   unixtime=$(git log -1 --format="%at" -- "${filename}");
   touchtime=$(date -d @$unixtime +'%Y%m%d%H%M.%S');
   touch -t ${touchtime} "${filename}";
}
export -f myfunc

git ls-tree -r --name-only HEAD | parallel myfunc

Use parallel -0 if you want to split on NUL.

If you want to run the above without installing GNU Parallel you can use:

parallel --embed > myscript.sh

and then append the above to myscript.sh.

0

It is possible to get either bash or ksh to run a list of independent commands concurrently, such that each stream begins a new command as soon as its previous task exits. The streams are kept busy except for the tail-end commands.

The basic method is to launch a number of asynchronous shells which all read from the same pipe: the pipe guarantees line-buffering and atomic reads (a file of commands can be used with cat file | but not by redirection).

Commands can be any shell one-liner (in the right syntax for the shell that owns the stream), but cannot rely on the results of previous commands, as the allocation of commands to streams is arbitrary. Complex commands are best set up as external scripts so they can be invokes as a simple command with arguments.

This is a test run of six jobs on three streams, illustrating the overlap of jobs. (I stress-tested 240 jobs in 80 streams on my Laptop also.)

Time now 23:53:47.328735254
Sleep until 00 seconds to make debug easier.
Starting 3 Streams
23:54:00.040   Shell   1 Job   1 Go    sleep 5
23:54:00.237   Shell   2 Job   2 Go    sleep 13
23:54:00.440   Shell   3 Job   3 Go    sleep 14
Started all Streams
23:54:05.048   Shell   1 Job   1   End sleep 5
23:54:05.059   Shell   1 Job   4 Go    sleep 3
23:54:08.069   Shell   1 Job   4   End sleep 3
23:54:08.080   Shell   1 Job   5 Go    sleep 13
23:54:13.245   Shell   2 Job   2   End sleep 13
23:54:13.255   Shell   2 Job   6 Go    sleep 3
23:54:14.449   Shell   3 Job   3   End sleep 14
23:54:16.264   Shell   2 Job   6   End sleep 3
23:54:21.089   Shell   1 Job   5   End sleep 13
All Streams Ended

This is the proxy script which provides the debug for those jobs.

#! /bin/bash

#.. jobProxy.
#.. arg 1: Job number.
#.. arg 2: Sleep time.
#.. idStream: Exported into the Stream's shell.

    fmt='%.12s   Shell %3d Job %3d %s sleep %s\n'
    printf "${fmt}" $( date '+%T.%N' ) "${idStream}" "${1}" "Go   " "${2}"
    sleep "${2}"
    printf "${fmt}" $( date '+%T.%N' ) "${idStream}" "${1}" "  End" "${2}"

This is the stream management script. It creates the job commands to run the proxies, and initiates the background shells.

#! /bin/bash

makeJobs () {

    typeset nJobs="${1}"

    typeset Awk='
BEGIN { srand( Seed % 10000000); fmt = "./jobProxy %s %3d\n"; }
{ printf (fmt, $1, 2 + int (14 * rand())); }
'
    seq 1 "${nJobs}" | awk -v Seed=$( date "+%N$$" ) "${Awk}"
}

runStreams () {

    typeset n nStreams="${1}"

    echo "Starting ${nStreams} Streams"
    for (( n = 1; n <= nStreams; ++n )); do
        idStream="${n}" bash -s &
        sleep 0.20
    done
    echo "Started all Streams"

    wait
    echo "All Streams Ended"
}

## Script Body Starts Here.

    date '+Time now %T.%N'
    echo 'Sleep until 00 seconds to make debug easier.'
    sleep $( date '+%S.%N' | awk '{ print 60 - $1; }' )

    makeJobs 6 | runStreams 3
2
  • same for me, many & and once every 20 lines just wait. – Archemar Jun 15 at 14:01
  • @Archemar ksh can do way better than that. Running 20 commands plus wait would wait for the longest of the 20 in each group, so most of the threads can go idle. My ksh starts 20 async jobs. If job 5 ends in ten seconds, shell 5 will immediately get job 21 from the pipe and start it. Every shell runs commands all the time, until supply of jobs runs dry. Source is long gone, but I can try to recreate it. – Paul_Pedant Jun 15 at 17:42
0

Instead of running git ls-tree and then git log, date, and touch multiple times in a bash while read loop, the following perl script takes the output of git log --name-only HEAD and stores the most recent timestamp for any file mentioned in a commit log in a hash called %files. It ignores filenames that do not exist.

It then builds a Hash of Arrays ("HoA" - see man perldsc) called %times, with the timestamps as the hash key and the values being an anonymous array containing the filenames with that timestamp. This is an optimisation so that the touch function only needs to be run once for each timestamp rather than once for each filename.

The commit id, commit message, author name, and blank lines from git log's output are ignored.

The script uses the unqqbackslash() function from String::Escape on each filename to correctly handle the way that git log prints filenames with embedded tabs, newlines, double-quotes, etc (i.e. as double-quoted strings with backslash escaped codes/characters).

I expect it should run dozens of times faster, at least, than your bash loop.

#!/usr/bin/perl

use strict;
use Date::Parse;
use File::Touch;
use String::Escape qw(unqqbackslash);

my %files = ();
my %times = ();
my $t;

while (<>) {
  chomp;
  next if (m/^$|^\s+|^Author: |^commit /);

  if (s/^Date:\s+//) {
    $t = str2time($_);

  } else {
    my $f = unqqbackslash($_);
    next unless -e $f;   # don't create file if it doesn't exist

    if (!defined($files{$f}) || $files{$f} < $t) {
      $files{$f} = $t;
    }

  };
};

# build %files HoA with timestamps containing the
# files modified at that time.
foreach my $f (sort keys %files) {
  push @{ $times{$files{$f}} }, $f;
}

# now touch the files
foreach my $t (keys %times) {
  my $tch = File::Touch->new(mtime_only => 1, time => $t);
  $tch->touch(@{ $times{$t} });
};

The script uses the Date::Parse, File::Touch, and String::Escape perl modules.

On Debian, apt install libtimedate-perl libfile-touch-perl libstring-escape-perl. Other distros probably have them packaged too. Otherwise, install them with cpan.

Example usage, in a git repo with a couple of junk files (file, and file2):

$ git log --date=format:'%Y-%m-%d %H:%M:%S' --pretty='%H  %ad %s' file*
d10c313abb71876cfa8ad420b10f166543ba1402  2021-06-16 14:49:24 updated file2
61799d2c956db37bf56b228da28038841c5cd07d  2021-06-16 13:38:58 added file1
                                                              & file2

$ touch file*
$ ls -l file*
-rw-r--r-- 1 cas cas  5 Jun 16 19:23 file1
-rw-r--r-- 1 cas cas 29 Jun 16 19:23 file2

$ git  log  --name-only HEAD file*  | ./process-git-log.pl 
$ ls -l file*
-rw-r--r-- 1 cas cas  5 Jun 16 13:38 file1
-rw-r--r-- 1 cas cas 29 Jun 16 14:49 file2

(very slightly faked - I've edited the commit messages to make it clear when both files were first committed, then file2 was changed and commited again. Other than that, it's copy-pasted directly from my terminal).


This is my second attempt: I originally tried using the Git::Raw module but couldn't figure out how to get the to give me a list of only the filenames modified in a particular commit. I'm sure there's a way, but I've given up on that. I just don't know the internals of git well enough.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.