5

Is it possible to 'partition' a directory listing - say in to blocks of some number, to perform different actions on each 'range'?

For example, lets say I have the following directories inside another folder:

$ ls test
abc/
def/
hij/
klm/
nop/
qrs/
tuv/
wxy/
zzz/

And I want to perform some action on the first 3 directories, another on the second 3, and so on.

My thought was that perhaps a loop across numbers would work, based on the output of something like ls | nl (but before anyone mentions it, I know parsing ls is a no-no!)

This obviously doesn't work, but illustrates my point I hope:

for dir in `ls | nl`; 
do
    do-something-with ${dir{1..3}} # Where $dir has taken on some numerical value linked to the folder)
    do-something-with ${dir{4..6}} 
# And so on...
    do-something-with ${dir{n-3..n}} 
done

The folders I intend to actually do this on, can be worked on in any order (i.e. the final splits can be totally arbitrary), but they have no logical naming consistency - by which I mean they can't be organised sensibly alphabetically or numerically based on any key within the directory name themselves.

3

There's no reason to do either of the following mistakes for this situation:

  1. Use non-portable GNU extensions (such as xargs -0)
  2. Parse the filenames as a stream of text only, pretending that they can't contain newlines.

You can handle this portably without much difficulty at all:

set -- */
while [ "$#" -gt 0 ]; do
  case "$#" in
    1)
      my-command "$1"
      shift 1
      ;;
    2)
      my-command "$1" "$2"
      shift 2
      ;;
    *)
      my-command "$1" "$2" "$3"
      shift 3
      ;;
  esac
done

Really this is more verbose than necessary, but it is readable this way.

For a different type of partitioning, where you want to split all directories into exactly three different methods of handling (but don't need to handle them at the same time), you could do it like so:

x=1
for d in *; do
  [ ! -d "$d" ] && continue
  case "$x" in
    1)
      # do stuff with "$d"
      ;;
    2)
      # do stuff with "$d"
      ;;
    3)
      # do stuff with "$d"
      ;;
  esac
  x=$(($x+1))
  [ "$x" -eq 4 ] && x=1
done
  • @don_crissti, what do you mean? The first example (using shift) does exactly what the OP described with his pseudocode. It handles dirs three at a time, and on the last "partition" handles however many are left (1 or 2 or 3). – Wildcard Apr 15 '16 at 17:16
  • @don_crissti, I thought of that, but if you're handling 5001 directories, are there really 1667 different commands you're going to want to run? His pseudocode shows the same command for each group of three: do-something-with – Wildcard Apr 15 '16 at 17:41
  • You're both sort of on the right track as far as what I was trying to achieve! In my actual usage case, I'm trying to separate 330 directories in to 11 lots of 30, but I only posted a sort of 'minimal example'. So in this case, the do-something-with will ultimately be a move command - which will be the same command for each, except that the directory each group will move to will be different, so the explicit command will change for each group. – Joe Healey Apr 17 '16 at 13:13
  • @JoeHealey, in that case my latter code example is what you want. Just extend it to have 11 different possible "cases" instead of three; it will then deal out the 330 directories amongst those 11 target dirs like dealing cards. :) – Wildcard Apr 17 '16 at 21:04
  • This was perfect Wildcard, thank you, I wouldn't have ever thought to use case switches. – Joe Healey Apr 18 '16 at 9:55
3

With printf/xargs:

printf "%s\0" */ | xargs -0 -n3 echo do-something
  • printf prints the current directory contents null delimited. The slash after the * matches only directories, not files.
  • xargs reads the input null delimited -0
    • -n3 splits the input in 3 parts.

The output (with your example directories):

do-something abc/ def/ hij/
do-something klm/ nop/ qrs/
do-something tuv/ wxy/ zzz/
  • This seems close to what I would like to do. If I then wanted my do-something to be to move groups of folders to another folder (i.e. to gather sets of n folders in to other folders), and have the folders end up named something like 1-3/ which contains abc/ def/ hij/, and 4-6/ which contains klm/ nop/ qrs/ and so on... how would I implement this with the output from xargs? – Joe Healey Apr 13 '16 at 16:49
0

The -n argument to xargs can be sometimes used for this. If you want to run jobs in parallel, the -P option can be used as well.

/tmp/t$ ls -1
dir1
dir2
dir3
dir4

/tmp/t$ ls -1 |xargs -n 3 echo runcmd
runcmd dir1 dir2 dir3
runcmd dir4

Don't worry too much about parsing the output of ls|nl. If you need to worry about that kind of thing, you can use find .. -print0 and the -0 argument to xargs. (assuming both are available).

0

If I could modify Otheus's answer little bit, it would be like this

for d in ``ls -1 | xargs -n 3 echo` ; do (cd "$d" && somecommand); done
and so on with other directories too.

  • This doesn't seem quite right from my invocation at least... It gives me a simple list of all the right directories, but with no apparent demarcation in to groupings (and there is one extra back tick before your ls ;) ) – Joe Healey Apr 13 '16 at 16:56
0

Since Wildcard's solution is the one I eventually used, I've marked his answer as accepted, but if it's useful for people stumbling across this in the future, below is my code for distributing 330 directories amongst 11 other directories evenly.

Perhaps worth pointing out that it seemed pretty damn fast too!

#!/bin/bash

numdirs=11

for ((i=1; i<=$numdirs; i++))
do
    mkdir -p ~/simulation_groups/group_${i}
done

x=1

for d in * ; do
  [ ! -d "$d" ] && continue
  case "$x" in
1)
  mv $d ~/simulation_groups/group_${x}
  ;;
2)
  mv $d ~/simulation_groups/group_${x}
  ;;
3)
  mv $d ~/simulation_groups/group_${x}
  ;;
4)
  mv $d ~/simulation_groups/group_${x}
  ;;
5)
  mv $d ~/simulation_groups/group_${x}
  ;;
6)
  mv $d ~/simulation_groups/group_${x}
  ;;
7)
  mv $d ~/simulation_groups/group_${x}
  ;;
8)
  mv $d ~/simulation_groups/group_${x}
  ;;
9)
  mv $d ~/simulation_groups/group_${x}
  ;;
10)
  mv $d ~/simulation_groups/group_${x}
  ;;
11)
  mv $d ~/simulation_groups/group_${x}
  ;;
  esac
  x=$(($x+1))
  [ "$x" -eq $numdirs+1 ] && x=1
done
  • For future reference, your first for loop could be handled by bash brace expansion: mkdir -p ~/simulation_groups/group_{1..11} Also this would be much better with "$d" double quoted in all uses. :) – Wildcard May 17 '16 at 4:24
  • Ah yes that would be much more sensible! – Joe Healey May 23 '16 at 16:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.