1

I want to parallelize spliting of many directories into subdirectories using parallel or using another tool or method.

E.g. I have 1 000 000 directories with content, but it's too much for one directory, so I want to create 10 dirs in the main dir and move in each of them 100 000 original dirs. I also want to use sorting by date. I already asked similar question here, but this isn't duplicate, because I tried new commands, got new results and now I reformulated the question.

So, I already tried this

ls -tr|parallel -n100000 mkdir "dir_{#}"\;mv {} "dir_{#}" 

and this

ls -tr | parallel -j10 -n100000 --no-notice -k 'mkdir -p dir_{#}; mv {} -t dir_{#}' 

commands, but it moves only ~10 000 into one subdir(sometimes ~6200, sometimes ~12 500) and create too much subdirs - sometimes in 10 times more than I need.

I also tried to use this:

ls -dtr * | parallel -j10 -n100000 --no-notice -k 'mkdir -p dir_{#}; mv {} -t dir_{#}'

but it gave bash: /bin/ls: Argument list too long.

Of course, I don't need exactly 100 000 dirs in each subdir, it can be 101 000 or 98 500 dirs, it should be a number in the range of 100 000

How can I execute this task in parallel or using parallel?

  • 2
    I/O actions don't parallelize well; everyone needs access to the disk, but the disk can only do so much at once. – Ignacio Vazquez-Abrams Nov 25 '17 at 3:01
  • 1
    @IgnacioVazquez-Abrams as I wrote, I already parallelized it(or just accelerated), the only problem is that I get the wrong number of dirs and subdirs. – don-prog Nov 25 '17 at 3:06
  • Regarding parallelization and I/O: Try sorting by inode: unix.stackexchange.com/questions/124527/… – Ole Tange Nov 27 '17 at 9:19
2

This problem deals with heavy IO. I doubt that parallel is really useful in this situation.

Anyway I suggest that you consider a "traditional" approach:

 mkdir dir_{1..10}
 ls -tr | nl | \
    awk '$2 !~ /^dir_/ {i=1+int($1/100000); print $2 | "xargs mv -t dir_"i}'

where

  • ls -tr | nl sorts the directories by date and adds an auxiliar dir-number
  • $2 !~ /^dir_/ is used to skip the just-created folders.
  • i=1+int($1/100000) calculates the number of the folder based on the dir-number
  • print $2 | "xargs mv -t dir_"i moves without process proliferation

If possible compare also the respective times: time .... (and share the results with us ☺)

  • Thanks for your answer! time results: your command - 1.24 minutes, etopylight's command - 2.45 minutes for 300 000 dirs. – don-prog Nov 27 '17 at 2:56
  • Nice, didn't know that print can be piped to an external command inside awk :) – etopylight Nov 27 '17 at 6:46
  • 1
    @etopylight, thank you! Awk has some very cool details. The main reason why why awk is fast is that disk mechanical elements end up making it "serial". Your solution (that I up-voted) is very useful for the (and faster) for the not-so-IO-intensitive situations. – JJoao Nov 27 '17 at 8:29
  • @don-prog, thank you the comment and for the time results. – JJoao Nov 27 '17 at 14:10
3

The problem is that there is an upper limit of bytes that a command line can take after shell expansion, this limit depends on the system's limit

getconf ARG_MAX

thus causing the number of argument by mv {} to vary depending on the input filename's length whenever you hit the max limit.

A solution to avoid this limit while not giving up using parallel is to split the task into two stages

ls -tr | parallel -N 100000 --pipe -k "mkdir dir_{#}; parallel -X mv -t dir_{#}"

Explaination

  • The first stage uses the option --pipe to split the stdin into a determined number of smaller stdins, each containing n lines as specified by the option -N. You can observe the effect by using this example

    seq 1000000 | parallel -N 100000 --pipe wc -l
    

    which gives an exact split at the 100000 mark

    100000
    100000
    100000
    ...
    
  • On the second stage, the inner parallels take the smaller stdins as their new stdin to execute their jobs, the -X option inserts as many arguments as the command line length permits

    mkdir dir_{#}; parallel -X mv -t dir_{#}
    
  • 1
    Thanks for your answer! time results: JJoao's command - 1.24 minutes, your command - 2.45 minutes for 300 000 dirs. – don-prog Nov 27 '17 at 2:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.