0

What I am trying to accomplish is I have a directory with lets say 176,000 files. I want the script to split the 176,000 files into folders with lets say 20,000 files each and the last one containing the odd ball #. I would like to copy the files sequentially and keeping them in order alpha numerical. This is what I currently have but it seems to be missing something, the first 2 folders it creates are out of order, the other 3 folders it creates are sequential.

Bottom line, folder contains 176,000 files, script would move the first 20,000 into a folder called 'split' then takes the next 20,000 and so on, creating split2, split3, etc..

Here is what I have:

#!/bin/bash

dir_size=20000
dir_name="split"
n=$((`find . -maxdepth 1 -type f | wc -l`/$dir_size+1))
for i in `seq 1 $n`;
do
    mkdir -p "$dir_name$i";
    find  .  -maxdepth 1 -type f | sort -n -z |  head -n $dir_size |  xargs -i mv "{}" "$dir_name$i"
done
0

1 Answer 1

0
  • Make a list of your files (note that you cannot use ls * or ls*.dpx because this would create a command line with 176000 parameters to ls) and use split to split this list in chunks (files named files.01, files.02...)(the lone - is telling split to work on standard input).
ls | split -d --lines 20000 - files.
  • For more stringent filtering, you can also use find as you did, piping output to split after a sort:
find . type f | sort | split -d --lines 20000 - files.
  • Iterate the produced files.*, create the directory name using the numeric part, and then move the files:
for f in files.*  # iterate the files.*
do 
    dirname="/path/to/dir${f#*.}"  # generate directory name from numeric suffix
    mkdir -p "$dirname" # create the directory
    xargs mv -t "$dirname" < $f # move the files
done

Warning: untested, try on a small sample first.

If you have spaces in the file names use find -print0, sort -z, split --separator '\0' and xargs -0.

6
  • So what would the full syntax be? I'm fairly new with bash I just dont want to break the whole script :( I know I'm close
    – CJ Aslan
    Feb 25, 2020 at 18:03
  • See augmented answer
    – xenoid
    Feb 25, 2020 at 19:21
  • thanks man! in this line, how can I specify it to make the folders in the current working directory vs a specified path? dirname="/path/to/dir${f#*.}" # generate directory name from numeric suffix
    – CJ Aslan
    Feb 26, 2020 at 17:56
  • Basically I want it to create the 'split' folders in the working directory
    – CJ Aslan
    Feb 26, 2020 at 18:14
  • ` dirname="dirprefix${f#*.}"` where "dirprefix" is anything you like.
    – xenoid
    Feb 26, 2020 at 18:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.