Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I've searched for this function, and there are lots of answers, but I haven't been able to find anything that works for me.

I have a folder with about 30,000 files, and I want a terminal command or bash script that will move these files into different folders. About 3000 files for each folder. Some of the filenames have underscores, spaces and hyphens, so the command should allow me to move those files as well as the files without underscores, spaces, and hyphens. The files are .jpgs, but I'd like a command that I can customize for any file type and file amount, in case I need to use it in the future for different formats and quantities.

UPDATE:

I found a bash script that answers my needs. It will move files in a folder into subfolders. It works with any type of file. The number "3001" represents the number of files you want moved to each subfolder. This number can be changed. So if you have a folder with 20000 files and you want the 20000 files divided into batches of 500 and moved to subfolders then you would replace "3001" with "500". You can also modify the script to only move particular file types, e.g.: to move only .jpg files, change "for file in *" to "for jpg_file in *.jpg". Also change "$file" to "$jpg_file".

#!/bin/bash

c=1; d=1; mkdir -p dir_${d}

for file in *
do
        if [ $c -eq 3001 ]
        then
                d=$(( d + 1 )); c=0; mkdir -p dir_${d}
        fi
        mv "$file" dir_${d}/
        c=$(( c + 1 ))
done
share|improve this question

closed as unclear what you're asking by Bernhard, garethTheRed, Michael Homer, Braiam, jasonwryan Aug 13 at 23:55

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question.If this question can be reworded to fit the rules in the help center, please edit the question.

4  
Are there any criteria you'd want to use to determine in which folder the files should go (e.g. alphabetical, year from the EXIF data), or simply 10 folders of 3000 files each? –  ph0t0nix Aug 13 at 17:57
2  
I don't get what you really want. What distincts one file from another. Why do you need this? –  Bernhard Aug 13 at 18:07
    
@ph0t0nix: Alphabetical would be best, because not all the files I will want to move have the necessary metadata. –  user8547 Aug 13 at 18:11
    
@Bernhard: I don't like manually going through a folder with 30000 files and sorting them into folders because it's tedious an can be slow due to system limitations. –  user8547 Aug 13 at 18:17
4  
@user8547 It is unclear to me based on what you want to divide them. IF it does not matter at all, but you just want them to be 3000 per subdirectory at random, mention that at least. I also don't see the purpose, but that is a different topic. –  Bernhard Aug 13 at 19:44

5 Answers 5

How about the following:

for letter in {A..Z}; do
   dir=/path/to/sorted/directories/$letter
   mkdir $dir
   mv "${letter}*.jpg" $dir/
done

And then of course the same loop with the lower case letters {a..z}, except that in that case you don't want to create a separate lower case directory. Then the move line changes to:

mv "${letter}*.jpg" ${dir^^}/

Using ${var^^} to create an upper case version of a variable requires Bash 4.0.

Edit: fixed a missing quote. Thanks G-man for pointing that out.

share|improve this answer
    
Could you modify the script to move any files, regardless of having numerical or alphabetical filenames, uppercase or lowercase, and regardless of whether there are spaces, underscores, and hyphens in the filenames? –  user8547 Aug 13 at 19:10
    
You seem to have a problem with quotes. In the first code block, you're quoting the *, which strips it of its meaning as a wildcard. In the second code example, you have only one (unbalanced) quote. –  G-Man Aug 13 at 22:21
    
@user8547: ph0t0nix's answer can trivially be extended to create a directory for each digit. And then you could simply do mkdir /path/to/sorted/directories/special; mv *.jpg /path/to/sorted/directories/special after you've moved all the files whose names begin with a letter (or digit). –  G-Man Aug 13 at 22:25

I think something like:

#!/bin/bash
shopt -s nullglob
for i in <directory>; do
    mv *.jpg "$dir"
done

This will move all jpegs to $dir which you will need to set or you will need to make possibly create an array with the directory name(s). You can easily customize this code by switching *.jpg to any different format. You will need to tweak around with this snippet to customize it to how you want it.

UPDATE For this to be a function try this out:

#!/bin/bash
echo "Enter Directory: ";
read dir
mkdir $dir

moveFiles(){
    arg1=$1    
    counter=0;
    while[$counter -ne 3000]; do
          mv <source> *.jpg "$dir"
          counter++;
          if[$counter -e 3000];
          then
               exit;
          fi
     done
}   
moveFiles "$dir"
exit

Again, this code may need tweaking. This is just an example.

share|improve this answer
    
Tx! Can you add a function to the command to automatically create folders? For example move 3000 files into folder001, 3000 files into folder002, etc. –  user8547 Aug 13 at 18:14
    
@ryekayo isn't for loop redundant if there is only one dir and mv * is used? –  Simply_Me Aug 13 at 18:54
    
Yes you are right, let me correct this. –  ryekayo Aug 13 at 18:57
    
@ryekayo: I just tried the updated script and it moved over 9000 files into a folder, which is too much. I'd like the script to automatically move 3000 files into separate directories, and create the directories on the fly. The name of the directories isn't important. As far as sorting, I'd like them to be moved in numerical-alphabetical order. –  user8547 Aug 13 at 19:08
    
@user8547: this snippet is serving as an example for moving files to a given directory. In the moveFile function, you will need to create a counter that will count files to 3000 and then break out of the function. I will look up what can go in there and update it shortly. –  ryekayo Aug 13 at 19:11

This will list files with ls *.jpg, take first 3000 with head -n 3000, make directory for them with another name defined in $FOLDERLIST and move files into it, this loop repeats 10 times

LISTFILESCMD='ls *.jpg' 
FQUANTITY=3000
FOLDERLIST=`seq -w 1 10`

for FOLDER in $FOLDERLIST; do mkdir $FOLDER; mv `$LISTFILESCMD | head -n $FQUANTITY` $FOLDER; done

u may alter parameters, read man ls for sorting options. this is just an example and not optimal solution (calling ls every time, give so many params to mv is not a good idea, it's better to use xargs instead if total length is too large) but I think it will help you to understand what do you really need.


test section

 $ ls|wc -w
41978
 $ LISTFILESCMD='ls *.jpg'
 $ FQUANTITY=3000
 $ FOLDERLIST=`seq -w 1 13`
 $ time(for FOLDER in $FOLDERLIST; do mkdir $FOLDER; mv `$LISTFILESCMD | head -n  $FQUANTITY` $FOLDER; done)
real    0m7.396s
user    0m4.543s
sys     0m2.513s
 $ ls|wc -w 
2991
 $ ls -d */
01/  02/  03/  04/  05/  06/  07/  08/  09/  10/  11/  12/  13/
 $ for fldr in `ls -d */`;do ls $fldr|wc -w;done
3000
3000
...
share|improve this answer
    
Tx! But it only partially works. It created 10 folders, but only moved files into some of the folders. And the files that were moved into some of the folders didn't amount to 3000, but varying quantities. The filenames are not uniform. Some have spaces, letters, numbers, underscores. I wonder if that's why it's not working. –  user8547 Aug 13 at 20:36
    
@Leben ls will as used above will not capture file names with special characters as described by the author. –  Simply_Me Aug 13 at 20:59
    
oh, it's probably because of spaces. –  Leben Gleben Aug 13 at 20:59

I tried all the scripts but none of them worked for some reason, or only partially worked. I found this bash script and tweaked it to look for .jpg files instead of .xml files:

#!/bin/bash

c=1; d=1; mkdir -p dir_${d}

for jpg_file in *.jpg
do
        if [ $c -eq 501 ]
        then
                d=$(( d + 1 )); c=0; mkdir -p dir_${d}
        fi
        mv "$jpg_file" dir_${d}/
        c=$(( c + 1 ))
done

It worked well and fast. Now my only problem is getting it to move any file into folders, not just ".jpg". It's also case sensitive and will not move ".JPG".Any ideas on how to tweak it to move any file, or at the very least any image file?

share|improve this answer
2  
In order to move any file (regardless of extension), simply change the for loop: for file in * (and then also rename any occurence of $jpg_file with $file. –  ph0t0nix Aug 13 at 22:17
    
@ph0t0nix: Tx! That worked like a charm! –  user8547 Aug 13 at 22:45
    
@user8547 make sure it deals with files that have space in them. –  Simply_Me Aug 13 at 22:47
    
@Simply_Me: What do you mean? The file names should have spaces in them? I tested it out with files that have no spaces, but do have hyphens and underscores, and it worked well. –  user8547 Aug 13 at 22:51
    
@user8547 yes, I just checked it, it works. –  Simply_Me Aug 13 at 23:01

Assuming you have one directory which you want to move files out of.

You can use:

$ mv <source folder/*.jpg <destination folder/>

Edit:

The script below finds all of the files that match the searching criteria defined by file_screen, then executes a while loop to go move the selected files.

New directories are created based on modulo criteria: loop iteration mod files_in_each. If mod returns zero, new directory is created and files will be moved to it.

Parameters to tweak in your runs:

files_in_each=3000  # controls how many files are placed in each directory
directory_to_move="/home/shadowe/test1/test2" # where are the files located
file_screen="jpg"   # only move files that match this criteria

Please tweak as needed.

#!/bin/bash

# basic definitions and calculations
files_in_each=3000
directory_to_move="/home/shadowe/test1/test2"
file_screen="jpg"
folders_created=0
i=0

# while loop through all of the files that match screening criteria
find $directory_to_move/* -maxdepth 1 -type f -name "*${file_screen}" -print0 | sort -n | while IFS= read -r -d '' file; 
do
    # modulo control for creating directories every files in each completion
    create_dir=`expr $i % $files_in_each`
    if [ $create_dir -eq "0" ]
    then
            new_folder=folder$folders_created
            mkdir $new_folder
            echo "created new folder: " $new_folder
            folders_created=$[$folders_created+1]
    fi
    mv "$file" $new_folder
    i=$[$i+1]
done

Small sample results after running:

$ ls test2/
not a picture.txt
$ ls folder0/
one.jpg      one*two.jpg       picture 1.jpg  two-one.jpg
one-two.jpg  picture 1111.jpg  picture *.jpg  two three.jpg

Large sample results:

$ ls folder0 | wc -l
3000
$ ls folder1 | wc -l
2008
$ ls test2 | wc -l
7501
$ ls test2/ | grep "jpg"
$

Script to generate test files:

#!/bin/bash

mkdir test2
touch test2/one.jpg
touch test2/'one-two.jpg'
touch test2/'one*two.jpg'
touch test2/'two-one.jpg'
touch test2/'two three.jpg'
touch test2/'picture 1.jpg'
touch test2/'picture *.jpg'
touch test2/'picture 1111.jpg'
touch test2/'not a picture.txt'
#for large test sample uncomment below
#for i in `seq 1 7500`; do touch test2/test$i.txt; done
#for j in `seq 1 5000`; do touch test2/picture$j.jpg; done
share|improve this answer
    
Tx! Can you add a function to the command to automatically create folders? For example move 3000 files into folder001, 3000 files into folder002, etc –  user8547 Aug 13 at 18:14
    
@user8547 check out the updated answer. –  Simply_Me Aug 13 at 20:15
    
Tx! But it doesn't work right. It created a bunch of folders, but only moved files into some of them. I changed "find test2/*" to "*jpg". Before I changed that it didn't work at all. –  user8547 Aug 13 at 20:51
1  
@user8547 no, your change is wrong. test2 should be changed to the directory name where you store the files. –  Simply_Me Aug 13 at 20:52
    
@user8547 I've modified the script to find only certain type of files (i.e. "jpg" etc.). Please review updated answer and examples that it works. –  Simply_Me Aug 13 at 22:38

Not the answer you're looking for? Browse other questions tagged or ask your own question.