4

I have one big folder with many file types in it (e.g .txt, .sh). It's about 40Gb. I would like to split it into four parts of 10Gb each. How can I achieve this?

5
  • 7
    Any try so far? Dec 20 '18 at 13:33
  • I don't suppose you have any individual files larger than 10 Gb?
    – Jeff Schaller
    Dec 20 '18 at 13:37
  • take a look at split command maybe ? Dec 20 '18 at 13:51
  • 1
    Do you need to access the files, or is it for archiving? One could use multi volume archive format.
    – Kusalananda
    Dec 20 '18 at 14:11
  • 2
    Why is this closed? It's perfectly clear what they're asking.
    – Mzzl
    Mar 18 '19 at 17:28
5

Without trying to solve the bin packing problem, you could use a script like this:

#!/bin/bash                                                                     
directory=${1:-testdir}                                                         
sizelimit=${2:-1000} # in MB                                                    
sizesofar=0                                                                     
dircount=1                                                                      
du -s --block-size=1M "$directory"/* | while read -r size file                  
do                                                                              
  if ((sizesofar + size > sizelimit))                                           
  then                                                                          
    (( dircount++ ))                                                            
    sizesofar=0                                                                 
  fi                                                                            
  (( sizesofar += size ))                                                       
  mkdir -p -- "$directory/sub_$dircount"                                           
  mv -- "$file" "$directory/sub_$dircount"                                           
done 
2
  • 1
    If the files are small it might be better to use kb or byte resolution but besides that +1 for the script and mentioning the optimization problem.
    – eckes
    Dec 20 '18 at 21:42
  • 1
    Awesome script - well done, perfect for what I needed ;-)
    – Matt B
    Aug 3 '19 at 13:06
0

If it is a single 10GB file, you can try

split -b 4000000000 filename
3
  • 5
    Note, the user does not want to split the files themselves, but to distribute them evenly between subfolders.
    – Kusalananda
    Dec 20 '18 at 14:03
  • 3
    ... I think....
    – Kusalananda
    Dec 20 '18 at 14:11
  • 2
    Not sure, myself, but the title does go in the direction of directories: "split a large folder into smaller folders of equal size".
    – Jeff Schaller
    Dec 20 '18 at 14:21

Not the answer you're looking for? Browse other questions tagged or ask your own question.