2

I have one big folder with many file types in it (e.g .txt, .sh). It's about 40Gb. I would like to split it into four parts of 10Gb each. How can I achieve this?

closed as unclear what you're asking by Jeff Schaller, Mr Shunz, RalfFriedl, roaima, GAD3R Dec 21 '18 at 12:20

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 7
    Any try so far? – Kevin Lemaire Dec 20 '18 at 13:33
  • I don't suppose you have any individual files larger than 10 Gb? – Jeff Schaller Dec 20 '18 at 13:37
  • take a look at split command maybe ? – Pierre-Alain TORET Dec 20 '18 at 13:51
  • 1
    Do you need to access the files, or is it for archiving? One could use multi volume archive format. – Kusalananda Dec 20 '18 at 14:11
  • Why is this closed? It's perfectly clear what they're asking. – Mzzl Mar 18 at 17:28
2

Without trying to solve the bin packing problem, you could use a script like this:

#!/bin/bash                                                                     
directory=${1:-testdir}                                                         
sizelimit=${2:-1000} # in MB                                                    
sizesofar=0                                                                     
dircount=1                                                                      
du -s --block-size=1M "$directory"/* | while read -r size file                  
do                                                                              
  if ((sizesofar + size > sizelimit))                                           
  then                                                                          
    (( dircount++ ))                                                            
    sizesofar=0                                                                 
  fi                                                                            
  (( sizesofar += size ))                                                       
  mkdir -p -- "$directory/sub_$dircount"                                           
  mv -- "$file" "$directory/sub_$dircount"                                           
done 
  • If the files are small it might be better to use kb or byte resolution but besides that +1 for the script and mentioning the optimization problem. – eckes Dec 20 '18 at 21:42
0

If it is a single 10GB file, you can try

split -b 4000000000 filename
  • 4
    Note, the user does not want to split the files themselves, but to distribute them evenly between subfolders. – Kusalananda Dec 20 '18 at 14:03
  • 3
    ... I think.... – Kusalananda Dec 20 '18 at 14:11
  • 2
    Not sure, myself, but the title does go in the direction of directories: "split a large folder into smaller folders of equal size". – Jeff Schaller Dec 20 '18 at 14:21

Not the answer you're looking for? Browse other questions tagged or ask your own question.