I have one big folder with many file types in it (e.g .txt
, .sh
). It's about 40Gb. I would like to split it into four parts of 10Gb each. How can I achieve this?
-
7Any try so far?– Kevin LemaireDec 20, 2018 at 13:33
-
I don't suppose you have any individual files larger than 10 Gb?– Jeff Schaller ♦Dec 20, 2018 at 13:37
-
take a look at split command maybe ?– Pierre-Alain TORETDec 20, 2018 at 13:51
-
1Do you need to access the files, or is it for archiving? One could use multi volume archive format.– Kusalananda ♦Dec 20, 2018 at 14:11
-
3Why is this closed? It's perfectly clear what they're asking.– MzzlMar 18, 2019 at 17:28
Add a comment
|
2 Answers
Without trying to solve the bin packing problem, you could use a script like this:
#!/bin/bash
directory=${1:-testdir}
sizelimit=${2:-1000} # in MB
sizesofar=0
dircount=1
du -s --block-size=1M "$directory"/* | while read -r size file
do
if ((sizesofar + size > sizelimit))
then
(( dircount++ ))
sizesofar=0
fi
(( sizesofar += size ))
mkdir -p -- "$directory/sub_$dircount"
mv -- "$file" "$directory/sub_$dircount"
done
-
1If the files are small it might be better to use kb or byte resolution but besides that +1 for the script and mentioning the optimization problem.– eckesDec 20, 2018 at 21:42
-
1
If it is a single 10GB file, you can try
split -b 4000000000 filename
-
5Note, the user does not want to split the files themselves, but to distribute them evenly between subfolders.– Kusalananda ♦Dec 20, 2018 at 14:03
-
3
-
2Not sure, myself, but the title does go in the direction of directories: "split a large folder into smaller folders of equal size".– Jeff Schaller ♦Dec 20, 2018 at 14:21