1

I'm writing something for backup. And I'm working with really Big files/folders (1 million++, 100TB++)

For improve my speed, at least I need to pick 10 biggest folders in tree and for that, I need to know some information in tree.

  1. What is 10-20 biggest folders in sub-tree? (disk usage)
  2. Which folders has most partial files. (Usage doesn't matter, its for 4k random speed)

I can't get the information with du -kh because its taking too long.

Now I'm thinking; creating folder tree first, after that I should calculate in my local storage without files, than I'm ready to go.

BUT! With this way, I just know folder tree. I still don't know which folder has biggest and most particulated files.

I couldn't any way to find both information and now I'm thinking "guessing method" with option 2.

What should I do?

  • How do you differentiate the partial files? Any special file extension? (e.g. .part or somesuch) – schaiba Jan 26 '17 at 7:40
  • I'm almost completely certain it may be better to pay for some sort of enterprise commercial backup solution for those volumes of data. But my knowledge ends at the border of the free Unices, and I furthermore don't know the nature of your data or system. – Kusalananda Jan 26 '17 at 7:55
  • #schaiba: I don't have files. It's global backup solution. #Kusalananda: My backup system working Great.. I can backup Almost +/- 200mb/s! And i will improve that if i figure out this problem. :/ – Morphinz Jan 26 '17 at 8:10
  • You don't have files? There is some essential info missing from this question. Could you please mention the Unix type and type of filesystem you're using? – Kusalananda Jan 26 '17 at 8:30
  • I said it's backup solution. So file type is ANY. My system is: Solaris & Zfs. But remote file system will be different.. – Morphinz Jan 26 '17 at 8:34
0

Unless you happen to be working with some special filesystem that holds the information and presents it back to you, polling with du is basically the best you can do.

Standard filesystems like ufs/ext don't keep track of the total size of folder contents, so they must be (expensively) interrogated.

1 million+ files means you can't do things instantly, but it's still possible to run a du and keep the output for the future.

  • I'm taking the files from NFS mount. The remote filesystem will be "any". "du" looking not an option beacuse its taking too long. I can't wait for that 3-4 hour for 1 million files. There must be a way. – Morphinz Jan 26 '17 at 8:44
  • Are you only running a backup once? Capture the data and keep it. Make a guess for a good split on the first run, then wait for du to tell you if you need to modify the split to improve the next run. something has to do the work, and its either the filesystem (which is rare) or you have to do it yourself (ahead of time). You could split ZFS into multiple filesystems, each of which can be polled quickly. But you have to set that up ahead of time. – BowlOfRed Jan 26 '17 at 8:49
  • Not once, its sync. but the problem is all about running first time and Capturing the data. You can't capture 100TB data quickly. "copy" running 5-10 mb/s. It's really big tree. Rsync doesn't work parallel or good enough. Normal Rsync is working 5mb/s for the directory, but i can do 200++mb/s. I just need to know equal 10 sub-tree.. If they're not equal, than some sync jobs finishs and file transfer speeds down. Thats why i need to guess equal 10 subtree. – Morphinz Jan 26 '17 at 10:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.