Context : In order to test some backup process which are supposed to work all nights, i want to select a random files on a large amount of datas. (Around 7 millions of files. This is a NFS server with ~8To of data used, mainly web applications.

The random selecting part need to be called 2 times, first with pure random selection, and then i want to pick some fresh files (something like : find /data/ -mtime 1|shuf -n 1 ) I wrote a script which made a parsing of backup configuration files, try to restore via the tool, compare original checksum with the old one and report all tests in a mail. Everything work except the "random selecting part", indeed i've some performance issues.

I've tested many ways to select random files on a large FS. here is some of my ideas :

  • Select random used inodes and getting the filename of associed inode (Performance issue, to lot of ram needed, process is very long.)
  • find /data/ -type f -mtime 1 |shuf -n 1 → (Too much files piped to shuf (time $command ~ 46seconds) )
  • RANDOM=$(shuf -i 1-7000000 -n 1) && find /data/ -type f -mtime 1 |head -n ${RANDOM} → Same performance problem when random > 1000000 (time $command ~ 49seconds))
  • Python discovery script with (os.listdir) → Good performance but i used with ctime, horrible performance issues

I'm very surprised to not find some library/tools/scripts (in Python, Bash, C or whatever) for making things like explained before. It seems not be a specific problem, i think that some admins in the world and to test randomly if backup are working correctly.

So i'm interrested of some ways to do it, with specific GNU/Linux/BSD/*nix tools, Python script/library. Hope you consider that i'm searching for "high performance" things. My script will call solution for each path backup config files.

Thanks in advance

  • The reason a tool like the one you describe is not widespread is because, in general, sysadmins want to check the integrity of their backups as a whole, not single files (why do you want to do this?). You can easily do this with (for example) duplicity, using duplicity verify. – Chris Down Nov 24 '15 at 10:51
  • Hi chris, thanks for your answer. This is a specific tools which make incremental backup of our backup in magnetic tape (already tested before). The goal is to monitore the process between backup1 and magnetic tape backup. There is no way to use duplicity even if duplicity is a great tool. :/ Otherwise i don't think that (in my context) restoring 8To of data and check all data is a good idea. This is why i want to test randomly from backup config files. I already did somes tests between backup1 and 2. The goal now is to test randomly if backup work between backup2 and magnetic tape robot. – DaapTik Nov 24 '15 at 14:11

The reason there isn't a 'standard tool' is because the logic is - as you've found - quite simple. The limiting factor is that you must do a deep directory traversal, and that's always an expensive process.

It doesn't really matter what approach you take in terms of scripting tools - the 'cost' is the disk IO.

So the optimisations I'd suggest would be:

  • Don't walk the whole FS. Bail out of your traversal when you've found enough. (find | shuf and find | head won't do this).
  • You can probably approximate directory sizing by referring to last traversals, and 'skip ahead' by some margin.
  • statting files as you go and recording mtime will help you build both lists. If you generate a random number, and select the 'last' file, and the last recent file before that number.

Something like this (In perl but I'm sure you could do it in Python)

#!/usr/bin/env perl
use strict;
use warnings;

use File::Find;

my $random_file;
my $recent_random_file;

my $limit = rand ( 7_000_000 ); #ideally set to file count on fs. 

sub search {
    if ( $count++ > $limit ) { 
        $File::Find::prune = 1; #stop traversing
    return unless -f; 
    if ( -M $File::Find::name < 1 ) { $recent_random_file = $File::Find::name }; 
    $random_file = $File::Find::name; 

find ( \&search, "/path/to/search");
print "$recent_random_file $random_file\n";
  • Thanks for your answer Sobrique. (You forget to declare $count on your example) Your piece of Perl is similar to what i did with Python. This is a good point because i'm sure to do it in the right way. Anyway if somebody have a workaround or an efficient way to do this (maybe picking radomly on used inodes and then match between inodes → filename. – DaapTik Nov 24 '15 at 14:22

I don't see any easy way to do what you are trying to accomplish. Getting a random "file" from big file system means that you must have a list of all files on that system first. Which takes a lot of time or resources. Still if you wish should stick to such kind of solution here is what i can recommend you:

  • Run multiple processes. Example: If you have your /data/ folder with 10 subfolders, instead of find /data/ ..... you can do something like:

    for i in "ls data";do find $i ....;done

    You will get 10 results 10 times faster (or something like that). Yet it will consume a lot of CPU.

  • You don't need to search the whole file system where a thousands of files remains the same for years, but only parts of the filesystem tree where you expect changes, and exclude parts of the three where caches are stored, as usually they are not intended to be backed up. This way you will make the time for building the list shorter.

  • Define maxdepth and mindepth (they could be a random file..) - you still will get a random file, but you will limit the subdirectories to be read and will safe some time and resources.

Such things could be done in order to tune up the thing a bit but as larger the file system became, harder will be to get what you need. So maybe better think of snapshot based backup solution or better way to verify that your backup is OK.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.