0

Is there an elegant and fast way to copy a certain directory structure and only select a random amount of files to be copied with it. So for example you have the structure:

--MainDir
  --SubDir1
    --SubSubDir1
      --file1
      --file2
      --...
      --fileN
    --...
    --SubSubDirN
      --file1
      --file2
      --...
      --fileN
  --...

I want to copy the entire folder structure but choose only a specific number of random files from {files1-filesN} of each SubSubDir to be copied along.

1

Since you tagged this as linux I'll assume GNU utilities.

Copy directory structure from $src to $dest:

find "$src" -type d -print0 | cpio -padmv0 "$dest"

Also copy a random sample of $nfile files from each leaf subdirectory of $src:

find "$src" -type d -links 2 -exec \
    sh -c 'find "$1" -type f -print0 | shuf -z -n "$2"' sh {} "$nfiles" \; | \
    cpio -padmv0  "$dest"

Here the first find finds leaf subdirectories (-links 2), then the second find finds files in each of these subdirectories. shuf chooses a random sample of files, and finally cpio copies them.

0

First find all directories:

find MainDir -type d

Then parse these directories to a script

find MainDir -type d -exec ./randomCopy.sh 2 {} \;

, which

  1. Creates the target directory
  2. Copy's an random amount of files.

In this case 2 random files are copied.

In my example the script randomCopy.sh looks like this:

#!/bin/bash                                                                                                                                                                                                                                                                    
cnt="$1"                                                                                                                                                                                                                                                                     
dir="$2"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
mkdir -p "TARGET/$dir"                                                                                                                                                                                                                                                       

# see: https://stackoverflow.com/questions/414164/how-can-i-select-random-files-from-a-directory-in-bash                                                                                                                                                                     
find "$dir" -maxdepth 1 -type f | sort -R | tail -n $cnt | while read file; do                                                                                                                                                                                               
  # copy the file                                                                                                                                                                                                                                                            
  cp "$file" "TARGET/$dir/"
done

And don't forget to make the script executable: chmod +x randomCopy.sh.

Replace the string TARGET with your target directory or use a third script-option.

This proof of concept is running inside my test directory, but there may be a lot to improve.

  • Please note, the script randomCopy.sh uses a lot of subshells and may scale very bad. Maybe someone does know how to avoid them. – ChristophS May 31 '17 at 11:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.