1

I have a directory with many subfolders that look like the following, (keep in mind that FA and T1 content is paired and also ..._L, ..._R content). The quantity of folders named here by letters (a,b,c, etc) can vary a lot:

.
├── FA
│   ├── CN
│   │   ├── CN_L
│   │   │   ├── GE_FSPGR
│   │   │   │   ├── a
│   │   │   │   └── b
│   │   │   ├── PHILIPS_MPRAGE
│   │   │   │   ├── c
│   │   │   │   └── d
│   │   │   └── SIEMENS_MPRAGE
│   │   │       ├── e
│   │   │       ├── f
│   │   │       ├── g
│   │   │       └── h
│   │   └── CN_R
│   │       ├── GE_FSPGR
│   │       │   ├── a
│   │       │   └── b
│   │       ├── PHILIPS_MPRAGE
│   │       │   ├── c
│   │       │   └── d
│   │       └── SIEMENS_MPRAGE
│   │           ├── e
│   │           ├── f
│   │           ├── g
│   │           └── h
│   └── Dementia
│       ├── Dementia_L
│       │   ├── GE_FSPGR
│       │   │   ├── i
│       │   │   └── j
│       │   ├── PHILIPS_MPRAGE
│       │   │   ├── k
│       │   │   └── l
│       │   └── SIEMENS_MPRAGE
│       │       ├── m
│       │       ├── n
│       │       ├── o
│       │       └── p
│       └── Dementia_R
│           ├── GE_FSPGR
│           │   ├── i
│           │   └── j
│           ├── PHILIPS_MPRAGE
│           │   ├── k
│           │   └── l
│           └── SIEMENS_MPRAGE
│               ├── m
│               ├── n
│               ├── o
│               └── p
└── T1
    ├── CN
    │   ├── CN_L
    │   │   ├── GE_FSPGR
    │   │   │   ├── a
    │   │   │   └── b
    │   │   ├── PHILIPS_MPRAGE
    │   │   │   ├── c
    │   │   │   └── d
    │   │   └── SIEMENS_MPRAGE
    │   │       ├── e
    │   │       ├── f
    │   │       ├── g
    │   │       └── h
    │   └── CN_R
    │       ├── GE_FSPGR
    │       │   ├── a
    │       │   └── b
    │       ├── PHILIPS_MPRAGE
    │       │   ├── c
    │       │   └── d
    │       └── SIEMENS_MPRAGE
    │           ├── e
    │           ├── f
    │           ├── g
    │           └── h
    └── Dementia
        ├── Dementia_L
        │   ├── GE_FSPGR
        │   │   ├── i
        │   │   └── j
        │   ├── PHILIPS_MPRAGE
        │   │   ├── k
        │   │   └── l
        │   └── SIEMENS_MPRAGE
        │       ├── m
        │       ├── n
        │       ├── o
        │       └── p
        └── Dementia_R
            ├── GE_FSPGR
            │   ├── i
            │   └── j
            ├── PHILIPS_MPRAGE
            │   ├── k
            │   └── l
            └── SIEMENS_MPRAGE
                ├── m
                ├── n
                ├── o
                └── p

I want to split these directories into test, train and validation based on percentage, that means taking say 60%, 20% and 20% of the children folders (a,b,c,d,etc... are also folders) and copy them to their corresponding set (train, test or validation) while keeping the same folder parents and same paired structure. I don't know if I made myself clear.

The output will be something like this:

.
├── test
│   ├── FA
│   │   ├── CN
│   │   │   ├── CN_L
│   │   │   │   ├── GE_FSPGR
│   │   │   │   │   ├── a
│   │   │   │   ├── PHILIPS_MPRAGE
│   │   │   │   │   ├── c
│   │   │   │   └── SIEMENS_MPRAGE
│   │   │   │       ├── e
│   │   │   │       ├── f
│   │   │   └── CN_R
│   │   │       ├── GE_FSPGR
│   │   │       │   ├── a
│   │   │       ├── PHILIPS_MPRAGE
│   │   │       │   ├── c
│   │   │       └── SIEMENS_MPRAGE
│   │   │           ├── e
│   │   │           ├── f
│   │   └── Dementia
│   │       ├── Dementia_L
│   │       │   ├── GE_FSPGR
│   │       │   │   ├── i
│   │       │   ├── PHILIPS_MPRAGE
│   │       │   │   ├── k
│   │       │   └── SIEMENS_MPRAGE
│   │       │       ├── m
│   │       │       ├── n
│   │       └── Dementia_R
│   │           ├── GE_FSPGR
│   │           │   ├── i
│   │           ├── PHILIPS_MPRAGE
│   │           │   ├── k
│   │           └── SIEMENS_MPRAGE
│   │               ├── m
│   │               ├── n
│   └── T1
│       ├── CN
│       │   ├── CN_L
│       │   │   ├── GE_FSPGR
│       │   │   │   ├── a
│       │   │   ├── PHILIPS_MPRAGE
│       │   │   │   ├── c
│       │   │   └── SIEMENS_MPRAGE
│       │   │       ├── e
│       │   │       ├── f
│       │   └── CN_R
│       │       ├── GE_FSPGR
│       │       │   ├── a
│       │       ├── PHILIPS_MPRAGE
│       │       │   ├── c
│       │       └── SIEMENS_MPRAGE
│       │           ├── e
│       │           ├── f
│       └── Dementia
│           ├── Dementia_L
│           │   ├── GE_FSPGR
│           │   │   ├── i
│           │   ├── PHILIPS_MPRAGE
│           │   │   ├── k
│           │   └── SIEMENS_MPRAGE
│           │       ├── m
│           │       ├── n
│           └── Dementia_R
│               ├── GE_FSPGR
│               │   ├── i
│               ├── PHILIPS_MPRAGE
│               │   ├── k
│               └── SIEMENS_MPRAGE
│                   ├── m
│                   ├── n
├── train
│   ├── FA
│   │   ├── CN
│   │   │   ├── CN_L
│   │   │   │   ├── GE_FSPGR
│   │   │   │   │   ├── b
│   │   │   │   ├── PHILIPS_MPRAGE
│   │   │   │   │   ├── d
│   │   │   │   └── SIEMENS_MPRAGE
│   │   │   │       ├── g
│   │   │   │       ├── h
│   │   │   └── CN_R
│   │   │       ├── GE_FSPGR
│   │   │       │   ├── b
│   │   │       ├── PHILIPS_MPRAGE
│   │   │       │   ├── d
│   │   │       └── SIEMENS_MPRAGE
│   │   │           ├── g
│   │   │           ├── h
│   │   └── Dementia
│   │       ├── Dementia_L
│   │       │   ├── GE_FSPGR
│   │       │   │   ├── j
│   │       │   ├── PHILIPS_MPRAGE
│   │       │   │   ├── l
│   │       │   └── SIEMENS_MPRAGE
│   │       │       ├── o
│   │       │       ├── p
│   │       └── Dementia_R
│   │           ├── GE_FSPGR
│   │           │   ├── j
│   │           ├── PHILIPS_MPRAGE
│   │           │   ├── l
│   │           └── SIEMENS_MPRAGE
│   │               ├── o
│   │               ├── p
│   └── T1
│       ├── CN
│       │   ├── CN_L
│       │   │   ├── GE_FSPGR
│       │   │   │   ├── b
│       │   │   ├── PHILIPS_MPRAGE
│       │   │   │   ├── d
│       │   │   └── SIEMENS_MPRAGE
│       │   │       ├── g
│       │   │       ├── h
│       │   └── CN_R
│       │       ├── GE_FSPGR
│       │       │   ├── b
│       │       ├── PHILIPS_MPRAGE
│       │       │   ├── d
│       │       └── SIEMENS_MPRAGE
│       │           ├── g
│       │           ├── h
│       └── Dementia
│           ├── Dementia_L
│           │   ├── GE_FSPGR
│           │   │   ├── j
│           │   ├── PHILIPS_MPRAGE
│           │   │   ├── l
│           │   └── SIEMENS_MPRAGE
│           │       ├── o
│           │       ├── p
│           └── Dementia_R
│               ├── GE_FSPGR
│               │   ├── j
│               ├── PHILIPS_MPRAGE
│               │   ├── l
│               └── SIEMENS_MPRAGE
│                   ├── o
│                   ├── p
└── validation
    ├── FA
    │   ├── CN
    │   │   ├── CN_L
    │   │   │   ├── GE_FSPGR
    │   │   │   │   ├── aa
    │   │   │   ├── PHILIPS_MPRAGE
    │   │   │   │   ├── bb
    │   │   │   └── SIEMENS_MPRAGE
    │   │   │       ├── cc
    │   │   │       ├── dd
    │   │   └── CN_R
    │   │       ├── GE_FSPGR
    │   │       │   ├── aa
    │   │       ├── PHILIPS_MPRAGE
    │   │       │   ├── bb
    │   │       └── SIEMENS_MPRAGE
    │   │           ├── cc
    │   │           ├── dd
    │   └── Dementia
    │       ├── Dementia_L
    │       │   ├── GE_FSPGR
    │       │   │   ├── ee
    │       │   ├── PHILIPS_MPRAGE
    │       │   │   ├── ff
    │       │   └── SIEMENS_MPRAGE
    │       │       ├── gg
    │       │       ├── hh
    │       └── Dementia_R
    │           ├── GE_FSPGR
    │           │   ├── ee
    │           ├── PHILIPS_MPRAGE
    │           │   ├── ff
    │           └── SIEMENS_MPRAGE
    │               ├── gg
    │               ├── hh
    └── T1
        ├── CN
        │   ├── CN_L
        │   │   ├── GE_FSPGR
        │   │   │   ├── aa
        │   │   ├── PHILIPS_MPRAGE
        │   │   │   ├── bb
        │   │   └── SIEMENS_MPRAGE
        │   │       ├── cc
        │   │       ├── dd
        │   └── CN_R
        │       ├── GE_FSPGR
        │       │   ├── aa
        │       ├── PHILIPS_MPRAGE
        │       │   ├── bb
        │       └── SIEMENS_MPRAGE
        │           ├── cc
        │           ├── dd
        └── Dementia
            ├── Dementia_L
            │   ├── GE_FSPGR
            │   │   ├── ee
            │   ├── PHILIPS_MPRAGE
            │   │   ├── ff
            │   └── SIEMENS_MPRAGE
            │       ├── gg
            │       ├── hh
            └── Dementia_R
                ├── GE_FSPGR
                │   ├── ee
                ├── PHILIPS_MPRAGE
                │   ├── ff
                └── SIEMENS_MPRAGE
                    ├── gg
                    ├── hh

I really have no idea how to do this, I'll appreciate it if you can help me.

I've seen this post Distributing thousands of files over subfolders, but I think it doesn't suit my problem

Thanks

Improve this question

2 Answers 2

Reset to default
0

This Shellcheck-clean Bash program might be adaptable for what you want:

#! /bin/bash -p

#### Utility functions

function quit
{
    local -r msg=${1-unknown error}

    printf '%s: ERROR: %s\n' "$0" "$msg" >&2
    exit 1
}

# Do a random shuffle of the 'dirs' array
# (Code adapted from "BashFAQ/026 - Greg's Wiki",
# <https://mywiki.wooledge.org/BashFAQ/026>)
function shuffle_dirs
{
   local i tmp size max rand

   size=${#dirs[@]}
   for ((i=size-1; i>0; i--)); do
      # RANDOM % (i+1) is biased because of the limited range of $RANDOM
      # Compensate by using a range which is a multiple of the rand modulus.

      max=$(( 32768 / (i+1) * (i+1) ))
      while (( (rand=RANDOM) >= max )); do :; done
      rand=$(( rand % (i+1) ))
      tmp=${dirs[i]} dirs[i]=${dirs[rand]} dirs[rand]=$tmp
   done

   return 0
}

# Copy an FA/..._L/... directory and directories paired with it under $srcdir
# to a "split" directory ('test', 'train', or 'validation') under $destdir
function copy_dirs
{
    local -r splitdir=$1    # 'test', 'train', or 'validation'
    local -r fa_l_dir=$2    # E.g. 'FA/CN/CN_L/GE_FSPGR/b'

    local -a parts
    local p0 p2 path

    IFS=/ read -r -a parts <<<"$fa_l_dir"
    for p0 in FA T1; do
        for p2 in "${parts[2]}" "${parts[2]%_L}_R"; do
            path="$p0/${parts[1]}/$p2/${parts[3]}/${parts[4]}"
            srcpath="$srcdir/$path"

            destpath="$destdir/$splitdir/$path"
            mkdir --parents --verbose -- "${destpath%/*}"
            cp --archive --verbose -- "$srcpath" "$destpath"
        done
    done

    return 0
}

#### "main"

if (( $# != 2 )); then
    printf 'usage: %s SRCDIR DESTDIR\n' "$0" >&2
    exit 1
fi

declare -r srcdir=$1
declare -r destdir=$2

# Check for existence of required directories
for d in "$srcdir" "$srcdir/FA" "$srcdir/T1" "$destdir"; do
    [[ -e $d ]] || quit "'$d' does not exist"
    [[ -d $d ]] || quit "'$d' is not a directory"
done

shopt -s nullglob

for groupdir in "$srcdir"/FA/*/*_L/*/; do
    # Get the list of FA/..._L/group/... directories to be copied
    dirs=()
    for d in "$groupdir"*; do
        dirs+=( "${d#"$srcdir/"}" )
    done

    ndirs=${#dirs[*]}
    if (( ndirs == 0 )); then
        printf "%s: WARNING: no directories match '$groupdir*'\\n" "$0" >&2
        continue
    fi

    # Randomly shuffle the list of directories to randomize which directories
    # get copied to which locations
    shuffle_dirs

    declare -i pc_sum=0 lo=0 hi
    for splitdir_pc in test:60 train:20 validation:20; do
        splitdir=${splitdir_pc%:*}
        pc=${splitdir_pc#*:}

        pc_sum+=$pc
        hi='(ndirs * pc_sum + 50)/100 - 1'

        for ((i=lo; i<=hi; i++)); do
            copy_dirs "$splitdir" "${dirs[i]}"
        done

        lo='hi+1'
    done
done

exit 0
  • The program takes two command line arguments:
    1. The path to the directory containing the FA and T1 directories.
    2. The path to a directory to contain the test, train, and validation directories (and the directories that will be copied under them).
  • If you are interested, I'm happy to add documentation to the code or answer any questions that you have.
Improve this answer
6
  • Thank you! I understand the script is taking all the folders under GE_FSPGR, PHILIPS_MPRAGE and SIEMENS_MPRAGE, shuffling them and copying 60, 20, 20% of them to the final destination, and that woks nice! But, when doing this, the random process sometimes leave one destination folder (train, test...) just with CN or just with Dementia but not both, and same for GE_FSPGR, PHILIPS_MPRAGE and SIEMENS_MPRAGE, so I lost the balance I want to maintain because not all "groups" will be represented in the test, train, val distributions...
    – Al_Mt
    Feb 18 at 17:32
  • So, How to keep this balance? I'm a bit new to bash so I don't know how to modify your script to get what I'm saying. Thanks again!
    – Al_Mt
    Feb 18 at 17:33
  • @Al_Mt, the requirement for "balance" at the GE_FSPGR, PHILIPS_MPRAGE and SIEMENS_MPRAGE was not specified in the original question, so I made no attempt to meet it. I think it can be done fairly easily. Look out for updates to the answer in the next 24 hours.
    – pjh
    Feb 18 at 18:46
  • @Al_Mt, I've modified the code to try to maintain balance between groups.
    – pjh
    Feb 18 at 23:28
  • Thanks @pjh! It is working very nice. I have some further questions I you don't mind. to change the percentages I just change the line with test:60 train:20 validation:20 with the desire percentage distribution, rigth? When these percentages are computed, it is rounding up or down? I've noticed you changed this line hi='(ndirs * pc_sum + 50)/100 - 1' in this update, previous was hi='((ndirs-1) * pc_sum + 50)/100', why so? what's the implication? Thank you again!!
    – Al_Mt
    Feb 19 at 16:33
0

Compute hashes, of either pathname or file contents. Here is an example hash function.

$ date | shasum
7ae62fd4e6483c966fd23d8eeafabc934226914d  -

Take the first character: ... shasum | cut -c1

Create sixteen symlinks, roughly corresponding to your desired percentages. Point them at your three existing directories.

ln -s /data/test     0
...
ln -s /data/test     9

ln -s /data/train    a
ln -s /data/train    b
ln -s /data/train    c

ln -s /data/validate d
ln -s /data/validate e
ln -s /data/validate f

Now it's easy. Obtain the initial character of the hash, and cp the src file to that destination directory.

You will obtain approximately the right number of files in each category. If you prefer resolution of 1/256 rather than 1/16, feel free to create symlinks for the first two characters of each hash.

EDIT

You said you had a bunch of files, for example

FA/CN/CN_L/GE_FSPGR/a/one.csv
FA/CN/CN_L/GE_FSPGR/a/two.csv
...
FA/CN/CN_L/GE_FSPGR/a/one_hundred.csv

You wish to choose one of three conditions for each file: train, test, validate.

Let's hash a particular pathname:

$ echo FA/CN/CN_L/GE_FSPGR/a/one.csv | shasum
16930ea7216d650735ea7c223a67371c5a5ed770  -
$
$ echo FA/CN/CN_L/GE_FSPGR/a/one.csv | shasum | cut -c1
1

Repeating this for the other files yields d and 2.

The several symlinks that have been set up will cause $ cp one.csv 1 and $ cp one_hundred.csv 2 to land in the "test" area, and $ cp two.csv d will become part of the validation data.

Improve this answer
3
  • Thanks for the answer. I can't run shasum on folders, every letter (a,b,c...) in my example are folders. Also, I don't understand your answer, would you mind to be a bit more detailed?
    – Al_Mt
    Feb 16 at 20:24
  • So, I have to define which files are going to test, train or validation set? I have almost 20k files, can't it be automatic?
    – Al_Mt
    Feb 16 at 20:51
  • The whole point of computing echo ${filename} | shasum | cut -c1 is to come up with a single character that determines the fate of the file (train / test / validate). And then the symlinks map sixteen possible characters to the three categories.
    – J_H
    Feb 17 at 2:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .