Track total data written in and read from a folder within bash script

Question

I have a bash script to do calculation. This calculation generates big scratch files as big as 12 GB and the disk usage of scratch folder is ~30 GB. I want to know how much total data is written to disk during the process and how much total data is read. This will help me to understand the disk IO bottlenecks and choose a better scratch disk type.

Question : Track written data (MB or GB) in a folder between a time interval. Similarly track read data from a folder between a time interval.

The present version of my script is below.

#!/bin/bash
# Running QM-JOB: helix HPC
    d="$1"  # .dal file
    m="$2"  # .mol file
    n="$3"  # number of CPU cores to be used for this calculation.
dir=$(pwd)
dt=$(date  +%Y-%m-%d:%H:%M:%S )
echo -e 'Job started @ '$dt'' >> /home/vayu/dalton/runlog.log
echo "-----------------------------------------------"
df -h /dev/md0
echo "-----------------------------------------------"

folder="<path/to/the/folder>" #Scratch folder

# start IO log on "scratch folder" (no idea how to implement this)
echo "-----------------------------------------------"

export OMP_NUM_THREADS=$n
source /opt/intel/parallel_studio_xe_2017.0.035/compilers_and_libraries_2017/linux/bin/compilervars.sh intel64
source /opt/intel/parallel_studio_xe_2017.0.035/compilers_and_libraries_2017/linux/mkl/bin/mklvars.sh intel64
source /opt/intel/parallel_studio_xe_2017.0.035/compilers_and_libraries_2017/linux/mpi/bin64/mpivars.sh intel64

./application_script "$d" "$m" "$n" "$folder"

    dt2=$(date '+%d/%m/%Y %H:%M:%S');

#stop "scratch folder" IO log
#print total data written in "scratch folder"
#print total data read from "scratch folder"

Answer 1

You can read the I/O stats from /proc/self/io before and after your task, and subtract the values from the "write_bytes" and "read_bytes" lines. See "man proc" for some details. It does not differentiate by device or folder, though.

Here's an example:

#!/bin/bash
cat /proc/$$/io
dd if=/dev/zero of=/tmp/iotest bs=1M count=5
sync
cat /proc/$$/io
rm /tmp/iotest