I have a problem /proc/[PID]/io and memmap
I need to profile application IO that uses python library astropy.
One of the problem I have is that IO read/write total bytes are incorrect in /proc/[PID]/io.
I have a script copy.sh
:
#!/bin/bash
cp huge.fits huge.2.fits
#python copyfits.py
#python copyfitsMemmapOFF.py
sleep 5
cat /proc/$$/io
I comment the cp
line to and the line I want to run for each test.
copyfits.py
contains:
from astropy.io import fits
hdulist = fits.open('huge.fits', mode='readonly')
hdulist.writeto('huge.2.fits')
hdulist.close()
copyfitsMemmapOFF.py
contains:
from astropy.io import fits
hdulist = fits.open('huge.fits', mode='readonly', memmap=False)
hdulist.writeto('huge.2.fits')
hdulist.close()
heres the IO results for each solution:
cp huge.fits huge.2.fits
rchar: 9749929202
wchar: 9749551680
python copyfits.py
**rchar: 8399421**
wchar: 9749551685
python copyfitsMemmapOFF.py
rchar: 9757502959
wchar: 9749551685
I understand that memmap being off will lead to inconsistent IO results when using this variable to monitor how much the application read files.
How/Why memmap are not counted in standard IO, and how can I found out about those IO ?
Because if it's kernel instead of application reading file, the file is still accessed.