I'm attempting to get a bash script working, and running into terrible errors along the way. The aim of the script is to break several large text files, into multiple files with a line count read from the file.
#!/bin/bash DIR="$( cd "$( dirname "$0" )" && pwd )" for i in 21 22 23 24 25 26 27 28 29 30 33 34 35 36 37 38 39 210 211 212 213 214 215 216 217 218 310 311 312 313 314 315 316 317 318 do lines="`head -1 $DIR/C$i/DOPC-C$i.xyz`" echo $lines lines=$((lines+2)) split -a4 -d -l$lines $DIR/C$i/DOPC-C$i.xyz $DIR/C$i/DOPC-C$i- done
The large text file has the number of molecules as its first line, so I'm using the head command to read it in, then passing this to split as the number of lines to. It has a formatting similar to:
3 Comment C 0.41238 0.2301928 0.123123 H 0.123123 0.123233 0.5234234 H 0.123123 0.123233 0.5234234 3 Comment C 0.41238 0.2301928 0.123123 H 0.123123 0.123233 0.5234234 H 0.123123 0.123233 0.5234234
However, when I run this from terminal, the memory usage of my system from free -m shoots up from 1.5GB to 16GB in use, and becomes extremely unresponsive. It works correctly for the first two files, and splits them how I wanted, though. Any advice?
EDIT: The source files are all ~200-300MB. When I run the split command directly on any of the files it works fine. There are 30 files which need to be split in this way, C21,C22, C23, etc. I reran the script and it got through the first ten files before it hit the memory limit this time.
EDIT2: So, I made a sort of very heavy handed work around. After running it through three files, I simply put
echo 3 | tee /proc/sys/vm/drop_caches
I noticed after the split command my memory in use would sharply spike according to free -m The spike also wouldn't go away when closing the terminal window where I ran the split command. I believe there is some issue in the configuration of disc caching on my system: Linux must be caching either the files I'm writing from or writing to and not cleaning up. When putting this to run every 3rd file, the script runs, albeit relatively slowly, through all of the files, and my system remains stable after. My suspicion is that this caching may also be related in part to it being a NTFS file system that I'm working on.