I'm attempting to get a bash script working, and running into terrible errors along the way. The aim of the script is to break several large text files, into multiple files with a line count read from the file.

DIR="$( cd "$( dirname "$0" )" && pwd )" 
for i in 21 22 23 24 25 26 27 28 29 30 33 34 35 36 37 38 39 210 211 212 213 214 215 216 217 218 310 311 312 313 314 315 316 317 318
    lines="`head -1 $DIR/C$i/DOPC-C$i.xyz`"
    echo $lines
    split -a4 -d -l$lines $DIR/C$i/DOPC-C$i.xyz $DIR/C$i/DOPC-C$i-

The large text file has the number of molecules as its first line, so I'm using the head command to read it in, then passing this to split as the number of lines to. It has a formatting similar to:

C 0.41238 0.2301928 0.123123
H 0.123123 0.123233 0.5234234
H 0.123123 0.123233 0.5234234
C 0.41238 0.2301928 0.123123
H 0.123123 0.123233 0.5234234
H 0.123123 0.123233 0.5234234

However, when I run this from terminal, the memory usage of my system from free -m shoots up from 1.5GB to 16GB in use, and becomes extremely unresponsive. It works correctly for the first two files, and splits them how I wanted, though. Any advice?

EDIT: The source files are all ~200-300MB. When I run the split command directly on any of the files it works fine. There are 30 files which need to be split in this way, C21,C22, C23, etc. I reran the script and it got through the first ten files before it hit the memory limit this time.

EDIT2: So, I made a sort of very heavy handed work around. After running it through three files, I simply put

echo 3 | tee /proc/sys/vm/drop_caches

I noticed after the split command my memory in use would sharply spike according to free -m The spike also wouldn't go away when closing the terminal window where I ran the split command. I believe there is some issue in the configuration of disc caching on my system: Linux must be caching either the files I'm writing from or writing to and not cleaning up. When putting this to run every 3rd file, the script runs, albeit relatively slowly, through all of the files, and my system remains stable after. My suspicion is that this caching may also be related in part to it being a NTFS file system that I'm working on.

  • How large are the files? – Volker Siegel Aug 3 '14 at 23:09
  • What's the number of lines for the third files? Does the third file have reasonable (not multi-GB) line lengths? – Gilles 'SO- stop being evil' Aug 3 '14 at 23:11
  • What happens when you use split on the third file directly? – Volker Siegel Aug 3 '14 at 23:12
  • Edited the post to answer you. – Decimak Aug 3 '14 at 23:19
  • Single files work? Strange... run it with set -x at the start, to see the actual command lines. If that looks ok, what happens when you insert a sleep 20 after the split, to make it more like the single file handling - does the system get responsive again in between? – Volker Siegel Aug 3 '14 at 23:38

Additional explanation based on NTFS filesystem performance

After writing the lower section ot this answer, the OP pointed out that the script is running it on a NTFS disk, and suspects that may be part of the problem.

This would not be too surprising: There are performance problems with NTFS speciffically related to handling many small files. And we are creating small files in the order of millions - per input file.

So, bad NTFS performance would be an alternative explanation for the performance degradation, while the extreme use of memmory seems still to be related to mmap().

Bad NTFS performance
Configuring NTFS file system for performance

Explaining memmory problem by strong use of mmap()

The memory problem that occurs with split in your script seems to be related to the use of mmap in 'split'.

strace shows the following calls for each output file:

28892 open("xx02", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
28892 fstat(3, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
28892 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f821582f000
28892 write(3, "sometext\n", 30) = 30
28892 close(3)                          = 0
28892 munmap(0x7f821582f000, 4096)      = 0

Based on the examples, for a rough estimate of the files to handle,
we assume input files of 300MB, output files of 100B:

That gives us about 3000000 files to write. We write only one at once. But we use mmap(). That means, for each of the files, at least one memory page is used, which is 4096 B in size.

Taking that into account, we touch about 12GB of memory (1) for one input file (but not all at once). Three million files, and 12 GB, sounds like it could cause some work for the kernel.

Basically, it looks like split is just not made for this job, because it uses mmap().
That is a good thing in other situations.
But in this extreme case of input, it will mess up the memory management badly - which then needs some time to clean up. (2)

(2) It will not really use too much memmory at the same time, but mmap a huge number of small files in short time.

(1) Or address space only?

  • +1 for a nice catch! The answer then woyld be to use a "head -1" to get the number of lines, then awk to read chunks of n+2 lines and write them in a file – Olivier Dulac Aug 4 '14 at 9:24
  • @OlivierDulac Yes, awk, I thought this too... That is, until I found that it used mmap() to write the output files... spooky! – Volker Siegel Aug 4 '14 at 9:36
  • In the awk solution, do you close the files as soon as you wrote into them? It should help (I can't do a test today :/ ) – Olivier Dulac Aug 4 '14 at 9:38
  • @OlivierDulac Yes, I do close, and confirmed by strace. – Volker Siegel Aug 4 '14 at 13:19

I'm curious to solve the memory problem with the split solution from the question, but independently, this alternative approach may be useful:

You could use csplit instead of split to split this kind of file.

For csplit, you need to define a pattern to define where to split, and you can use the lines with a single number as separator - if you know there are no such lines in the comments.

It's not clear to me what the memory problem is about, but using a different tool may work around it.

But there is also the advantage that the command gets simpler, no need to get the number first.

The command would be something like:

csplit --elide-empty-files -n4 in.txt '/^[0-9]\+$/' '{*}'

  • I attempted to use the csplit command, and I ran into the same problem with memory running out, but it happened after only one file. – Decimak Aug 4 '14 at 4:04
  • Hmmm... was hoping it works "streaming-like"... but it's still unclear where the issue is... Nest to try sould be sed or awk probably - to be sure it's working on a stream. – Volker Siegel Aug 4 '14 at 4:22
  • I took a look at strace of csplit: it does mmap for the input, and each output file. I don't see whu that should be a problem, though. – Volker Siegel Aug 4 '14 at 4:29
  • Maybe not so thrifty. It is probably better if you combine these two. – mikeserv Aug 4 '14 at 8:30
  • I almost added a third, using awk - but then I found it's using mmap() too for writing (?!). These two answers are not related, I think - one is purely an alternative, avoiding the need to understand the problem. The other is underestanding the problem. At least, that was the plan. – Volker Siegel Aug 4 '14 at 8:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.