2

I'm a complete newbie, so please excuse my ignorance and/or potentially wrong terminology.

I'm using an Ubuntu server through ssh for brain image processing (one command with several programs, execution takes ~4-5 hours per brain), which I run through the Terminal. Since the server has limited storage (~200GB) and the brain data are big (2-3 GB input, 500 MB output), I'm constantly downloading processed data and uploading new to-be-processed ones, using FileZilla.

The brain image processing is quite RAM-intensive and has failed several times due to memory issues, so I'm now doing these two procedures (procedure 1=brain image processing vs. procedure 2=uploading/downloading) separately and manually -- i.e., when I'm doing one, I won't do the other at the same time. But I was wondering if there's a more efficient way of doing this, while still ensuring that the brain image processing doesn't fail.

In a nutshell, I would like procedure 1 to take up as much RAM as it needs, with the "rest" being allocated to procedure 2. I'm currently assigning procedure 1 all 8 cores, but it only uses all 8 only so often (because of how the program is written). Is there a way to achieve this, ideally one that still allows me to use FileZilla (because it's so fast and simple, though I'm not opposed to uploading/downloading through the Terminal)? For example, might it be the case that whichever process I start first takes "precedence" and just takes whatever memory it needs at a given point in time and any other processes just take what's left? Or how does RAM get allocated between concurrently running processes (especially if started from different software, if that matters)?

I hope all of this made sense. Thanks in advance!

4
  • 1
    Are you sure the limiting factor is RAM and not disk space? Up- or downloding files shouldn't need significant RAM.
    – terdon
    Commented Feb 6, 2022 at 13:30
  • some systems are configured with no swap space under the mistaken idea that swap is bad. In fact, it is for the exact case of temporary high memory use without having to kill your processes. Check that sufficient swap is enabled.
    – stark
    Commented Feb 6, 2022 at 13:41
  • Use the free command to see your memory and swap.
    – waltinator
    Commented Feb 6, 2022 at 19:21
  • Wha cool ! I played with brain image processing (state of the art 35 years ago). 5-6 hours processing... on an MC68000 10Mhz, don't tell me you just can't compete. The very best advice I can give is : A/ Rework your dataset. There must be more than 50% useless. B/ don't use any sort of bloated distro launching tons of useless daemons C/ Can't do anything with A & B ? => Buy more cores ! BTW, supporting @terdon comment : Would RAM be the limiting factor, the first troubles you'd get would be stack overflows.
    – MC68020
    Commented Feb 10, 2022 at 2:08

1 Answer 1

1

It sound like a job for GNU Parallel:

Let us assume you have enough CPU power to run 16 brains in parallel, and that transferring data take very little time compared to the run time (4-5 hours).

From your sending machine I would run:

parallel --delay 1m -j16 -S server --trc {}.output 'process_brain {} > {}.output' ::: brain*.input
Option Description
--delay 1m At most start a job every minute (this will give the first jobs full bandwidth for a minute to transfer the first file)
-j16 run at most 16 jobs in parallel
-S server on server
--trc {.}.output transfer inputfile to server, transfer output back when done, clean up (remove both files on server)
'process_brain {} > {.}.output' run process_brain on brain*.input and save it in brain*.output
::: input source separator
brain*.input files to use for input

Consider spending 20 minutes on reading chapter 1+2 of https://doi.org/10.5281/zenodo.1146014. Your command line will love you for it.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .