2

Due to an unpredicted scenario I am currently in need of finding a solution to the fact that an application (which I do not wish to kill) is slowly hogging the entire disk space. To give more context

  • I have an application in Python that uses multiprocessing.Pool to start 5 threads. Each thread writes some data to its own file.

  • The program is running on Linux and I do not have root access to the machine.

  • The program is CPU intensive and has been running for months. It still has a few days to write all the data.

  • 40% of the data in the files is redundant and can be removed after a quick test.

  • The system on which the program is running only has 30GB of remaining disk space and at the current rate of work it will surely be hogged before the program finishes.

Given the above points I see the following solutions with respective problems

  • Given that the process number i is writing to file_i, is it safe to move file_i to an external location? Will the OS simply create a new instance of file_i and write to it? I assume moving the file would remove it and the process would end up writing to a "dead" file?

  • Is there a "command line" way to stop 4 of the 5 spawned workers and wait until one of them finishes and then resume their work? (I am sure one single worker thread would avoid hogging the disk)

  • Suppose I use CTRL+Z to freeze the main process. Will this stop all the other processes spawned by multiprocessing.Pool? If yes, can I then safely edit the files as to remove the redundant lines?

Given the three options that I see, would any of them work in this context? If not, is there a better way to handle this problem? I would really like to avoid the scenario in which the program crashes just few days before its finish.

migrated from serverfault.com Jun 5 '14 at 14:36

This question came from our site for system and network administrators.

  • Do you know how the file is being written? Is it simply appending data to the end of the file, using or random access based on an offset? Some programs will not write to the file once you move it - even if you recreate it because the file handle pointer is destroyed. So I recommend you do some testing. – jeffatrackaid Jun 5 '14 at 14:32
  • 1
    If this is a simple log type file with data being added to the end, you may be able to copy the file to another server and then just use >filename to truncate the file. You run the risk of possibly losing a bit of data in this process but could be better than losing everything. – jeffatrackaid Jun 5 '14 at 14:34
  • @jeffatrackaid Each worker opens its own (text) file and writes data line by line. – Jernej Jun 5 '14 at 14:49
  • Are they threads of a single process, or different processes? – Stéphane Chazelas Jun 5 '14 at 15:06
  • @StéphaneChazelas They all seem to have each own PID, according to htop's output. – Jernej Jun 5 '14 at 16:25
2

If you move a file to a different filesystem, what happens under the hood is that the current contents of the file are copied and the original file is deleted. If the program was still writing to the file, it keeps writing to the now-deleted file. A deleted-but-opened file is in fact not deleted, but merely detached (it no longer has a name); the file is deleted for real when the program closes it. So you get the worst of both worlds: the file still uses as much disk space but you lose the remainder of the output.

You can press Ctrl+Z to suspend the foreground process, and resume it with the command bg or fg. All threads are suspended unless the program went through hoops to behave otherwise. (A program designed to spawn children over the network might behave otherwise. A single-process multi-thread program is highly likely to behave normally.) If the program consists of different processes, use the ps command to locate them all and something like kill -STOP 1234 1238 1239 to suspend them all (use kill -CONT … to resume them later).

If the program writes or even reads back and forth in the file, you can't remove its data under its nose. Moving the data at this stage might be doable but it would be difficult and dependent on how the program works. But given your description, the program probably just keeps appending to each file, in which case removing some data at the beginning is doable.

Don't edit the files: this is unlikely to do what you want. Most editors work by saving a new file and moving it in place of the old ones (this is more robust in case of a crash while saving). You can save disk space by truncating the beginning of the file. First, copy the file to save the data elsewhere. Then truncate the file to length 0. The program will keep appending at the position where it was before; if that position was 12345 then as soon as the program appends another byte the file will start with 12345 null bytes. Most of these null bytes will not take up any disk space: the file will be a sparse file.

# Suspend the program first, otherwise you'll lose output produced between cp and truncation!
for x in *.out; do
  cp "$x" /elsewhere/
  : >|"$x"  # truncate $x to size 0
done

Once the program has finished, you can append the remaining data to the files saved elsewhere. The tail utility can copy a file omitting the first N bytes; note that the argument is one plus the number of bytes to omit.

for x in *.out; do
  existing_size=$(stat -c %s "/elsewhere/$x")
  tail -c +$((existing_size+1)) "$x" >>"/elsewhere/$x"
done

If you have rsync 3.0.0 or above, you can use

rsync --append *.out /elsewhere/

Note that older rsync versions would overwrite the existing portion of the files with the newly-appeared null bytes from the source! Check your rsync versions before doing this.

  • Thanks for this extensive answer. Given that my program actually spawn processes I assume (from what you say) that it should be safe to simply send to all but one thread the SIGSTOP signal? – Jernej Jun 6 '14 at 7:20
  • @Jernej I recommend that you suspend all the processes. You can finagle and suspend only one process, work on its output file, resume that process and go on with the next process and so on, but that increases the risk of mistakes. – Gilles Jun 6 '14 at 9:16
  • My plan was to suspend all but one process and wait for it to finish. Then move its file to another server and resume the second worker and so on... Do you see any problems with this approach? – Jernej Jun 6 '14 at 9:43
  • @Jernej As long as you're very careful to only move a file which no currently-running process is writing to, that's fine. – Gilles Jun 6 '14 at 10:01
1

Without root access, your options are very limited. Your best bet, and only realistic chance of success IMHO, is to compress the files that have already been written and closed and hope that doing so frees up enough space for your program to complete. There might be other options available from the folks on SO, have you tried asking there?

  • The issue is that the files are fixed. Each worker works on its own file the entire time of the compuation. – Jernej Jun 5 '14 at 14:34
1

Given that the process number i is writing to file_i, is it safe to move file_i to an external location? Will the OS simply create a new instance of file_i and write to it? I assume moving the file would remove it and the process would end up writing to a "dead" file?

If the program is constantly opening the file, writing data, and closing again then yes, you can just move the file and it will create a new one. Most programs do not work this way. It's likely that the OS will show the file has been moved, but actually take up the disk space until the program closes the file handle.

Is there a "command line" way to stop 4 of the 5 spawned workers and wait until one of them finishes and then resume their work? (I am sure one single worker thread would avoid hogging the disk)

It would depend on exactly what kind of worker threads the program uses, and you would almost certainly need root access (you'd probably have to use debugging tools, and the exact sequence would be highly dependent on how the program work). Without root access and an intimate knowledge of the programs inner workings this is likely unfeasible.

Suppose I use CTRL+Z to freeze the main process. Will this stop all the other processes spawned by multiprocessing.Pool? If yes, can I then safely edit the files as to remove the redundant lines?

Depends again on exactly how the program is running. It's likely that suspending the process will suspend the workers as well, but not a guarantee. I don't believe this will change the problem however, as the file pointer of the file handle will still point to the same offset as before the file was edited.


Can you move any other files off the system? Or is this a VM where you could expand the virtual disks?

  • To address the first comment. Each worker opens up its file and it does not close it until it finishes its work. As for your final question this is an academic server and I am only allowed to manipulate files within my home directory. The system has a 440GB disk out of which ~290GB is used by the files I've described. – Jernej Jun 5 '14 at 14:33
  • 2
    Emphasis on "Most programs do not work this way." – goldilocks Jun 5 '14 at 14:38
  • There is no reason why root access would be needed to suspend a program. Unprivileged users can suspend and resume their own processes. – Gilles Jun 6 '14 at 1:56
  • @Gilles Who said anything about needing root access to suspend programs?? – Chris S Jun 6 '14 at 2:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.