61

I developed an algorithm for a fairly hard problem in mathematics which is likely to need several months to finish. As I have limited resources only, I started this on my Ubuntu 12.04 (x86) laptop. Now I want to install some updates and actually restart the laptop (the "please reboot" message is just annoying).

Is there a way to save an entire process including its allocated memory for continuation beyond a reboot?

Here is some information about the process you might need. Please feel free to ask for further information if needed.

  • I called the process in a terminal with the command "./binary > ./somefile &" or "time ./binary > ./somefile &", I cannot really remember.
  • It's printing some debug information to std::cerr (not very often).
  • It's currently using roughly 600.0 kiB and even though this will increase, it's unlikely to increase rapidly.
  • the process runs with normal priority
  • the kernel is 3.2.0-26-generic-pae, the cpu is an AMD, the operating system is Ubuntu 12.04 x86.
  • it runs since 9 days and 14 hours (so too long to cancel it ;-) )
Improve this question
6
  • 4
    Out of curiosity, what are you calculating?
    – Viktor Mellgren
    Commented Jul 25, 2012 at 9:48
  • 5
    @user1261166: I study the Target Visitation Problem (which is an extension of the Travelling Salesman Problem) with Branch-and-Cut approach. Thus I need to know as many facets of some special high-dimensional polytope as possible. Basically, it's blowing up a big problem to a gigantic one and then trying to solve just a bit to reduce it afterwards.
    – stefan
    Commented Jul 25, 2012 at 21:58
  • 6
    It doesn't quite answer your question but have you considered running your code on a dedicated cluster in the future? Those are hardly shut down and I'm sure there is some computing grid available to you. Not only are they on all the time but also quite a bit faster (especially if you can parallelise your code). You could even have a go at setting one up yourself (look up Oracle Grid Engine).
    – Wojtek
    Commented Aug 2, 2012 at 8:25
  • I never thought of this beeing such a popular question (at least way more popular than every other question by me so for). Since the process finished now (unexpectedly, though without a crash), I will try out each method shortly. Thanks everyone!
    – stefan
    Commented Aug 8, 2012 at 9:20
  • 1
    stackoverflow.com/questions/2134771/…
    – Ciro Santilli OurBigBook.com
    Commented Dec 8, 2016 at 23:37

4 Answers 4

Reset to default
51

The best/simplest solution is to change your program to save the state to a file an reuse that file to restore the process.

Based upon the wikipedia page about application snapshots there are multiple alternatives:

  1. There is also cryopid but it seems to be unmaintained.
  2. Linux checkpoint/restart seems to be a good choice but your kernel needs to have CONFIG_CHECKPOINT_RESTORE enabled.
  3. criu is probably the most up to-date project and probably your best shot but depends also on some specific Kernel options which your distribution probably hasn't set.

This is already too late but another more hands-on approach is to start your process in a dedicated VM and just suspend and restore the whole Virtual machine. Depending on your hypervisor you can also move the machine between different hosts.

For the future think about where you run your long-running processes, how to parallize them and how to handle problems, e.g. full disks, process gets killed etc.

Improve this answer
0
23

A fairly "cheap" way to do this would be to do the processing in a VM (e.g., with VirtualBox). Before you shut down suspend the VM and save the state. After booting restore the VM & state.

This does have the disadvantage of requiring killing and restarting the job. But if it's actually going to be running for several months then a nine days difference becomes trivial (5% increase over 6 months).

Edit: I just realized that Ulrich already mentioned this in unnumbered item 4 on his list.

I would still encourage you to consider this as an option, especially since none of the alternatives seem like a robust solution. Each has a reason why it may not work.

I suppose the best thing to do would be to try one of those and if it doesn't work restart the job in a VM.

Improve this answer
0
18

Take a peek at the tool CryoPID.

From the home page: "CryoPID allows you to capture the state of a running process in Linux and save it to a file. This file can then be used to resume the process later on, either after a reboot or even on another machine."

Improve this answer
4
  • 6
    Used this before to save the state of a python script running on a Linux box and moved it to a FreeBSD box and resumed there. Some arcane magic going on there ;)
    – Tim
    Commented Jul 24, 2012 at 18:30
  • I didn't know FreeBSD and Linux were binary-compatible; that's something very interesting I just learned. But does that mean they have exactly identical memory models? It seems incredulous to me that they have the same syscall conventions, the same libc (i guess fbsd use glibc), the same exact calling conventions at the asm level, etc. The incompatibilities sound to me as if you took a MacOS process and dumped it onto a Windows box; that's really quite amazing.
    – cat
    Commented May 7, 2016 at 15:33
  • Has anyone tried this recently? The site is gone, I can't find a .deb, building from source fails, etc. I'd like to know if it's possible before spending any longer on it. I'm on Debian if it matters.
    – John P
    Commented Jan 25, 2017 at 21:20
  • 3
    @JohnP It's available on GitHub now: github.com/maaziz/cryopid
    – starbeamrainbowlabs
    Commented Jun 6, 2017 at 9:18
7

If you end up needing to restart your program, I would encourage you to spend some time adding some features to your code that might save you time in the future.

If the process is going to be run for a long time, being able to save the entire process state when you restart the machine is perhaps not hugely helpful if your process crashes while it is running.

I would encourage you to have your program output to a file "checkpoint" data. This data should be sufficient that your program will be able to resume from the state it was at when the checkpoint file was saved. You need not save the entire process, just a snapshot of the relevant variables being used in your calculation, sufficient for your calculation to resume where it left off. Your code would also need to include some way of reading in the data from this file to obtain it's starting state.

You could set up your code so when you send it a signal, it saves one of these checkpoint files, so you can save the "state" of your calculation at any point.

Additionally, being able to see how the data changes as the calculation progresses might be interesting in itself!

Improve this answer
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .