Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I developed an algorithm for a fairly hard problem in mathematics which is likely to need several months to finish. As I have limited resources only, I started this on my Ubuntu 12.04 (x86) laptop. Now I want to install some updates and actually restart the laptop (the "please reboot" message is just annoying).

Is there a way to save an entire process including its allocated memory for continuation beyond a reboot?

Here is some information about the process you might need. Please feel free to ask for further information if needed.

  • I called the process in a terminal with the command "./binary > ./somefile &" or "time ./binary > ./somefile &", I cannot really remember.
  • It's printing some debug information to std::cerr (not very often).
  • It's currently using roughly 600.0 kiB and even though this will increase, it's unlikely to increase rapidly.
  • the process runs with normal priority
  • the kernel is 3.2.0-26-generic-pae, the cpu is an AMD, the operating system is Ubuntu 12.04 x86.
  • it runs since 9 days and 14 hours (so too long to cancel it ;-) )
share|improve this question
2  
Out of curiosity, what are you calculating? –  user1261166 Jul 25 '12 at 9:48
2  
@user1261166: I study the Target Visitation Problem (which is an extension of the Travelling Salesman Problem) with Branch-and-Cut approach. Thus I need to know as many facets of some special high-dimensional polytope as possible. Basically, it's blowing up a big problem to a gigantic one and then trying to solve just a bit to reduce it afterwards. –  stefan Jul 25 '12 at 21:58
4  
Oh, cool. Hope you discover something cool :) –  user1261166 Jul 26 '12 at 7:49
3  
It doesn't quite answer your question but have you considered running your code on a dedicated cluster in the future? Those are hardly shut down and I'm sure there is some computing grid available to you. Not only are they on all the time but also quite a bit faster (especially if you can parallelise your code). You could even have a go at setting one up yourself (look up Oracle Grid Engine). –  Wojtek Rzepala Aug 2 '12 at 8:25
    
I never thought of this beeing such a popular question (at least way more popular than every other question by me so for). Since the process finished now (unexpectedly, though without a crash), I will try out each method shortly. Thanks everyone! –  stefan Aug 8 '12 at 9:20
show 1 more comment

4 Answers

up vote 25 down vote accepted

The best/simplest solution is to change your program to save the state to a file an reuse that file to restore the process.

Based upon the wikipedia page about application snapshots there are multiple alternatives:

  1. There is also cryopid but it seems to be unmaintained.
  2. Linux checkpoint/restart seems to be a good choice but your kernel needs to have CONFIG_CHECKPOINT_RESTORE enabled.
  3. criu is probably the most up to-date project and probably your best shot but depends also on some specific Kernel options which your distribution probably hasn't set.

This is already too late but another more hands-on approach is to start your process in a dedicated VM and just suspend and restore the whole Virtual machine. Depending on your hypervisor you can also move the machine between different hosts.

For the future think about where you run your long-running processes, how to parallize them and how to handle problems, e.g. full disks, process gets killed etc.

share|improve this answer
    
@stefan So which one did you choose? –  Tobias Kienzler Feb 4 '13 at 8:13
add comment

A fairly "cheap" way to do this would be to do the processing in a VM (e.g., with VirtualBox). Before you shut down suspend the VM and save the state. After booting restore the VM & state.

This does have the disadvantage of requiring killing and restarting the job. But if it's actually going to be running for several months then a nine days difference becomes trivial (5% increase over 6 months).


Edit: I just realized that Ulrich already mentioned this in unnumbered item 4 on his list.

I would still encourage you to consider this as an option, especially since none of the alternatives seem like a robust solution. Each has a reason why it may not work.

I suppose the best thing to do would be to try one of those and if it doesn't work restart the job in a VM.

share|improve this answer
    
I like this idea, thinking outside of the box ;) –  Tim Jul 24 '12 at 18:56
2  
"Appreciate mundanity: after all, a pencil is high technology" –Alan Kay, 1982 (Probably not a direct quote). –  bahamat Jul 24 '12 at 19:01
    
Good read, thanks man. –  Tim Jul 24 '12 at 19:34
add comment

Take a peek at the tool CryoPID.

From the home page: "CryoPID allows you to capture the state of a running process in Linux and save it to a file. This file can then be used to resume the process later on, either after a reboot or even on another machine."

share|improve this answer
1  
Used this before to save the state of a python script running on a Linux box and moved it to a FreeBSD box and resumed there. Some arcane magic going on there ;) –  Tim Jul 24 '12 at 18:30
add comment

If you end up needing to restart your program, I would encourage you to spend some time adding some features to your code that might save you time in the future.

If the process is going to be run for a long time, being able to save the entire process state when you restart the machine is perhaps not hugely helpful if your process crashes while it is running.

I would encourage you to have your program output to a file "checkpoint" data. This data should be sufficient that your program will be able to resume from the state it was at when the checkpoint file was saved. You need not save the entire process, just a snapshot of the relevant variables being used in your calculation, sufficient for your calculation to resume where it left off. Your code would also need to include some way of reading in the data from this file to obtain it's starting state.

You could set up your code so when you send it a signal, it saves one of these checkpoint files, so you can save the "state" of your calculation at any point.

Additionally, being able to see how the data changes as the calculation progresses might be interesting in itself!

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.