I just realised something rather stupid (my side) happened. I am currently running multiple programs on several machines while appending their output.

E.g. machine01 is running "./calc 1 >> calc_one.txt", machine02 is running "./calc 2 >> calc_two.txt" etc.

These calculations run long, and I just found out that something annoying happened. I'm using git, and I was running these programs on a number of the machines. I wrote a new mode of my program and called to run it on machine03.

Because I'm using git, I figured it was save to remove the .txt files in the directory.

I just realised the machines share a directory. So I have removed "calc_one.txt" and "calc_two.txt", to which output is being appended.

Now, the programs are such that they actually only print output at the end. In the program are a large number of threads (OMP) and these generate results (C++ vector) which are printed after all threads finish.

As such, the stream is still coming. But an experiment with BASH and "sleep" indicates that replacing the .txt file won't make it append to the new file. Is there anything I can do or do I need to start these simulations from scratch?

My sincere thanks for your help, Daimonie

PS: The system is fedora. I have no administrative privileges.


 1. Get the process PID. For example:

pgrep calc

 2. Find deleted files. For example:

find /proc/12345/fd -ls | grep '(deleted)'
# or (no pid)
lsof | grep calc_one.txt
# ...
lsof -p 12345

 3. a) Copy the file, (from previous result, here 3). A snapshot:

cp /proc/12345/fd/3 mycopy1.txt

 3. b) New append file:

tail -c +0 -f /proc/12345/fd/3 > mycopy2.txt


-c +K, Output bytes starting with the Kth. As we want entire file we say +0
-f   , Follow. Output appended data as the file grows.

  • Hey, on my test this does work. But on the actual machine, I can't find the correct file. I get something like 5882926 0 lrwx------ 1 daimonie institute 64 Jan 2 01:53 /proc/59158/fd/2 -> /dev/pts/1\ (deleted). I'm guessing it's too old and has been removed? – Daimonie Jan 2 '16 at 0:55
  • @Daimonie: Ach. Sorry. I somehow managed to ignore the redirect >> and went for the calc program opening internal file descriptor. What does your /proc/PID/fd/1 hold? – Runium Jan 2 '16 at 1:06
  • Isn't it "pgrep" by the way? Also, it holds the file we're looking for. For some reason, it is blinking. – Daimonie Jan 2 '16 at 1:08
  • Additionally, the program is running in background mode. I just added an ampersand to the tail... line before, which worked - I'm assuming this is proper? – Daimonie Jan 2 '16 at 1:09
  • @Daimonie: That sounds good. – Runium Jan 2 '16 at 1:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.