Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am using clusters at my university's high performance computing center, they are using SGE system to do the job management. You might know that by default the SGE will output two files: NAME.oXXXXX, and NAME.poXXXXX. I want to know if this output will slow down my simulation.

Usually my simulation would take a few weeks, and the NAME.oXXXXX file would grow very big, like several gigabytes, so I am kind of worrying such a big file will slow down my simulation.

Does anybody have any experience, and if so how to turn it off. Actually I do need this output log to check my results occasionally.

PS: the storage I am using now is not afs, but pscratch, so no use of fsync.

Thanks a lot!

share|improve this question
add comment

1 Answer

It depends on how much you write to the log, but my guess is that it's a line every so many steps, so you shouldn't worry about that - most simulations will usually do many more steps (each being thousands of computations) in between writing the log lines.

The .oXXXX output file is the standard output of your simulation package, which would get written to the console otherwise (.poXXXX is the 'parallel output'), so the frequency of writing to them depends on your program. So how much you write entirely depends on your simulation package - refer to its manual to check how to reduce the frequency of output, if you can do that. Either way, that should have a minimal effect on the speed of your simulation.

share|improve this answer
    
"my guess is that it's a line every so many steps", no, it writes around 20 lines every time step. "Also, if you are worried about the size of the file, I'm sure your simulation package will have an option to reduce the frequency of writing the log.", okay, this I will try, but I'd really hope this could be solved on SGE or system's side, not on the original code. –  Daniel Jun 12 '12 at 22:15
    
Oops, edited answer slightly before your comment. What goes into the log is the output of your program - SGE only provides an environment to execute it. These log files essentially catch whatever goes to the standard output, so whatever your simulation package would normally write to the screen goes there. –  Wojtek Rzepala Jun 12 '12 at 22:21
    
That's exactly my question, I am curious to know what happens if this file grows into a big file like 2G big. How did the *nix system write into this file, by some fopen command as in C++, or doing some append command, which might be keep writing in a certain storage address? Or what? I think if no open and close repeating operations then it would be not slow. Because the file is really big, I am not sure if linux, ext system like windows has some file size limitations....... –  Daniel Jun 12 '12 at 22:34
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.