0

I'm trying to fix my YARN problem. When task is submitted to YARN it's creates new directory with all needed settings and scripts. After finishing this task directory is removed.

My task is failing after 0-2 seconds so files are removed so fast I cannot save them. I also don't know exact name of file before running task (there is autoincementing counter), but I know parent directory and I could try to guess exact path.

I would like to protect or copy this files (whole directory) in some way. I guess I could set up cron running every second and copying parent directory (I don't know if size of files wouln't exceed cp capabilities). I probably could also change (every second) every file in directory into read-only mode (but it could interfere with YARN processes and create new problems). I can't change permissions to directory - YARN wouldn't be able to create them in first place.

Is there better solution? (YARN detail is not important, but maybe someone knows how to use some unknown to me YARN features).

3

Several of your ideas won't work.

  • Cron works only at the minute intervall.
  • Making files read only doesn't prevent deletion.
  • Making the directory read only prevents deletion, but also creation.
  • On the other hand, no size of a file exceeds cp capabilities.

You best solution is to find a way to disable the deletion of the temporary files.

If that doesn't work, the best way is not to copy, but to link the files. If you know the directory where the files will be created, use this in a second shell:

while true; do
    ln sourcedir/* targetdir &> /dev/null
done

You have to terminate this after you are done. sourcedir and targetdir must be on the same file system. this will create a hard link of every file. When it runs again, the target exists, so it would display an error message, therefor the redirect to /dev/null. Unless the files in the source are removed very quickly, you should have all your files in the target directory.

Edit

For a limited number of nested directories, use

ln sourcedir/* sourcedir/*/* sourcedir/*/*/* targetdir &> /dev/null

For an arbitrary level of nesting, use find

find sourcedir -type f -exec ln --target-directory targetdir {} +
  • Thank. I suspected my ideas were not going to work :) Is i possible to use this method if files in my directories could be nested? I tested it on another directory and error hard link not allowed for directory. – franiis Aug 31 '18 at 6:28
  • I added some ideas. – RalfFriedl Aug 31 '18 at 6:32
  • Thanks, it works. It flattens directory structure, but files are accessible. – franiis Aug 31 '18 at 7:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.