Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Current scripts create and delete multiple files over NFS. I have been told that this is the problem, so was thinking if the run directory the script ran in looked like a single file to NFS then I might alleviate the problem.

I should add that we have hundreds of these scripts working in their own run directories but all over NFS at any one time, and we have a bottleneck due to the creation and deletion of many tens of files for each run directory.

I have since found http://code.google.com/p/fuse-zip/ that I will investigate. Could someone share their experiences with this?

share|improve this question
    
-1 you keep saying "the problem" but you didn't actually say what problem you're having! –  Celada Sep 10 '12 at 12:28
    
@Celada: The problem is stated in the title? I'm asking if anyone knows of a means to create a directory inside a file as clearly stated. Try taking time to digest instead of automatically assuming something is missing. I don't want, and so have not asked for, information of why NFS gives me a problem. I just want to know if anyone knows how to create a filesystem inside a file in a 'host' filesystem. If you tstill think this is not clear then please elucidate. –  Paddy3118 Sep 10 '12 at 13:22
    
You should probably try to sanitize the question a little, maybe make it a bit more succinct. –  vitiv Sep 10 '12 at 14:45
2  
I think Celada is thinking of the XyProblem in wondering why you're trying to do this. Most of us do not have a problem creating and deleting files over NFS. –  derobert Sep 10 '12 at 14:48
1  
With zip it's not really the compression that will cost you, it's the fragmentation: deleting a file from the middle of a zip archive leaves a hold which must be filled by moving everything else. You'd probably be better off with a loopback-mounted image of a standard filesystem (e.g. ext3) but, again, that depends on what original problem you've trying to solve! –  Celada Sep 10 '12 at 15:39

1 Answer 1

up vote 6 down vote accepted

Ok, from the comments, it seems like it'll fall into one of these suggestions:

  • You're making and deleting a bunch of temporary files, taking relatively little disk space.

    Suggestion: Mount a tmpfs somewhere on the local machine, and use that. (You may already have one under /tmp anyway)

  • Most of them are temporary files, but a few need to be kept

    Suggestion: tmpfs again, but before unmounting the tmpfs at the end (thus losing its contents), copy the few files you need off it.

  • They're temporary, but they're not small.

    Suggestion: Local storage.

  • You need to access these many small files from multiple machines, concurrently.

    Suggestion: Beefier NFS server and/or network. Dedicated NFS server for just this load. Distributed filesystems.

  • I'm having a hard time coming up with a scenario where the approach in your question is the best idea. But if you do want to do it...

    Suggestion: Make a single file on the NFS server. Use losetup to map it to a loop device, then mkfs that, and mount it. (Depending on which filesystem you're using, you make be able to mkfs the file directly, that'll work with mke2fs for example). Probably, mount it with the async flag (or similar). This will perform much better than a zip file.

Another thing I'll note, depending on what you're making and deleting all these temporary files for, files may not be the best approach. E.g., maybe they should be rows in a database of some sort.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.