Question summary: Is there any way to lower the ram usage of ratarmount? What factors affect the ram usage (other than tar size)?

Edit/clarification of question: What method could I use to reduce the resident memory to a minimum?

My case:

  • I'm using ratarmount to mount massive tar files (800gb is the largest one I have for example).
  • I have a server with 6x6tb drives, 20 cores, and 256gb ram.
  • I have around 20-25tb of these types of tar files mounted
  • With the above amount of tar files mounted, I just hit the max amount of ram on my sever ~250gb
  • The tar files have a crazy amount of (possibly hundreds of millions) small files and folders..

My Issue: Ratarmount has now hit 240gb of ram usage for me and its pretty much maxing out my servers ram. I still need to mount another 5-10tb of tar files.

It seems a bit inconsistent with how much ram it uses. For example the first time you mount it will end up using more ram than if it mounts from an index file. Its a bit of a mystery to me of what I can experiment with to make it more inefficient. Open to any ideas. I need to decrease the ram usage by ~25% to finish my project.

Side note... I saw that this user created ratarmount. Its an awesome program that completely saved me, but I have no idea of how to get a hold of him or whats considered the established/acceptable way of reaching someone on here or github

  • What do you mean by "maxing out my servers ram"? Is the server swapping? By design, the operating system will try to use up all the memory available; after all, you paid for those 256 GB of RAM, and leaving it unused would be wasteful. Basic rule is that if the server is not swapping then all is fine.
    – AlexP
    Nov 12 '19 at 23:27
  • Can you use zswap? It is likely that the index in memory can be compressed. By using zswap you are trading CPU time for RAM.
    – Ole Tange
    Nov 13 '19 at 0:57
  • @AlexP My main question is: what method could I use to reduce the resident memory to a minimum? I update the post above too.
    – lessharm
    Nov 13 '19 at 2:56
  • 1
    If it's on GitHub, open an issue there. Unless they have specified otherwise, that's an acceptable way of communicating about the project on GitHub
    – muru
    Nov 13 '19 at 3:08
  • 1
    If ratarmount uses more RAM when mounting for the first time than from an index, it sounds like a memory leak. If I had been the author, I would have appreciated a bug report, so consider filing that. It is also likely that he never optimized the code for memory usage, because he never envisioned it would be used so heavily. If you earn money from getting it to work, offer to pay him to make it use less memory.
    – Ole Tange
    Nov 13 '19 at 8:25

I just tested ratarmount with a file with 10M files. If you are not actively stating all the files, ratarmount will eventually swap out.

So a very simply solution is to add swap space to your server. If you use zswap you may even be able to avoid touching the physical disks.

I also confirmed your observation that it takes much more memory to do indexing+mounting than it does to simply do mounting. This means there is a memory leak in ratarmount (probably the data structure is copied and an unneeded copy is not destroyed). So you can minimize data usage by index+mount, followed immediately by umount and then mounting again.

  • Hey, thanks for that. Thats what we have been doing.
    – lessharm
    Feb 4 '20 at 22:41
  • Also try the SQLite change: github.com/mxmlnkn/ratarmount/issues/12
    – Ole Tange
    Feb 5 '20 at 12:36
  • Wow, that's awesome, thank you. I'm going to look into that. Ill also try to get in touch with mxmlnkn to send him some btc.
    – lessharm
    Feb 7 '20 at 7:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.