Hot answers tagged

68

Yes. You should most definitively always have swap enabled, except if there is a very compelling, forbidding reason (like, no disk at all, or only network disk present). The reason is that swap is not only useful when your applications consume more memory than there is physical RAM (actually, in that case, swap is not very useful at all because it ...


49

I'm going to disagree with a few of the opinions that I see stated here. I'd still be creating a SWAP partition especially in a production environment. I do it for my home machines and VMs as well. These days I'm sizing them around 1-1.5 times memory. 2 times memory used to be the rule of thumb. The swap disk is "cheap" in that it does not need to be ...


28

Additional information provided in the comments reveals that the OP is using a GUI method to create the .tar.gz file. GUI software often includes a lot more bloat than the equivalent command line equivalent software, or performs additional unnecessary tasks for the sake of some "extra" feature such as a progress bar. It wouldn't surprise me if the GUI ...


17

Maybe: I've given a lot of thought to this topic and seen opinions landing on both sides of the argument more times than I can count. My approach was to develop a way to find out. Start with an active swap partition of what you think is a sufficient size. Then, open a terminal in a workspace and issue the command free -hs 1 which will report usage once ...


13

A swap partition has significant value above and beyond simply acting as some extra RAM when you run out. For one, Linux uses as much memory as possible to cache files and IO operations, if you have some swap you may find that more memory goes into caching IO and making it faster (by minimizing disk access and also lowering wear on SSDs) as opposed to ...


13

"five million" files, and 1TB in total? Your files must be very small, then. I'd simply try rsync: rsync -alPEmivvz /source/dir remote.host.tld:/base/dir If you don't have that - or your use-case doesn't allow for using rsync, I'd at least check if 7z works with your data. It might not, but I think it's still worth a try: 7z a archive.7z /source/dir Or ...


11

It might help to up /proc/sys/vm/page-cluster (default: 3). From the kernel documentation (sysctl/vm.txt): page-cluster page-cluster controls the number of pages up to which consecutive pages are read in from swap in a single attempt. This is the swap counterpart to page cache readahead. The mentioned consecutivity is not in terms of ...


10

PROBABLY! I have run into problems in the past with an "appliance" I built, running Linux - running on a compact flash device, I did not want to wear my CF by using swap, and there was enough memory for the application. Most of these appliances worked fine, but on a particularly busy box, I ran into a problem: MEMORY FRAGMENTATION Without swap space, the ...


9

In modern systems none of the memory is actually copied just because a fork system call is used. It is all marked read only in the page table such that on first attempt to write a trap into kernel code will happen. Only once the first process attempt to write will the copying happen. This is known as copy-on-write. However it may be necessary to keep track ...


8

Only if you want to be able to hibernate to swap (This feature is also called "suspend to disk" and involves saving the entire contents of RAM and turning off the power). Typically this is only used on laptops and other mobile devices, so it depends.


8

Unless you can do better than 25:1 compression you are unlikely to gain anything from compressing this before snail-mailing, unless you have some hardware tape format that you can exchange the the third party. The largest common storage is blue ray and that will roughly get you 40Gb. You would need 25 to 1 compression on your data to get it to fit on that. ...


8

In your second reference, where it says The memory allocator gets a big chunk of memory (say, 4 pages, or 4 * 4 KiB) and divides this into much smaller chunks "the memory allocator" is not the kernel but the libc routines like malloc. The kernel allocates memory to the process in page-sized chunks. malloc keeps track of used and unused portions of ...


6

The linux swapping algorithm works with the concept of "last recently used pages". Each page in virtual memory has an age associated with it. If the page is being frequently accessed then that page is supposed to be quite young in age while if a page is not being accessed, then that page becomes older. The older the pages get, the more likely they may get ...


5

It seems like the -o flag will take the actual column name. So if the top command shows only "mem" then the command should be "top -o mem". For the ubuntu machine I am testing with, the column is called "%MEM". On the OSX Yosemite I tried, it is "mem".


5

You should never have swap larger than the maximum size you'd be able to tolerate waiting for the kernel to swap in/out; otherwise, you're just creating a new failure mode for your system (becoming unrecoverably bogged down in swapping). Note that, despite modern drives being able to transfer on the order of GB/sec, Linux is typically only able to move swap ...


5

There's no universal and clear answer because it depends on a task you're about to perform. If you're about to run DB, HTTP, Virtualization or Cache server you should never enable any kind of swap, regardless of the ram amount you have. If you have a desktop or mixed-task host and you have 16+ Gb of fast RAM - take a look here : zRam


5

You may try adding the programs you most care about to a cgroup and tuning swappiness so that the next time the application runs the programs you add are less likely to be candidates for swapping. Some of their pages will likely still be swapped out but it may get around your performance problems. A large part of it is probably just the "stop and start" ...


5

Linux kernel maintainers are listed in the MAINTAINERS file in the kernel source code. There's a specific section for memory management: MEMORY MANAGEMENT L: linux-mm@kvack.org W: http://www.linux-mm.org S: Maintained F: include/linux/mm.h F: include/linux/gfp.h F: include/linux/mmzone.h F: include/linux/memory_hotplug.h ...


5

Did you consider torrent? Peer-to-Peer might be your best option for an over-the-internet transfer: At least as fast as other internet transfers: your upload speed will determine the transfer speed No data corruption Choose which files to transfer first No extra local/cloud storage space needed Free You didn't tell which OS you were using, but as you're ...


5

This is not a loop, but recursion and the memory increases linear over the time, which is what you don't want. If you want a loop with constant memory usage, you can do it this way: #!/bin/sh while 1; do mysql -h "localhost" -u "root" "-p********" "database" < "update.sql" sleep 5 done


5

How can I see the raw memory data used by an application... Once you have obtained the process' PID (using ps(1) or pidof(8) for instance), you may access the data in its virtual address space using /proc/PID/maps and /proc/PID/mem. Gilles wrote a very detailled answer about that here. ... and all the files its accessing in my filesystem, network ...


5

If the system supports process accounting, and accounting is enabled, then there may be records available in the process accounting file, e.g. on RedHat Linux: $ grep -q BSD_PROCESS_ACCT=y /boot/config-* && echo hooray hooray $ sudo touch /var/log/pacct $ sudo accton /var/log/pacct $ ... $ sudo accton # turn it off Parsing the acct(5) file is ...


5

The short story: If your mobo posts, and your system boots, and free/top show your ram as 16 gB, then it works. Even mobo makers can under-report capacity of system boards, so the real test is if ram is installed correctly, matched correctly, runs, ie, boots, and runs with stability, ie, doesn't crash, then it works. You can also test by trying to use all ...


5

niceload --noswap yourprg is made for exactly that situation: It looks at swapping activity: If swapping out: Let process run If swapping in: Let process run If swapping in and out: Suspend process until swapping stops and resume the process when swapping has stopped It does not suspend the process before the swapping starts, but lets swapping run for 1 ...


4

There is no way to tell if you need swap space or not if the only parameter we know is the amount of RAM installed. In any case, there is a common misconception that having a swap space is negatively affecting system performance. This is incorrect. As long as you have enough RAM, having a swap area whatever its size doesn't hurt performance at all. What ...


4

It seems to me that you can't magically "make the system responsive again". You either incur the penalty or reading pages back from swap space into memory now or you incur it later, but one way or the other you incur it. Indeed, if you do something like swapoff -a && swapon -a then you may feel more pain rather than less, because you force some pages ...


4

There is kernel setting /proc/sys/vm/overcommit_memory Citation from excellent article: Since 2.5.30 the values are: 0 (default): as before: guess about how much overcommitment is reasonable, 1: never refuse any malloc(), 2: be precise about the overcommit - never commit a virtual address space larger than swap space plus a fraction overcommit_ratio ...


3

Even though the topic is quite old, I want to share another project that emerged from the cgroups Linux kernel feature. https://github.com/gsauthof/cgmemtime: cgmemtime measures the high-water RSS+CACHE memory usage of a process and its descendant processes. To be able to do so it puts the process into its own cgroup. For example process A ...


3

This problem might be caused by an incorrect sizing of the maximum size of the connection tracking table and the hash table. The Linux kernel tries to allocate contiguous pages to track the connection tables for the iptables nf_conntrack module. As you don't have enough physical memory, conntrack fails back to vmalloc. This table is not dynamically created ...


3

Swapping allows physical pages to be moved around, in the sense that a page used for one purpose could have its content swapped out then used for another purpose. Under a garden variety virtual memory management system, there is no such thing as fragmentation of physical memory as far as applications are concerned. Each page allocated by an application can ...



Only top voted, non community-wiki answers of a minimum length are eligible