Tag Info

Hot answers tagged

67

Yes. You should most definitively always have swap enabled, except if there is a very compelling, forbidding reason (like, no disk at all, or only network disk present). The reason is that swap is not only useful when your applications consume more memory than there is physical RAM (actually, in that case, swap is not very useful at all because it ...


49

I'm going to disagree with a few of the opinions that I see stated here. I'd still be creating a SWAP partition especially in a production environment. I do it for my home machines and VMs as well. These days I'm sizing them around 1-1.5 times memory. 2 times memory used to be the rule of thumb. The swap disk is "cheap" in that it does not need to be ...


28

Additional information provided in the comments reveals that the OP is using a GUI method to create the .tar.gz file. GUI software often includes a lot more bloat than the equivalent command line equivalent software, or performs additional unnecessary tasks for the sake of some "extra" feature such as a progress bar. It wouldn't surprise me if the GUI ...


22

The command line option -o (o standing for "Override-sort-field") also works on my Xubuntu machine and according to the Mac man page of top it should work on a Macintosh too. If I want to short by memory usage I usually use top -o %MEM which sorts by the column %MEM. But I can use VIRT, RES or SHR too. On a Macintosh I would probably use mem or vsize. I ...


17

Maybe: I've given a lot of thought to this topic and seen opinions landing on both sides of the argument more times than I can count. My approach was to develop a way to find out. Start with an active swap partition of what you think is a sufficient size. Then, open a terminal in a workspace and issue the command free -hs 1 which will report usage once ...


13

A swap partition has significant value above and beyond simply acting as some extra RAM when you run out. For one, Linux uses as much memory as possible to cache files and IO operations, if you have some swap you may find that more memory goes into caching IO and making it faster (by minimizing disk access and also lowering wear on SSDs) as opposed to ...


13

"five million" files, and 1TB in total? Your files must be very small, then. I'd simply try rsync: rsync -alPEmivvz /source/dir remote.host.tld:/base/dir If you don't have that - or your use-case doesn't allow for using rsync, I'd at least check if 7z works with your data. It might not, but I think it's still worth a try: 7z a archive.7z /source/dir Or ...


11

It might help to up /proc/sys/vm/page-cluster (default: 3). From the kernel documentation (sysctl/vm.txt): page-cluster page-cluster controls the number of pages up to which consecutive pages are read in from swap in a single attempt. This is the swap counterpart to page cache readahead. The mentioned consecutivity is not in terms of ...


10

PROBABLY! I have run into problems in the past with an "appliance" I built, running Linux - running on a compact flash device, I did not want to wear my CF by using swap, and there was enough memory for the application. Most of these appliances worked fine, but on a particularly busy box, I ran into a problem: MEMORY FRAGMENTATION Without swap space, the ...


9

In modern systems none of the memory is actually copied just because a fork system call is used. It is all marked read only in the page table such that on first attempt to write a trap into kernel code will happen. Only once the first process attempt to write will the copying happen. This is known as copy-on-write. However it may be necessary to keep track ...


8

Only if you want to be able to hibernate to swap (This feature is also called "suspend to disk" and involves saving the entire contents of RAM and turning off the power). Typically this is only used on laptops and other mobile devices, so it depends.


8

Unless you can do better than 25:1 compression you are unlikely to gain anything from compressing this before snail-mailing, unless you have some hardware tape format that you can exchange the the third party. The largest common storage is blue ray and that will roughly get you 40Gb. You would need 25 to 1 compression on your data to get it to fit on that. ...


6

> /usr/bin/time -v sleep 1 Command being timed: "sleep 1" User time (seconds): 0.00 System time (seconds): 0.00 Percent of CPU this job got: 0% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.01 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set ...


6

The linux swapping algorithm works with the concept of "last recently used pages". Each page in virtual memory has an age associated with it. If the page is being frequently accessed then that page is supposed to be quite young in age while if a page is not being accessed, then that page becomes older. The older the pages get, the more likely they may get ...


5

One option to do a quick test could be to use a KGDB enabled kernel and stop the kernel manually and test, see this link. On another note, things I remember that could cause your pauses: cpufreq, cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_transition_latency, the value is in ns (4000 in my AMD FX(tm)-8120 Eight-Core Processor) shouldn't be a problem, ...


5

So, I did this thing in testing and, yeah, it consumes a lot of memory. I pointedly used a smaller number as well. I can imagine that bash hogging those resources for days on end could be a little irritating. ps -Fp "$$"; : {1..10000000}; ps -Fp "$$" UID PID PPID C SZ RSS PSR STIME TTY TIME CMD mikeserv 32601 4241 0 3957 3756 4 ...


5

I don't know of any place where the kernel exposes the filenames associated with the blocks that it has cached. According to this answer http://stackoverflow.com/a/4941371 The best you could probably do even with a custom kernel module would be to get a list of inodes and devices. From there you would still likely need to walk the filesystem looking for ...


5

You should never have swap larger than the maximum size you'd be able to tolerate waiting for the kernel to swap in/out; otherwise, you're just creating a new failure mode for your system (becoming unrecoverably bogged down in swapping). Note that, despite modern drives being able to transfer on the order of GB/sec, Linux is typically only able to move swap ...


5

There's no universal and clear answer because it depends on a task you're about to perform. If you're about to run DB, HTTP, Virtualization or Cache server you should never enable any kind of swap, regardless of the ram amount you have. If you have a desktop or mixed-task host and you have 16+ Gb of fast RAM - take a look here : zRam


5

You may try adding the programs you most care about to a cgroup and tuning swappiness so that the next time the application runs the programs you add are less likely to be candidates for swapping. Some of their pages will likely still be swapped out but it may get around your performance problems. A large part of it is probably just the "stop and start" ...


5

Linux kernel maintainers are listed in the MAINTAINERS file in the kernel source code. There's a specific section for memory management: MEMORY MANAGEMENT L: linux-mm@kvack.org W: http://www.linux-mm.org S: Maintained F: include/linux/mm.h F: include/linux/gfp.h F: include/linux/mmzone.h F: include/linux/memory_hotplug.h ...


5

Did you consider torrent? Peer-to-Peer might be your best option for an over-the-internet transfer: At least as fast as other internet transfers: your upload speed will determine the transfer speed No data corruption Choose which files to transfer first No extra local/cloud storage space needed Free You didn't tell which OS you were using, but as you're ...


5

This is not a loop, but recursion and the memory increases linear over the time, which is what you don't want. If you want a loop with constant memory usage, you can do it this way: #!/bin/sh while 1; do mysql -h "localhost" -u "root" "-p********" "database" < "update.sql" sleep 5 done


5

How can I see the raw memory data used by an application... Once you have obtained the process' PID (using ps(1) or pidof(8) for instance), you may access the data in its virtual address space using /proc/PID/maps and /proc/PID/mem. Gilles wrote a very detailled answer about that here. ... and all the files its accessing in my filesystem, network ...


5

If the system supports process accounting, and accounting is enabled, then there may be records available in the process accounting file, e.g. on RedHat Linux: $ grep -q BSD_PROCESS_ACCT=y /boot/config-* && echo hooray hooray $ sudo touch /var/log/pacct $ sudo accton /var/log/pacct $ ... $ sudo accton # turn it off Parsing the acct(5) file is ...


4

If you're looking for memory numbers that add up have a look at smem: smem is a tool that can give numerous reports on memory usage on Linux systems. Unlike existing tools, smem can report proportional set size (PSS), which is a more meaningful representation of the amount of memory used by libraries and applications in a virtual memory system. ...


4

You might be running into trouble because /dev/mem has holes in it: memory locations that don't exist can't be read. strings is designed to read a regular file (or pipe or other input stream) from beginning to end and assumes there can't be holes in its input. It probably aborts the first time it gets a read error, which would correspond to the first hole ...


4

It may be possible that your program used shared memory and didn't clean that up. There are three variants of shared memory on linux: 1.) POSIX shared memory (the one implemented by glibc) is accessible via files on the tmpfs pseudo-filesystem and are usually mounted by the system on places like /dev/shm, /run, /run/shm or /run/lock. The best way to find ...


4

cat doesn't use any significant CPU time (unless maybe on-disk decryption or decompression is involved and accounted to the cat process which is the one reading from disk) or memory. It just reads the content of the files and writes it to the pipe in small chunks in a loop. However, here, you don't need it. You can just do: gzip -c file1 file2 file3 file4 ...


4

There is no way to tell if you need swap space or not if the only parameter we know is the amount of RAM installed. In any case, there is a common misconception that having a swap space is negatively affecting system performance. This is incorrect. As long as you have enough RAM, having a swap area whatever its size doesn't hurt performance at all. What ...



Only top voted, non community-wiki answers of a minimum length are eligible