Tag Info

Hot answers tagged

63

Yes. You should most definitively always have swap enabled, except if there is a very compelling, forbidding reason (like, no disk at all, or only network disk present). The reason is that swap is not only useful when your applications consume more memory than there is physical RAM (actually, in that case, swap is not very useful at all because it ...


59

Vim sometimes has trouble with files that have unusually long lines. It's a text editor, so it's designed for text files, with line lengths that are usually at most a few hundred characters wide. A database file may not contain many newline characters, so it could conceivably be one single 100 Mb long line. Vim will not be happy with that, and although it ...


48

I'm going to disagree with a few of the opinions that I see stated here. I'd still be creating a SWAP partition especially in a production environment. I do it for my home machines and VMs as well. These days I'm sizing them around 1-1.5 times memory. 2 times memory used to be the rule of thumb. The swap disk is "cheap" in that it does not need to be ...


47

In my experience Vim chokes not on large files, but on long lines. Use this command to have mysqldump use shorter lines at the expense of a larger file: $ mysqldump --complete-insert -u -p Additionally, you can open Vim and ask it not to parse your .vimrc file or load any plugins with this command: $ vim -u NONE output.sql Loading Vim in this manner ...


37

It can. There are 2 different out of memory conditions you can encounter in linux. Which you encounter depends on the value of sysctl vm.overcommit_memory (/proc/sys/vm/overcommit_memory) Introduction: The kernel can perform what is called 'memory overcommit'. This is when the kernel allocates programs more memory than is really present in the system. This ...


17

Maybe: I've given a lot of thought to this topic and seen opinions landing on both sides of the argument more times than I can count. My approach was to develop a way to find out. Start with an active swap partition of what you think is a sufficient size. Then, open a terminal in a workspace and issue the command free -hs 1 which will report usage once ...


14

The command line option -o (o standing for "Override-sort-field") also works on my Xubuntu machine and according to the Mac man page of top it should work on a Macintosh too. If I want to short by memory usage I usually use top -o %MEM which sorts by the column %MEM. But I can use VIRT, RES or SHR too. On a Macintosh I would probably use mem or vsize. I ...


13

This can be do the same thing with purge: sync && echo 3 > /proc/sys/vm/drop_caches From man proc: /proc/sys/vm/drop_caches (since Linux 2.6.16) Writing to this file causes the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free. To free ...


13

A swap partition has significant value above and beyond simply acting as some extra RAM when you run out. For one, Linux uses as much memory as possible to cache files and IO operations, if you have some swap you may find that more memory goes into caching IO and making it faster (by minimizing disk access and also lowering wear on SSDs) as opposed to ...


12

Try using less instead of vim if you want to view a large file directly. Vim tries to do a lot of different stuff when it first loads - scanning the file (potentially in multiple passes) to try to determine what syntax to use, and performing syntax highlighting, and searching for modelines at the top and bottom of the file. Then as you edit the file, vim ...


12

The truth is that regardless of which way you look at it - whether your process choked up due to the system's memory manager or due to something else - it is still a bug. What happened to all of that data you were just processing in memory? It should have been saved. While overcommit_memory= is the most general way of configuring Linux OOM management, it is ...


11

"load VIM without .vimrc and plugins (clean VIM) e.g. for HUGE files gvim -u NONE -U NONE -N largefile.sql


11

It might help to up /proc/sys/vm/page-cluster (default: 3). From the kernel documentation (sysctl/vm.txt): page-cluster page-cluster controls the number of pages up to which consecutive pages are read in from swap in a single attempt. This is the swap counterpart to page cache readahead. The mentioned consecutivity is not in terms of ...


10

PROBABLY! I have run into problems in the past with an "appliance" I built, running Linux - running on a compact flash device, I did not want to wear my CF by using swap, and there was enough memory for the application. Most of these appliances worked fine, but on a particularly busy box, I ran into a problem: MEMORY FRAGMENTATION Without swap space, the ...


9

Vim does not just load the file as-is into memory. It converts it into internal structures (lines, words, etc), performs syntax highlighting using an internal script language, and so on; all of which consumes memory (a whole lot more than a byte for a character) and CPU time.


8

Active: Memory currently being used by a process Inactive: Memory that has been freed but is still cached since it may be used again. If more Free memory is required, this memory can be cleared and become free. This memory is not cleared before it is needed, because "free memory is wasted memory", it doesn't cost anything to keep the old data around in case ...


8

Only if you want to be able to hibernate to swap (This feature is also called "suspend to disk" and involves saving the entire contents of RAM and turning off the power). Typically this is only used on laptops and other mobile devices, so it depends.


7

It depends on the settings you're running with, in particular memory overcommit (/proc/sys/vm/overcommit_memory; see man 5 proc for details). If memory overcommit is disabled, the editor's (and possibly other programs attempting at the same time) attempt to allocate memory will fail. They'll get a failure result from the system call. Its up to each program ...


7

If your memory is exhaustively used up by the processes to the extent which can possibly threaten the stability of the system, then the OOM killer comes into picture. It is the task of the OOM Killer to kill the processes until enough memory is freed for the smooth functioning of the rest of the process. The OOM Killer has to select the best process to ...


6

It appears that the stack memory limit is not allocated (anyway, it couldn't with unlimited stack). https://www.kernel.org/doc/Documentation/vm/overcommit-accounting says: The C language stack growth does an implicit mremap. If you want absolute guarantees and run close to the edge you MUST mmap your stack for the largest size you think you will ...


6

> /usr/bin/time -v sleep 1 Command being timed: "sleep 1" User time (seconds): 0.00 System time (seconds): 0.00 Percent of CPU this job got: 0% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.01 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set ...


5

You may try loading it as a binary. I've had luck with that for really big, non-text files vim -b HUGEFILE It's also possible IIRC to use vim as a hex editor see: http://usevim.com/2012/06/20/vim-binary-files/


5

So, I did this thing in testing and, yeah, it consumes a lot of memory. I pointedly used a smaller number as well. I can imagine that bash hogging those resources for days on end could be a little irritating. ps -Fp "$$"; : {1..10000000}; ps -Fp "$$" UID PID PPID C SZ RSS PSR STIME TTY TIME CMD mikeserv 32601 4241 0 3957 3756 4 ...


5

One option to do a quick test could be to use a KGDB enabled kernel and stop the kernel manually and test, see this link. On another note, things I remember that could cause your pauses: cpufreq, cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_transition_latency, the value is in ns (4000 in my AMD FX(tm)-8120 Eight-Core Processor) shouldn't be a problem, ...


5

There's no universal and clear answer because it depends on a task you're about to perform. If you're about to run DB, HTTP, Virtualization or Cache server you should never enable any kind of swap, regardless of the ram amount you have. If you have a desktop or mixed-task host and you have 16+ Gb of fast RAM - take a look here : zRam


5

You may try adding the programs you most care about to a cgroup and tuning swappiness so that the next time the application runs the programs you add are less likely to be candidates for swapping. Some of their pages will likely still be swapped out but it may get around your performance problems. A large part of it is probably just the "stop and start" ...


5

Linux kernel maintainers are listed in the MAINTAINERS file in the kernel source code. There's a specific section for memory management: MEMORY MANAGEMENT L: linux-mm@kvack.org W: http://www.linux-mm.org S: Maintained F: include/linux/mm.h F: include/linux/gfp.h F: include/linux/mmzone.h F: include/linux/memory_hotplug.h ...


4

Hopefully your problem is more to do with VIMs need for temporary files (such as swap) more than RAM. In many cases, the temporary files created by VIM are in the same directory of the file you are opening. If this is the case for you, then you can verify by checking the available disk space in the current directory. Fortunately, there is good ...


4

Per the gawk manual, which is a good general awk language reference: An important aspect to remember about arrays is that array subscripts are always strings. That is, awk arrays are always associative, and numeric keys are stringified. Only the keys that are in use are stored in the array (and maybe some extra space for the future). Numeric indices ...


4

It may be possible that your program used shared memory and didn't clean that up. There are three variants of shared memory on linux: 1.) POSIX shared memory (the one implemented by glibc) is accessible via files on the tmpfs pseudo-filesystem and are usually mounted by the system on places like /dev/shm, /run, /run/shm or /run/lock. The best way to find ...



Only top voted, non community-wiki answers of a minimum length are eligible