Tag Info

Hot answers tagged

56

Vim sometimes has trouble with files that have unusually long lines. It's a text editor, so it's designed for text files, with line lengths that are usually at most a few hundred characters wide. A database file may not contain many newline characters, so it could conceivably be one single 100 Mb long line. Vim will not be happy with that, and although it ...


46

In my experience Vim chokes not on large files, but on long lines. Use this command to have mysqldump use shorter lines at the expense of a larger file: $ mysqldump --complete-insert -u -p Additionally, you can open Vim and ask it not to parse your .vimrc file or load any plugins with this command: $ vim -u NONE output.sql Loading Vim in this manner ...


36

It can. There are 2 different out of memory conditions you can encounter in linux. Which you encounter depends on the value of sysctl vm.overcommit_memory (/proc/sys/vm/overcommit_memory) Introduction: The kernel can perform what is called 'memory overcommit'. This is when the kernel allocates programs more memory than is really present in the system. This ...


18

NOTE: I'm going to assume that your machine has a memory mapping unit (MMU). There is a Linux version (µClinux) that doesn't require an MMU, and this answer doesn't apply there. What is an MMU? It's hardware—part of the processor and/or memory controller. Understanding shared library linking doesn't require you to understand exactly how an MMU works, just ...


14

Red Hat Enterprise Linux (RHEL) These are probably a good basis, looking at RHEL6's capabilities, they're covered here, titled: Red Hat Enterprise Linux 6 technology capabilities and limits.     NOTE: [5] The architectural limits are based on the capabilities of the Red Hat Enterprise Linux kernel and the physical hardware. Red Hat ...


14

Use quick tip using top command in linux/unix $top hit Shift + f , then choose the display to order by memory usage by hitting key n then press Enter. You will see active process ordered by memory usage. Or you can just press M after running top command. References ...


13

Use cron (or anacron). Cron is designed for running things at intervals. That is the only thing it does, and there has been a lot of work put into cron for many years to make it what it is today. The chances that you're going to write a better scheduler in your script are effectively nil. Using cron will work better, avoid having unnecessary code in your ...


12

The truth is that regardless of which way you look at it - whether your process choked up due to the system's memory manager or due to something else - it is still a bug. What happened to all of that data you were just processing in memory? It should have been saved. While overcommit_memory= is the most general way of configuring Linux OOM management, it is ...


12

Try using less instead of vim if you want to view a large file directly. Vim tries to do a lot of different stuff when it first loads - scanning the file (potentially in multiple passes) to try to determine what syntax to use, and performing syntax highlighting, and searching for modelines at the top and bottom of the file. Then as you edit the file, vim ...


12

This can be do the same thing with purge: sync && echo 3 > /proc/sys/vm/drop_caches From man proc: /proc/sys/vm/drop_caches (since Linux 2.6.16) Writing to this file causes the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free. To free ...


11

Let's consider how each solution works. uniq This requires that the file already be sorted. If not, you have to pipe it through sort first, which means that sort has to read the entire file into memory, reorder it (O(n log n)), and then write it into the pipe. The work of uniq is very cheap, since it only has to compare adjacent lines of its input. sort -u ...


11

"load VIM without .vimrc and plugins (clean VIM) e.g. for HUGE files gvim -u NONE -U NONE -N largefile.sql


9

When in top, typing "E" cycles through different memory units (kb, mb, gb etc) in the total memory info. "e" does the same individual process lines.


9

Vim does not just load the file as-is into memory. It converts it into internal structures (lines, words, etc), performs syntax highlighting using an internal script language, and so on; all of which consumes memory (a whole lot more than a byte for a character) and CPU time.


8

There is no difference betweem tmpfs and shm. tmpfs is the new name for shm. shm stands for SHaredMemory. See: Linux tmpfs. The main reason tmpfs is even used today is this comment in my /etc/fstab on my gentoo box. BTW Chromium won't build with the line missing: # glibc 2.2 and above expects tmpfs to be mounted at /dev/shm for # POSIX shared memory ...


7

It depends on the settings you're running with, in particular memory overcommit (/proc/sys/vm/overcommit_memory; see man 5 proc for details). If memory overcommit is disabled, the editor's (and possibly other programs attempting at the same time) attempt to allocate memory will fail. They'll get a failure result from the system call. Its up to each program ...


7

If your memory is exhaustively used up by the processes to the extent which can possibly threaten the stability of the system, then the OOM killer comes into picture. It is the task of the OOM Killer to kill the processes until enough memory is freed for the smooth functioning of the rest of the process. The OOM Killer has to select the best process to ...


6

Use cron because it is a better and more standard practice. At least if this is something that will regularly run (not just something you patched together in a minute). cron is a cleaner and more standard way. It's also better because it runs the shell detached from a terminal - no problem with accidental termination and dependencies on other processes. ...


6

It appears that the stack memory limit is not allocated (anyway, it couldn't with unlimited stack). https://www.kernel.org/doc/Documentation/vm/overcommit-accounting says: The C language stack growth does an implicit mremap. If you want absolute guarantees and run close to the edge you MUST mmap your stack for the largest size you think you will ...


5

Another way to limit this is to use Linux's control groups. This is especially useful if you want to limit a process's (or group of processes') allocation of physical memory distinctly from virtual memory. For example: $ cgcreate -g memory:/myGroup $ echo $(( 500 * 1024 * 1024 )) > /sys/fs/cgroup/memory/myGroup/memory.limit_in_bytes $ echo $(( 5000 ...


5

1. Virtual memory The system will ensure that processes will get the requested amount of memory despite being greater than physical memory. By this way the kernel allocates a virtual memory space of the maximum physical memory size it can handle. E.g. on a 32bit machine, the kernel will allocate a total of 2^32 i.e. 4GB of virtual addresses to every process ...


5

I used bcache only in a writethrough configuration, and IIRC even then bcache doesn't like at all if the cache device vanishes while the machine is running. Expect the bcache device to stall completely if that happens. I haven't tried to remove the cache device while the machine is powered down, so I can't say anything about that. I do think though that ...


5

From the Forensic's Wiki: Tools:Memory Imaging excerpt Linux /dev/mem On older Linux systems, the program dd can be used to read the contents of physical memory from the device file /dev/mem. On recent Linux systems, however, /dev/mem provides access only to a restricted range of addresses, rather than the full physical memory of a system. On ...


5

When an server process starts it issues some system calls (socket() and listen()). The system then opens the port and creates a socket file descriptor for the process to interact with. You can see this with: Find the Apache master process id: root@frisbee:~# ps -ef | grep apache | grep root root 27440 1 0 16:06 ? 00:00:00 /usr/sbin/apache2 ...


5

Should be safe but can't guarentee it. From kernel docs: min_free_kbytes: This is used to force the Linux VM to keep a minimum number of kilobytes free. The VM uses this number to compute a watermark[WMARK_MIN] value for each lowmem zone in the system. Each lowmem zone gets a number of reserved free pages based proportionally on its size. Some minimal ...


5

Active: Memory currently being used by a process Inactive: Memory that has been freed but is still cached since it may be used again. If more Free memory is required, this memory can be cleared and become free. This memory is not cleared before it is needed, because "free memory is wasted memory", it doesn't cost anything to keep the old data around in case ...


5

You may try loading it as a binary. I've had luck with that for really big, non-text files vim -b HUGEFILE It's also possible IIRC to use vim as a hex editor see: http://usevim.com/2012/06/20/vim-binary-files/


5

So, I did this thing in testing and, yeah, it consumes a lot of memory. I pointedly used a smaller number as well. I can imagine that bash hogging those resources for days on end could be a little irritating. ps -Fp "$$"; : {1..10000000}; ps -Fp "$$" UID PID PPID C SZ RSS PSR STIME TTY TIME CMD mikeserv 32601 4241 0 3957 3756 4 ...


4

It says right there in the article: This has no effect on Linux. man setrlimit says it used to work only in ancient versions. The setrlimit man page says: RLIMIT_RSS Specifies the limit (in pages) of the process's resident set (the number of virtual pages resident in RAM). This limit has effect only in Linux 2.4.x, x < ...


4

Hopefully your problem is more to do with VIMs need for temporary files (such as swap) more than RAM. In many cases, the temporary files created by VIM are in the same directory of the file you are opening. If this is the case for you, then you can verify by checking the available disk space in the current directory. Fortunately, there is good ...



Only top voted, non community-wiki answers of a minimum length are eligible