Tag Info

Hot answers tagged

78

You can write a C program to malloc the required memory and then use mlock() to prevent the memory from being swapped out. Then just let the program wait for keyboard input, and unlock the memory, free the memory and exit.


56

Vim sometimes has trouble with files that have unusually long lines. It's a text editor, so it's designed for text files, with line lengths that are usually at most a few hundred characters wide. A database file may not contain many newline characters, so it could conceivably be one single 100 Mb long line. Vim will not be happy with that, and although it ...


46

In my experience Vim chokes not on large files, but on long lines. Use this command to have mysqldump use shorter lines at the expense of a larger file: $ mysqldump --complete-insert -u -p Additionally, you can open Vim and ask it not to parse your .vimrc file or load any plugins with this command: $ vim -u NONE output.sql Loading Vim in this manner ...


42

stress is a workload generator that simulates cpu/mem/io/hdd stress on POSIX systems. This call should do the trick on Linux: stress --vm-bytes $(awk '/MemFree/{printf "%d\n", $2 * 0.9;}' < /proc/meminfo)k --vm-keep -m 1 Adapt the /proc/meminfo call with free(1)/vm_stat(1)/etc. if you need it portable.


42

I would suggest running a VM with limited memory and testing the software in that would be a more efficient test than trying to fill memory on the host machine. That method also has the advantage that if the low memory situation causes OOM errors elsewhere and hangs the whole OS, you only hang the VM you are testing in not your machine that you might have ...


34

It can. There are 2 different out of memory conditions you can encounter in linux. Which you encounter depends on the value of sysctl vm.overcommit_memory (/proc/sys/vm/overcommit_memory) Introduction: The kernel can perform what is called 'memory overcommit'. This is when the kernel allocates programs more memory than is really present in the system. This ...


26

run linux; boot with mem=nn[KMG] kernel boot parameter (look in linux/Documentation/kernel-parameters.txt for details).


21

From this HN comment: https://news.ycombinator.com/item?id=6695581 Just fill /dev/shm via dd or similar. swapoff -a dd if=/dev/zero of=/dev/shm/fill bs=1k count=1024k


17

NOTE: I'm going to assume that your machine has a memory mapping unit (MMU). There is a Linux version (µClinux) that doesn't require an MMU, and this answer doesn't apply there. What is an MMU? It's hardware—part of the processor and/or memory controller. Understanding shared library linking doesn't require you to understand exactly how an MMU works, just ...


16

You can also use htop. It's much cooler than top. If you are using Debian or one of its derivatives, then you can install it using sudo apt-get install htop.


14

Red Hat Enterprise Linux (RHEL) These are probably a good basis, looking at RHEL6's capabilities, they're covered here, titled: Red Hat Enterprise Linux 6 technology capabilities and limits.     NOTE: [5] The architectural limits are based on the capabilities of the Red Hat Enterprise Linux kernel and the physical hardware. Red Hat ...


13

There is a command-line option which does that: -M : Detect memory units Show memory units (k/M/G) and display floating point values in the memory summary. So it is sufficient to run top like that: top -M


12

top -M doesn't work on any of the Fedora, Debian or Ubuntu distros to my knowledge. I just tried it and it's not in the procps-ng package that provides top. There are many implementations of top so one needs to pay special attention to which they use. In general it's best to use free with switching to get the amount of memory free on Linux. procps vs. ...


12

Use cron (or anacron). Cron is designed for running things at intervals. That is the only thing it does, and there has been a lot of work put into cron for many years to make it what it is today. The chances that you're going to write a better scheduler in your script are effectively nil. Using cron will work better, avoid having unnecessary code in your ...


12

Try using less instead of vim if you want to view a large file directly. Vim tries to do a lot of different stuff when it first loads - scanning the file (potentially in multiple passes) to try to determine what syntax to use, and performing syntax highlighting, and searching for modelines at the top and bottom of the file. Then as you edit the file, vim ...


12

This can be do the same thing with purge: sync && echo 3 > /proc/sys/vm/drop_caches From man proc: /proc/sys/vm/drop_caches (since Linux 2.6.16) Writing to this file causes the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free. To free ...


11

Let's consider how each solution works. uniq This requires that the file already be sorted. If not, you have to pipe it through sort first, which means that sort has to read the entire file into memory, reorder it (O(n log n)), and then write it into the pipe. The work of uniq is very cheap, since it only has to compare adjacent lines of its input. sort -u ...


11

The truth is that regardless of which way you look at it - whether your process choked up due to the system's memory manager or due to something else - it is still a bug. What happened to all of that data you were just processing in memory? It should have been saved. While overcommit_memory= is the most general way of configuring Linux OOM management, it is ...


11

"load VIM without .vimrc and plugins (clean VIM) e.g. for HUGE files gvim -u NONE -U NONE -N largefile.sql


10

How about ramfs if it exists? Mount it and copy over a large file? If there's no /dev/shm and no ramfs - I guess a tiny C program that does a large malloc based on some input value? Might have to run it a few times at once on a 32 bit system with a lot of memory.


9

Vim does not just load the file as-is into memory. It converts it into internal structures (lines, words, etc), performs syntax highlighting using an internal script language, and so on; all of which consumes memory (a whole lot more than a byte for a character) and CPU time.


8

The text segment is the mapping at 0x400000 - it's marked 'r-x' for readable and executable. The mapping at 0x600000 is read-only, so that's almost certainly the ".rodata" section of the executable file. GCC puts C string literals into a read-only section. The mapping at 0x601000 is 'rw-', so that's probably the famed heap. You could have your executable ...


7

I keep a function to do something similar in my dotfiles. https://github.com/sagotsky/.dotfiles/blob/master/.functions#L248 function malloc() { if [[ $# -eq 0 || $1 -eq '-h' || $1 -lt 0 ]] ; then echo -e "usage: malloc N\n\nAllocate N mb, wait, then release it." else N=$(free -m | grep Mem: | awk '{print int($2/10)}') if [[ $N -gt $1 ]] ...


7

If you want to test a particular process with limited memory you might be better off using ulimit to restrict the amount of allocatable memory.


7

The difference you are observing isn't actually due to swap space being unaccounted for. The "(deleted)" that the kernel sometimes appends to /proc/*/exe links is output by readlink and is causing parse errors in your awk script, and you are effectively not counting processes whose binaries are no longer present in your total. Some kernels append the word ...


7

It depends on the settings you're running with, in particular memory overcommit (/proc/sys/vm/overcommit_memory; see man 5 proc for details). If memory overcommit is disabled, the editor's (and possibly other programs attempting at the same time) attempt to allocate memory will fail. They'll get a failure result from the system call. Its up to each program ...


6

The memory represented by "buffers/cache" in free is your filesystem cache, which Linux caches to speed up reading data from your disk, as hitting the disk is generally a fairly slow way to access data repeatedly. As such, they are cached in memory, and transparently served from there if available. You can see which blocks are currently in your cache by ...


6

There is no difference betweem tmpfs and shm. tmpfs is the new name for shm. shm stands for SHaredMemory. See: Linux tmpfs. The main reason tmpfs is even used today is this comment in my /etc/fstab on my gentoo box. BTW Chromium won't build with the line missing: # glibc 2.2 and above expects tmpfs to be mounted at /dev/shm for # POSIX shared memory ...


6

It appears that the stack memory limit is not allocated (anyway, it couldn't with unlimited stack). https://www.kernel.org/doc/Documentation/vm/overcommit-accounting says: The C language stack growth does an implicit mremap. If you want absolute guarantees and run close to the edge you MUST mmap your stack for the largest size you think you will ...


5

Thanks to @Mat: # mkdir -p /mnt/ram # mount -t ramfs -o size=20m ramfs /mnt/ram



Only top voted, non community-wiki answers of a minimum length are eligible