Tag Info

Hot answers tagged

56

Vim sometimes has trouble with files that have unusually long lines. It's a text editor, so it's designed for text files, with line lengths that are usually at most a few hundred characters wide. A database file may not contain many newline characters, so it could conceivably be one single 100 Mb long line. Vim will not be happy with that, and although it ...


46

In my experience Vim chokes not on large files, but on long lines. Use this command to have mysqldump use shorter lines at the expense of a larger file: $ mysqldump --complete-insert -u -p Additionally, you can open Vim and ask it not to parse your .vimrc file or load any plugins with this command: $ vim -u NONE output.sql Loading Vim in this manner ...


34

It can. There are 2 different out of memory conditions you can encounter in linux. Which you encounter depends on the value of sysctl vm.overcommit_memory (/proc/sys/vm/overcommit_memory) Introduction: The kernel can perform what is called 'memory overcommit'. This is when the kernel allocates programs more memory than is really present in the system. This ...


18

NOTE: I'm going to assume that your machine has a memory mapping unit (MMU). There is a Linux version (µClinux) that doesn't require an MMU, and this answer doesn't apply there. What is an MMU? It's hardware—part of the processor and/or memory controller. Understanding shared library linking doesn't require you to understand exactly how an MMU works, just ...


17

You can also use htop. It's much cooler than top. If you are using Debian or one of its derivatives, then you can install it using sudo apt-get install htop.


15

There is a command-line option which does that: -M : Detect memory units Show memory units (k/M/G) and display floating point values in the memory summary. So it is sufficient to run top like that: top -M


14

Red Hat Enterprise Linux (RHEL) These are probably a good basis, looking at RHEL6's capabilities, they're covered here, titled: Red Hat Enterprise Linux 6 technology capabilities and limits.     NOTE: [5] The architectural limits are based on the capabilities of the Red Hat Enterprise Linux kernel and the physical hardware. Red Hat ...


14

top -M doesn't work on any of the Fedora, Debian or Ubuntu distros to my knowledge. I just tried it and it's not in the procps-ng package that provides top. There are many implementations of top so one needs to pay special attention to which they use. In general it's best to use free with switching to get the amount of memory free on Linux. procps vs. ...


12

Use cron (or anacron). Cron is designed for running things at intervals. That is the only thing it does, and there has been a lot of work put into cron for many years to make it what it is today. The chances that you're going to write a better scheduler in your script are effectively nil. Using cron will work better, avoid having unnecessary code in your ...


12

Try using less instead of vim if you want to view a large file directly. Vim tries to do a lot of different stuff when it first loads - scanning the file (potentially in multiple passes) to try to determine what syntax to use, and performing syntax highlighting, and searching for modelines at the top and bottom of the file. Then as you edit the file, vim ...


12

This can be do the same thing with purge: sync && echo 3 > /proc/sys/vm/drop_caches From man proc: /proc/sys/vm/drop_caches (since Linux 2.6.16) Writing to this file causes the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free. To free ...


11

Let's consider how each solution works. uniq This requires that the file already be sorted. If not, you have to pipe it through sort first, which means that sort has to read the entire file into memory, reorder it (O(n log n)), and then write it into the pipe. The work of uniq is very cheap, since it only has to compare adjacent lines of its input. sort -u ...


11

The truth is that regardless of which way you look at it - whether your process choked up due to the system's memory manager or due to something else - it is still a bug. What happened to all of that data you were just processing in memory? It should have been saved. While overcommit_memory= is the most general way of configuring Linux OOM management, it is ...


11

"load VIM without .vimrc and plugins (clean VIM) e.g. for HUGE files gvim -u NONE -U NONE -N largefile.sql


9

Vim does not just load the file as-is into memory. It converts it into internal structures (lines, words, etc), performs syntax highlighting using an internal script language, and so on; all of which consumes memory (a whole lot more than a byte for a character) and CPU time.


8

When in top, typing "E" cycles through different memory units (kb, mb, gb etc) in the total memory info. "e" does the same individual process lines.


8

Use quick tip using top command in linux/unix $top hit Shift + f , then choose the display to order by memory usage by hitting key n then press Enter. You will see active process ordered by memory usage. Or you can just press M after running top command. References ...


7

There is no difference betweem tmpfs and shm. tmpfs is the new name for shm. shm stands for SHaredMemory. See: Linux tmpfs. The main reason tmpfs is even used today is this comment in my /etc/fstab on my gentoo box. BTW Chromium won't build with the line missing: # glibc 2.2 and above expects tmpfs to be mounted at /dev/shm for # POSIX shared memory ...


7

It depends on the settings you're running with, in particular memory overcommit (/proc/sys/vm/overcommit_memory; see man 5 proc for details). If memory overcommit is disabled, the editor's (and possibly other programs attempting at the same time) attempt to allocate memory will fail. They'll get a failure result from the system call. Its up to each program ...


6

The memory represented by "buffers/cache" in free is your filesystem cache, which Linux caches to speed up reading data from your disk, as hitting the disk is generally a fairly slow way to access data repeatedly. As such, they are cached in memory, and transparently served from there if available. You can see which blocks are currently in your cache by ...


6

It appears that the stack memory limit is not allocated (anyway, it couldn't with unlimited stack). https://www.kernel.org/doc/Documentation/vm/overcommit-accounting says: The C language stack growth does an implicit mremap. If you want absolute guarantees and run close to the edge you MUST mmap your stack for the largest size you think you will ...


5

Another way to limit this is to use Linux's control groups. This is especially useful if you want to limit a process's (or group of processes') allocation of physical memory distinctly from virtual memory. For example: $ cgcreate -g memory:/myGroup $ echo $(( 500 * 1024 * 1024 )) > /sys/fs/cgroup/memory/myGroup/memory.limit_in_bytes $ echo $(( 5000 ...


5

You can't load a kernel module at a specific physical address. You can't load a kernel module at a specific virtual address. The kernel decides where it loads the module. Inside the kernel, of course, you can do what you want. But I think arranging to load a driver at a specific address would require a lot of deep changes. I fail to see what would require ...


5

1. Virtual memory The system will ensure that processes will get the requested amount of memory despite being greater than physical memory. By this way the kernel allocates a virtual memory space of the maximum physical memory size it can handle. E.g. on a 32bit machine, the kernel will allocate a total of 2^32 i.e. 4GB of virtual addresses to every process ...


5

Use cron because it is a better and more standard practice. At least if this is something that will regularly run (not just something you patched together in a minute). cron is a cleaner and more standard way. It's also better because it runs the shell detached from a terminal - no problem with accidental termination and dependencies on other processes. ...


5

From the Forensic's Wiki: Tools:Memory Imaging excerpt Linux /dev/mem On older Linux systems, the program dd can be used to read the contents of physical memory from the device file /dev/mem. On recent Linux systems, however, /dev/mem provides access only to a restricted range of addresses, rather than the full physical memory of a system. On ...


5

When an server process starts it issues some system calls (socket() and listen()). The system then opens the port and creates a socket file descriptor for the process to interact with. You can see this with: Find the Apache master process id: root@frisbee:~# ps -ef | grep apache | grep root root 27440 1 0 16:06 ? 00:00:00 /usr/sbin/apache2 ...


5

You may try loading it as a binary. I've had luck with that for really big, non-text files vim -b HUGEFILE It's also possible IIRC to use vim as a hex editor see: http://usevim.com/2012/06/20/vim-binary-files/


5

If your memory is exhaustively used up by the processes to the extent which can possibly threaten the stability of the system, then the OOM killer comes into picture. It is the task of the OOM Killer to kill the processes until enough memory is freed for the smooth functioning of the rest of the process. The OOM Killer has to select the best process to ...


5

So, I did this thing in testing and, yeah, it consumes a lot of memory. I pointedly used a smaller number as well. I can imagine that bash hogging those resources for days on end could be a little irritating. ps -Fp "$$"; : {1..10000000}; ps -Fp "$$" UID PID PPID C SZ RSS PSR STIME TTY TIME CMD mikeserv 32601 4241 0 3957 3756 4 ...



Only top voted, non community-wiki answers of a minimum length are eligible