Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
I have a 100MB MySQL database backup file and I have trouble to open it in Vim on my Linux box that has 16G of RAM.
Vim just hangs (at least unusable). This is something I don't understand. I have 16 GB RAM, why can't I load a 100 MB file in an editor?
Is it because of Vim? I thought all the memory management is handled by the OS.
Vim sometimes has trouble with files that have unusually long lines. It's a text editor, so it's designed for text files, with line lengths that are usually at most a few hundred characters wide.
A database file may not contain many newline characters, so it could conceivably be one single 100 Mb long line. Vim will not be happy with that, and although it will probably work, it might take quite a long time to load the file.
I have certainly opened text files much larger than 100 Mb with Vim. The file doesn't even need to fit in memory all at once (since Vim can swap changes to disk as needed).
set synmaxcol=120 (or some other appropriate number). I've noticed huge speedups from this in the past.
In my experience Vim chokes not on large files, but on long lines. Use this command to have mysqldump use shorter lines at the expense of a larger file:
$ mysqldump --complete-insert -u -p
Additionally, you can open Vim and ask it not to parse your .vimrc file or load any plugins with this command:
$ vim -u NONE output.sql
Loading Vim in this manner will use less memory and not require Vim to parse the entire file as many plugins do.
"load VIM without .vimrc and plugins (clean VIM) e.g. for HUGE files
gvim -u NONE -U NONE -N largefile.sql
Try using less instead of vim if you want to view a large file directly. Vim tries to do a lot of different stuff when it first loads - scanning the file (potentially in multiple passes) to try to determine what syntax to use, and performing syntax highlighting, and searching for modelines at the top and bottom of the file. Then as you edit the file, vim is saving swapfiles and keeping undo trees (undo history in vim is branching, not linear like in every (?) other editor), and constantly re-evaluating the syntax highlighting as the text changes, etc.
None of that is necessarily a justification for why it must be so unusable with giant files, but it's more of an explanation of some of the reasons why it is.
Vim does not just load the file as-is into memory. It converts it into internal structures (lines, words, etc), performs syntax highlighting using an internal script language, and so on; all of which consumes memory (a whole lot more than a byte for a character) and CPU time.
You may try loading it as a binary. I've had luck with that for really big, non-text files
vim -b HUGEFILE
It's also possible IIRC to use vim as a hex editor see: http://usevim.com/2012/06/20/vim-binary-files/
Hopefully your problem is more to do with VIMs need for temporary files (such as swap) more than RAM.
In many cases, the temporary files created by VIM are in the same directory of the file you are opening. If this is the case for you, then you can verify by checking the available disk space in the current directory.
Fortunately, there is good documentation on how you can specify a different location for VIM's indexing/swap files:
You can also disable the swap file
I occasionally open large database backups in .sql text format. Very large files, or files with very long lines often sometimes seem to take a long time to open in vim. This might be related to syntax processing and colour highlighting, as mentioned in answers by @zzapper and @demonkoryu.
A quick workaround might be to press "control-G" during loading of the file to cancel the syntax highlighting pre-processing.
hexer.