Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a 100MB MySQL database backup file and I have trouble to open it in Vim on my Linux box that has 16G of RAM.

Vim just hangs (at least unusable). This is something I don't understand. I have 16 GB RAM, why can't I load a 100 MB file in an editor?

Is it because of Vim? I thought all the memory management is handled by the OS.

share|improve this question
3  
Consider using a HEX editor instead of a text editor to view such files. An example of hex editor with vi-like interface would be hexer. –  Ruslan Jun 26 at 6:19
10  
Don't forget that RAM has not been what we run out of when we run out of memory for decades now. Memory is now virtualized; it is divided up into pages and those pages can be swapped to disk. The amount of memory allocated out of a process's address space and the amount of RAM consumed have very little to do with each other. When you run out of memory, you've run out of address space, not RAM. The best way to think of it is memory is disk space, each process gets a certain fixed amount of that space, and RAM is hardware that makes your disk faster. –  Eric Lippert Jun 26 at 13:43
19  
@EricLippert Except that traditional disks are so slow (compared to RAM) that they are only suitable for storing the virtual memory pages that are not in active use. If a process hangs (or is at least unusable, as the OP put it) due to swap thrashing, it is precisely because RAM is what it has run out of. –  depquid Jun 26 at 15:08
6  
@EricLippert running out of address space is only relevant on 32 bit systems today. I doubt that the user with 16G RAM will still use 32 bit PAE kernel instead of normal 64 bit one. –  Ruslan Jun 26 at 18:06
2  
@depquid: That is a good point; the thrust of my comment is that the OP seems to have the belief that "I loaded 100MB of stuff, I have 16000MB of RAM, so therefore 100MB of my 16000MB of RAM was consumed". This belief system is outdated. –  Eric Lippert Jun 26 at 18:31

9 Answers 9

up vote 55 down vote accepted

Vim sometimes has trouble with files that have unusually long lines. It's a text editor, so it's designed for text files, with line lengths that are usually at most a few hundred characters wide.

A database file may not contain many newline characters, so it could conceivably be one single 100 Mb long line. Vim will not be happy with that, and although it will probably work, it might take quite a long time to load the file.

I have certainly opened text files much larger than 100 Mb with Vim. The file doesn't even need to fit in memory all at once (since Vim can swap changes to disk as needed).

share|improve this answer
1  
I also noticed the very long lines, tried with another file without very long lines, see a big improvement. Thanks –  Ask and Learn Jun 26 at 2:15
10  
@AskandLearn Depending on the file type, you may see a performance increase if you set synmaxcol=120 (or some other appropriate number). I've noticed huge speedups from this in the past. –  sapi Jun 26 at 7:30
    
Does anyone know if the recent neovim fork will handle longer lines better? I guess it isn't a particularly common issue... –  Hemmer Jun 30 at 12:39
    
@GregHewgill it's true, I've also observe that, but how did you know that ? –  Rahul Patil Jul 2 at 5:37

In my experience Vim chokes not on large files, but on long lines. Use this command to have mysqldump use shorter lines at the expense of a larger file:

$ mysqldump --complete-insert -u -p

Additionally, you can open Vim and ask it not to parse your .vimrc file or load any plugins with this command:

$ vim -u NONE output.sql

Loading Vim in this manner will use less memory and not require Vim to parse the entire file as many plugins do.

share|improve this answer

Try using less instead of vim if you want to view a large file directly. Vim tries to do a lot of different stuff when it first loads - scanning the file (potentially in multiple passes) to try to determine what syntax to use, and performing syntax highlighting, and searching for modelines at the top and bottom of the file. Then as you edit the file, vim is saving swapfiles and keeping undo trees (undo history in vim is branching, not linear like in every (?) other editor), and constantly re-evaluating the syntax highlighting as the text changes, etc.

None of that is necessarily a justification for why it must be so unusable with giant files, but it's more of an explanation of some of the reasons why it is.

share|improve this answer
    
See my answer for how to prevent VIM from performing the heavy operations such as file parsing. –  dotancohen Jun 26 at 8:52
    
Yup, syntax highlighting on things like XML and SQL can get very slow on larger files. –  Marcin Jun 26 at 15:02

"load VIM without .vimrc and plugins (clean VIM) e.g. for HUGE files

  gvim -u NONE -U NONE -N largefile.sql
share|improve this answer

Vim does not just load the file as-is into memory. It converts it into internal structures (lines, words, etc), performs syntax highlighting using an internal script language, and so on; all of which consumes memory (a whole lot more than a byte for a character) and CPU time.

share|improve this answer
    
Consuming memory is not even the issue. CPU time being taken up (and a visible freeze whilst you wait), is. –  Lightness Races in Orbit Jun 29 at 21:07
    
That CPU time is mostly taken up by the syntax highlighting script. –  demonkoryu Jul 1 at 8:45
    
Yes, I agree. I'm just saying that memory usage is very unlikely to (a) be a problem, or (b) cause a long delay, contrary to what your answer says. –  Lightness Races in Orbit Jul 1 at 8:54
    
You're right, I've updated my answer accordingly. –  demonkoryu Jul 2 at 8:19

You may try loading it as a binary. I've had luck with that for really big, non-text files

vim -b HUGEFILE

It's also possible IIRC to use vim as a hex editor see: http://usevim.com/2012/06/20/vim-binary-files/

share|improve this answer

Hopefully your problem is more to do with VIMs need for temporary files (such as swap) more than RAM.

In many cases, the temporary files created by VIM are in the same directory of the file you are opening. If this is the case for you, then you can verify by checking the available disk space in the current directory.

Fortunately, there is good documentation on how you can specify a different location for VIM's indexing/swap files:

You can also disable the swap file

share|improve this answer

I'm not entirely sure what form your database is in, but there is a decent chance the file is in binary or some other format which allows a dbms to handle it quickly and securely, which vim wasn't designed to open.

I'm still not sure what you are trying to do, but I would recommend using mysql or another dbms to make your changes. They will maintain the integrity of your database.

If you really want to dump the data in your database, you can use mysqldump to get the information out in an encoding that vim can use.

share|improve this answer
    
It is a text file, I am just use it as an example. Same problem if I am trying to open a huge text file. –  Ask and Learn Jun 26 at 1:45
    
That is a better question then. One, don't open a database outside of a dbms, you are asking for trouble. Two 100Mb is large, but not that large. I just generated a words list that is 102Mb and vim opened it in half a second. Try using less, try cutting down on operations in your vimrc, if that doesn't work maybe try re-installing vim because it should be able to open it. –  Evan Jun 26 at 1:53

I occasionally open large database backups in .sql text format. Very large files, or files with very long lines often sometimes seem to take a long time to open in vim. This might be related to syntax processing and colour highlighting, as mentioned in answers by @zzapper and @demonkoryu.

A quick workaround might be to press "control-G" during loading of the file to cancel the syntax highlighting pre-processing.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.