2

I found myself asking this question while doing small experiments and reading upon how memory is managed in Linux.

I have a 64-bit Centos 6.5 machine with 512M RAM and 1G SWAP. I created a 1GB file '/mytest' (using dd) and opened the file with vi. Once the file was completely opened, I opened another terminal to look at the memory usage. As expected, the vi showed up at the top when the top output is ordered by memory usage. So I checked free -m expecting to see the swap usage going up, but to my surprise, the SWAP usage is NIL. Since the size of the file opened is twice the size of RAM, where is the file being held in memory?

[root@server ~]# ps aux | grep mytest
root     18940  8.5 53.3 287880 267156 pts/0   S+   09:25   0:28 vi /mytest
root     18976  0.0  0.0    364   144 pts/1    R+   09:31   0:00 grep mytest

[root@server ~]# ll -h /mytest
-rw-r--r-- 1 root root 1000M Oct 24 09:25 /mytest

[root@server ~]# free -m
             total       used       free     shared    buffers     cached
Mem:           489        481          7          0          2        140
-/+ buffers/cache:        337        151
Swap:         1023          0       1023

Swappiness is turned ON and is at the default value 60. And in the top output, memory usage under VIRT and RES is not adding upto around the 1G as I was expecting.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
18940 root      20   0  281m 260m 1172 S  0.0 53.3   0:28.85 vi /mytest

I even scrolled through the opened 1G file to see if the swap usage is going up only when I am scrolling (used sleep in the other terminal), but still the swap usage was nil. It now looks like my whole understanding of how memory and swap works is incorrect. Can someone throw light to how this huge file is held up in the memory when the size of the file is twice the memory size?

5
  • 4
    What makes you think that vi has to hold the whole file in memory? :)
    – PM 2Ring
    Oct 24, 2014 at 11:36
  • Are you only concerned about how vi does this, or what all the ways of doing this are? There are a few different ways a file can be opened/modified without loading the entire thing.
    – phemmer
    Oct 24, 2014 at 15:50
  • I am not concerned about vi. I know of other methods to view the contents if the filesize is huge. I was doing various tests to learn how linux manages memory and had this question.
    – Sreeraj
    Oct 24, 2014 at 16:19
  • Wait, what? You're not concerned about vi, and you already know how to do this. So then what are you asking? P.S. when responding to someone, mention their name such as @Patrick so they are notified.
    – phemmer
    Oct 29, 2014 at 16:15
  • @Patrick, thank you, I wasn't aware of tagging usernames. And yes, I am not concerned about opening files in 'vi'. As mentioned in the beginning of my post, I was just doing experiments to learn how memory is managed in Linux.
    – Sreeraj
    Oct 29, 2014 at 16:34

1 Answer 1

1

Most likely you need to use a "dumber" editor to use the RAM and I think it's VI that does "swapping" for you. Normally Vi uses a file to hangle its buffer.

You may want to look into other answers on this site. Eg.: What happens if I use vi on large files?

3
  • I had always thought that 'vi' always loads the whole file into memory. I did a 'pmap' and saw it is actually using only much less amount of memory and the value is coinciding with the output from top. I was wrong all the way. Gotta read a lot more. Thanks Alex.
    – Sreeraj
    Oct 24, 2014 at 13:04
  • 1
    @Sree There are several vi implementations. Some load the whole file into memory, others don't. Oct 25, 2014 at 14:18
  • @Gilles: Thanks for the comment. You're right, there are many vi implementations. I assumed Sree is using VIM variant since that would be defacto standard on most linux distros. Eg. traditional VI woundn't even load the file in the test due to the line length limitation. ;)
    – Alex
    Oct 25, 2014 at 22:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.