31

This question already has an answer here:

Let's say I'm running a script (e.g. in Python). In order to find out how long the program took, one would run

time python script1.py

Is there a command which keeps track of how much RAM was used as the script was running? In order to find how much RAM is available, one could use free, but this command doesn't fit the task above.

marked as duplicate by Thomas Dickey, Stephen Rauch, Romeo Ninov, jimmij, don_crissti Jul 8 '17 at 18:17

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 2
  • 2
    Also related, if not a duplicate: unix.stackexchange.com/q/164653/85039 – Sergiy Kolodyazhnyy Jul 7 '17 at 8:33
  • Why exactly do you ask? Why do you care more about physical RAM than about virtual address space consumed by your python process? – Basile Starynkevitch Jul 8 '17 at 6:34
  • @BasileStarynkevitch There's nothing in the question that indicates that the user only wants to know about physical RAM. – pipe Jul 8 '17 at 7:18
  • RAM is only physical. It is a concrete device. Virtual memory (or just memory) is an abstraction. There is no such thing as "non-physical" RAM, and we generally don't care about RAM, we care about (virtual) memory. OP could have asked "how much memory was used" – Basile Starynkevitch Jul 8 '17 at 7:21
46

The time(1) command (you may need to install it -perhaps as the time package-, it should be in /usr/bin/time) accepts many arguments, including a format string (with -f or --format) which understands (among others)

  %M     Maximum resident set size of the process during its lifetime,
          in Kbytes.

  %K     Average total (data+stack+text) memory use of the process, in
          Kbytes.

Don't confuse the /usr/bin/time command with the time bash builtin. You may need to type the full file path /usr/bin/time (to ask your shell to run the command not the builtin) or type command time or \time (thanks to Toby Speight & to Arrow for their comments).

So you might try (RSS being the resident set size)

/usr/bin/time -f "mem=%K RSS=%M elapsed=%E cpu.sys=%S .user=%U" python script1.py

You could also try

/usr/bin/time --verbose  python script1.py

You are asking:

how much RAM was used as the script was running?

and this shows a misconception from your part. Application programs running on Linux (or any modern multi-process operating system) are using virtual memory, and each process (including the python process running your script) has its own virtual address space. A process don't run directly in physical RAM, but has its own virtual address space (and runs in it), and the kernel implements virtual memory by sophisticated demand-paging using lazy copy-on-write techniques and configures the MMU. The RAM is a physical device and resource used -and managed internally by the kernel- to implement virtual memory (read also about the page cache and about thrashing).

You may want to spend several days understanding more about operating systems. I recommend reading Operating Systems : Three Easy Pieces which is a freely downloadable book. The RAM is used by the entire operating system (not -directly- by individual processes) and the actual pages in RAM for a given process can vary during time (and could be somehow shared with other processes). Hence the RAM consumption of a given process is not well defined since it is constantly changing (you may want its average, or its peak value, etc...), and likewise for the size of its virtual address space.

You could also use (especially if your script runs for several seconds) the top(1) utility (perhaps in some other terminal), or ps(1) or pmap(1) -maybe using watch(1) to repeat that ps or pmap command. You could even use directly /proc/ (see proc(5)...) perhaps as watch cat /proc/$(pidof python)/status or /proc/$(pidof python)/stat or /proc/$(pidof python)/maps etc...

But RAM usage (by the kernel for some process) is widely varying with time for a given process (and even its virtual address space is changing, e.g. by calls to mmap(2) and munmap used by ld-linux(8), dlopen(3), malloc(3) & free and many other functions needed to your Python interpreter...).

You could also use strace(1) to understand the system calls done by Python for your script (so you would understand how it uses mmap & munmap and other syscalls(2)). You might restrict strace with -e trace=%memory or -e trace=memory to get only memory (i.e. virtual address space) related system calls.

BTW, the tracemalloc Python feature could be also useful.

I guess that you only care about virtual memory, that is about virtual address space (but not about RAM), used by the Python interpreter to run your Python script. And that is changing during execution of the process. The RSS (or the maximal peak size of the virtual address space) could actually be more useful to know.

See also LinuxAteMyRAM.

  • 1
    You expect me to spend days on reading a book with probably 99% already known to me to find the interpretation of a term? A simple "yes" or now" could have save me a lot of time. )-: – Philippos Jul 7 '17 at 7:38
  • 2
    But the notion of RSS (related to that of virtual memory) is not simple to explain. Perhaps the wikipage could be enough to you. My point is that your naive question (how much RAM does a python script spends) has no precise meaning, because processes don't use RAM (but virtual address space, which is not the same). What exactly happens requires indeed at least a whole book (and perhaps more) to be explained. – Basile Starynkevitch Jul 7 '17 at 7:48
  • 7
    I'm not interested in your style of discussion if you prefer to become personal when you can't distinguish between concepts and terms. – Philippos Jul 7 '17 at 8:08
  • 4
    You don't need the full path - command time will tell Bash to ignore the builtin. That's useful if you can't remember which bin directory holds it. – Toby Speight Jul 7 '17 at 8:51
  • 4
    @TobySpeight Or simpler: \time will tell a shell to search in the PATH. A backslash will not match exactly the time builtin name, so a search will be initiated in the PATH. – Arrow Jul 7 '17 at 14:09
16

You can also use the legendary valgrind, although you may need to install it from your package manager.

$ valgrind c-projects/objtest/bin/objtest 
==6543== Memcheck, a memory error detector
==6543== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==6543== Using Valgrind-3.12.0.SVN and LibVEX; rerun with -h for copyright info
==6543== Command: c-projects/objtest/bin/objtest
==6543== 
|ERROR|array:189: array_delete: delete index 0 but the highest is -1 (delete from empty array): index out of bounds
==6543== 
==6543== HEAP SUMMARY:
==6543==     in use at exit: 480 bytes in 20 blocks
==6543==   total heap usage: 7,390 allocs, 7,370 frees, 256,217 bytes allocated
==6543== 
==6543== LEAK SUMMARY:
==6543==    definitely lost: 96 bytes in 4 blocks
==6543==    indirectly lost: 384 bytes in 16 blocks
==6543==      possibly lost: 0 bytes in 0 blocks
==6543==    still reachable: 0 bytes in 0 blocks
==6543==         suppressed: 0 bytes in 0 blocks
==6543== Rerun with --leak-check=full to see details of leaked memory
==6543== 
==6543== For counts of detected and suppressed errors, rerun with: -v
==6543== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If you are a power user and want fancy graphs, you can use ms_print and a command like:

valgrind --tool=massif --pages-as-heap=yes --massif-out-file=massif.out ./test.sh; grep mem_heap_B massif.out | sed -e 's/mem_heap_B=\(.*\)/\1/' | sort -g | tail -n 1

see Peak memory usage of a linux/unix process.

  • 2
    This is a good suggestion - it may be worth pointing out that you can't tell the difference between a large amount of memory allocated for a long time and many smaller short-term allocations. It's hard to know whether the questioner is also interested in other memory usage (stack, mmapped files, text segments etc). – Toby Speight Jul 7 '17 at 8:55
  • @TobySpeight If you look at the linked answer, you can actually get histogram output of much more specific data – cat Jul 7 '17 at 15:27
  • Consider editing to add that comment about the link, so that others know why they might want to follow it. I haven't used Massif very much, but did think it was probably a better choice than Memcheck here - thanks for confirming that! – Toby Speight Jul 10 '17 at 8:07
2

You can use pmap command to view the amount of memory used by a process. In your case you need to give the PID of the script as input to the pmap command like

pmap $(ps -ef | grep **<SCRIPT NAME>** | grep -v grep | awk '{print $2}')

Not the answer you're looking for? Browse other questions tagged or ask your own question.