Hot answers tagged

28

Additional information provided in the comments reveals that the OP is using a GUI method to create the .tar.gz file. GUI software often includes a lot more bloat than the equivalent command line equivalent software, or performs additional unnecessary tasks for the sake of some "extra" feature such as a progress bar. It wouldn't surprise me if the GUI ...


13

"five million" files, and 1TB in total? Your files must be very small, then. I'd simply try rsync: rsync -alPEmivvz /source/dir remote.host.tld:/base/dir If you don't have that - or your use-case doesn't allow for using rsync, I'd at least check if 7z works with your data. It might not, but I think it's still worth a try: 7z a archive.7z /source/dir Or ...


11

In modern systems none of the memory is actually copied just because a fork system call is used. It is all marked read only in the page table such that on first attempt to write a trap into kernel code will happen. Only once the first process attempt to write will the copying happen. This is known as copy-on-write. However it may be necessary to keep track ...


11

This statement is odd: split("0,2,4,5,7,9,11,12",a,","); It repetitively splits a constant string to create an array a. If you move that into a BEGIN section, the program should work the same — without allocating a new copy of the a array for each input-record. Addressing comments: the for-loop and expression do not allocate memory in a simple ...


8

Unless you can do better than 25:1 compression you are unlikely to gain anything from compressing this before snail-mailing, unless you have some hardware tape format that you can exchange the the third party. The largest common storage is blue ray and that will roughly get you 40Gb. You would need 25 to 1 compression on your data to get it to fit on that. ...


8

In your second reference, where it says The memory allocator gets a big chunk of memory (say, 4 pages, or 4 * 4 KiB) and divides this into much smaller chunks "the memory allocator" is not the kernel but the libc routines like malloc. The kernel allocates memory to the process in page-sized chunks. malloc keeps track of used and unused portions of ...


5

Did you consider torrent? Peer-to-Peer might be your best option for an over-the-internet transfer: At least as fast as other internet transfers: your upload speed will determine the transfer speed No data corruption Choose which files to transfer first No extra local/cloud storage space needed Free You didn't tell which OS you were using, but as you're ...


5

You can press the following keys: e -- Change the scaling factor on the summary display Shift+e -- Change the scaling factor on the task Shift+w -- Save current settings


5

It seems like the -o flag will take the actual column name. So if the top command shows only "mem" then the command should be "top -o mem". For the ubuntu machine I am testing with, the column is called "%MEM". On the OSX Yosemite I tried, it is "mem".


5

This is not a loop, but recursion and the memory increases linear over the time, which is what you don't want. If you want a loop with constant memory usage, you can do it this way: #!/bin/sh while 1; do mysql -h "localhost" -u "root" "-p********" "database" < "update.sql" sleep 5 done


5

How can I see the raw memory data used by an application... Once you have obtained the process' PID (using ps(1) or pidof(8) for instance), you may access the data in its virtual address space using /proc/PID/maps and /proc/PID/mem. Gilles wrote a very detailled answer about that here. ... and all the files its accessing in my filesystem, network ...


5

If the system supports process accounting, and accounting is enabled, then there may be records available in the process accounting file, e.g. on RedHat Linux: $ grep -q BSD_PROCESS_ACCT=y /boot/config-* && echo hooray hooray $ sudo touch /var/log/pacct $ sudo accton /var/log/pacct $ ... $ sudo accton # turn it off Parsing the acct(5) file is ...


5

The short story: If your mobo posts, and your system boots, and free/top show your ram as 16 gB, then it works. Even mobo makers can under-report capacity of system boards, so the real test is if ram is installed correctly, matched correctly, runs, ie, boots, and runs with stability, ie, doesn't crash, then it works. You can also test by trying to use all ...


5

niceload --noswap yourprg is made for exactly that situation: It looks at swapping activity: If swapping out: Let process run If swapping in: Let process run If swapping in and out: Suspend process until swapping stops and resume the process when swapping has stopped It does not suspend the process before the swapping starts, but lets swapping run for 1 ...


4

There is kernel setting /proc/sys/vm/overcommit_memory Citation from excellent article: Since 2.5.30 the values are: 0 (default): as before: guess about how much overcommitment is reasonable, 1: never refuse any malloc(), 2: be precise about the overcommit - never commit a virtual address space larger than swap space plus a fraction overcommit_ratio ...


4

Don't worry, it makes a lazy copy (copy-on-write). The virtual memory addresses of both processes point to the same pages initially, but when the forked process tries to modify it, it actually makes a physical copy of the page (from then on, that page resides in two places in your RAM). Beware, none of the reported memory footprints actually tell you how ...


4

From what you have posted it doesn't seems like you under stand how memory works in Linux. I recommend reading http://www.linuxnix.com/find-ram-size-in-linuxunix/ http://www.itworld.com/article/2722141/it-management/making-sense-of-memory-usage-on-linux.html http://www.linuxatemyram.com/ The jist of those sites is that you have more "free" ram then ...


4

MemAvailable is included in /proc/meminfo since version 3.14 of the kernel; it was added by commit 34e431b0a. That's the determining factor in the output variations you show. The commit message indicates how to estimate available memory without MemAvailable: Currently, the amount of memory that is available for a new workload, without pushing the system ...


4

Here's a perl equivalent that doesn't leak: perl -lne 'BEGIN { @a=(0,2,4,5,7,9,11,12);} for ($i = 0; $i < 1; $i+= 0.0001) { printf("%08X\n", 100*sin(1382*exp($a[$F[0] % 8]/12)*log(2))*$i) }' It's almost identical. $1 gets replaced by $F[0] and i is replaced with $i. The hash a is replaced with an actual array, @a. You would be wise to generate ...


4

free is provided by procps-ng; Debian 8 has version 3.3.9, which uses the old style with a separate line for buffers/cache, while Gentoo and presumably RHEL 7.x have version 3.3.10 or later which uses the new style. You can see the reasoning behind the change in the corresponding commit message. If you really want the old-style output you can run an older ...


4

Processes are managed by the kernel. The kernel doesn't care how the programmer allocates variables. All it knows is that certain blocks of memory belong to the process. The C runtime matches C memory management features to kernel features: automatic variables go into a memory block called “stack” and dynamic storage (malloc and friends) go into a memory ...


3

I right clicked the folder and clicked "create archive" and selected the .tar.gz option. The directory structure is deep, over 500,000 directories Yeah, good luck getting that to package up. And the GUI tool will try to do that on the same volume, which means a) you need another 1Tb of free space and b) the head thrashing of reading one file and ...


3

7z would be my choice. It allows auto-splitting of archives and supports multi-threaded compression. No, xz doesn't, despite what the help message says. Try with: 7za a -v100m -m0=lzma2 -mx=9 -ms=on -mmt=$THREADS archive.7z directory/ The output is split in 100MB blocks (change it with the -v switch). The only real downside is that 7z does not retain ...


3

It would depend on what kind of stats you want, but if you're writing a program in C running on Linux, you'd definitely better know about Valgrind. Valgrind can, not only profile detailed memory usage of your program, but also detect memory access violations which are common in C and possibly very hard to debug. For your profiling purpose, take a look at ...


3

I figured out that a large portion of the memory usage was, in fact, attributable to inactive memory used by exited processes. The most accurate way to determine how much memory is available post-January 2014 is to look at MemAvailable in /proc/meminfo. You can also see the amount of inactive memory is this file.


3

You could - probably. From the man page: While this is a good point in terms of report speed and safeness, this also makes the presented information possibly unreliable. You'll also still need to work out exactly what type and what speed that you'll need for your laptop. Why not use http://www.crucial.com or a similar site and enter the details ...


3

I'm not sure why you want to drop the caches every hour - that is going to kill performance. There are a few problems with what you've done: You need to edit the crontab of root, as you cannot write to /proc/sys/vm/drop_caches as a non-root user. Your crontab entry is not correct. It should be: 2 * * * * /usr/bin/free && sync && echo 3 ...


3

Partial answer: You can see the files it accesses in real-time by using strace something.sh Specifically, it shows you every system call made by the process.


3

You can get this information through the virtual /proc file-system (under Linux only). Try to run this command when the process is running (replace the <pid> by the PID of the observed process): grep 'VmSize' /proc/<pid>/status Beware, you have to have read access to the process to get these information (you cannot access it if you do not ...


3

Indeed you need to use /proc/; so read carefully proc(5). For process 1234 you want to read /proc/1234/maps (or /proc/1234/smaps) to get the address space, and to read /proc/1234/status & /proc/1234/statm For your own process (programmatically) use /proc/self/maps, /proc/self/status, /proc/self/statm Notice that memory usage is a very ambiguous term ...



Only top voted, non community-wiki answers of a minimum length are eligible