Hot answers tagged

28

Additional information provided in the comments reveals that the OP is using a GUI method to create the .tar.gz file. GUI software often includes a lot more bloat than the equivalent command line equivalent software, or performs additional unnecessary tasks for the sake of some "extra" feature such as a progress bar. It wouldn't surprise me if the GUI ...


13

"five million" files, and 1TB in total? Your files must be very small, then. I'd simply try rsync: rsync -alPEmivvz /source/dir remote.host.tld:/base/dir If you don't have that - or your use-case doesn't allow for using rsync, I'd at least check if 7z works with your data. It might not, but I think it's still worth a try: 7z a archive.7z /source/dir Or ...


11

This statement is odd: split("0,2,4,5,7,9,11,12",a,","); It repetitively splits a constant string to create an array a. If you move that into a BEGIN section, the program should work the same — without allocating a new copy of the a array for each input-record. Addressing comments: the for-loop and expression do not allocate memory in a simple ...


9

In modern systems none of the memory is actually copied just because a fork system call is used. It is all marked read only in the page table such that on first attempt to write a trap into kernel code will happen. Only once the first process attempt to write will the copying happen. This is known as copy-on-write. However it may be necessary to keep track ...


8

Unless you can do better than 25:1 compression you are unlikely to gain anything from compressing this before snail-mailing, unless you have some hardware tape format that you can exchange the the third party. The largest common storage is blue ray and that will roughly get you 40Gb. You would need 25 to 1 compression on your data to get it to fit on that. ...


8

In your second reference, where it says The memory allocator gets a big chunk of memory (say, 4 pages, or 4 * 4 KiB) and divides this into much smaller chunks "the memory allocator" is not the kernel but the libc routines like malloc. The kernel allocates memory to the process in page-sized chunks. malloc keeps track of used and unused portions of ...


6

The linux swapping algorithm works with the concept of "last recently used pages". Each page in virtual memory has an age associated with it. If the page is being frequently accessed then that page is supposed to be quite young in age while if a page is not being accessed, then that page becomes older. The older the pages get, the more likely they may get ...


5

Linux kernel maintainers are listed in the MAINTAINERS file in the kernel source code. There's a specific section for memory management: MEMORY MANAGEMENT L: linux-mm@kvack.org W: http://www.linux-mm.org S: Maintained F: include/linux/mm.h F: include/linux/gfp.h F: include/linux/mmzone.h F: include/linux/memory_hotplug.h ...


5

Did you consider torrent? Peer-to-Peer might be your best option for an over-the-internet transfer: At least as fast as other internet transfers: your upload speed will determine the transfer speed No data corruption Choose which files to transfer first No extra local/cloud storage space needed Free You didn't tell which OS you were using, but as you're ...


5

It seems like the -o flag will take the actual column name. So if the top command shows only "mem" then the command should be "top -o mem". For the ubuntu machine I am testing with, the column is called "%MEM". On the OSX Yosemite I tried, it is "mem".


5

The short story: If your mobo posts, and your system boots, and free/top show your ram as 16 gB, then it works. Even mobo makers can under-report capacity of system boards, so the real test is if ram is installed correctly, matched correctly, runs, ie, boots, and runs with stability, ie, doesn't crash, then it works. You can also test by trying to use all ...


5

niceload --noswap yourprg is made for exactly that situation: It looks at swapping activity: If swapping out: Let process run If swapping in: Let process run If swapping in and out: Suspend process until swapping stops and resume the process when swapping has stopped It does not suspend the process before the swapping starts, but lets swapping run for 1 ...


5

This is not a loop, but recursion and the memory increases linear over the time, which is what you don't want. If you want a loop with constant memory usage, you can do it this way: #!/bin/sh while 1; do mysql -h "localhost" -u "root" "-p********" "database" < "update.sql" sleep 5 done


5

How can I see the raw memory data used by an application... Once you have obtained the process' PID (using ps(1) or pidof(8) for instance), you may access the data in its virtual address space using /proc/PID/maps and /proc/PID/mem. Gilles wrote a very detailled answer about that here. ... and all the files its accessing in my filesystem, network ...


5

If the system supports process accounting, and accounting is enabled, then there may be records available in the process accounting file, e.g. on RedHat Linux: $ grep -q BSD_PROCESS_ACCT=y /boot/config-* && echo hooray hooray $ sudo touch /var/log/pacct $ sudo accton /var/log/pacct $ ... $ sudo accton # turn it off Parsing the acct(5) file is ...


4

From what you have posted it doesn't seems like you under stand how memory works in Linux. I recommend reading http://www.linuxnix.com/find-ram-size-in-linuxunix/ http://www.itworld.com/article/2722141/it-management/making-sense-of-memory-usage-on-linux.html http://www.linuxatemyram.com/ The jist of those sites is that you have more "free" ram then ...


4

You can press the following keys: e -- Change the scaling factor on the summary display Shift+e -- Change the scaling factor on the task Shift+w -- Save current settings


4

There is kernel setting /proc/sys/vm/overcommit_memory Citation from excellent article: Since 2.5.30 the values are: 0 (default): as before: guess about how much overcommitment is reasonable, 1: never refuse any malloc(), 2: be precise about the overcommit - never commit a virtual address space larger than swap space plus a fraction overcommit_ratio ...


4

Here's a perl equivalent that doesn't leak: perl -lne 'BEGIN { @a=(0,2,4,5,7,9,11,12);} for ($i = 0; $i < 1; $i+= 0.0001) { printf("%08X\n", 100*sin(1382*exp($a[$F[0] % 8]/12)*log(2))*$i) }' It's almost identical. $1 gets replaced by $F[0] and i is replaced with $i. The hash a is replaced with an actual array, @a. You would be wise to generate ...


4

MemAvailable is included in /proc/meminfo since version 3.14 of the kernel; it was added by commit 34e431b0a. That's the determining factor in the output variations you show. The commit message indicates how to estimate available memory without MemAvailable: Currently, the amount of memory that is available for a new workload, without pushing the system ...


4

Processes are managed by the kernel. The kernel doesn't care how the programmer allocates variables. All it knows is that certain blocks of memory belong to the process. The C runtime matches C memory management features to kernel features: automatic variables go into a memory block called “stack” and dynamic storage (malloc and friends) go into a memory ...


4

free is provided by procps-ng; Debian 8 has version 3.3.9, which uses the old style with a separate line for buffers/cache, while Gentoo and presumably RHEL 7.x have version 3.3.10 or later which uses the new style. You can see the reasoning behind the change in the corresponding commit message. If you really want the old-style output you can run an older ...


3

Swap space is located on hard drives, which have a slower access time than RAM. Also the CPU cannot access/address hard drives directly because there is no direct physical data connection between the CPU and the HDD.


3

No. Even if there were the possibility of dumping the memory of the application to read the content of the unwritten logs, once you restarted the application the old process ended, and its memory is gone.


3

Not the program as it is executed in memory but the page cache keeps all the files (executable, libraries, and configuration files). Thus the time for the disk access may be saved on the second execution. But the dynamic linking has to be done again.


3

HardwareCorrupted show the amount of memory in "poisoned pages", i.e. memory which has failed (as flagged by ECC typically). ECC stands for "Error Correcting Code". ECC memory is capable of correcting small errors and detecting larger ones; on typical PCs with non-ECC memory, memory errors go undetected. If an uncorrectable error is detected using ECC (in ...


3

7z would be my choice. It allows auto-splitting of archives and supports multi-threaded compression. No, xz doesn't, despite what the help message says. Try with: 7za a -v100m -m0=lzma2 -mx=9 -ms=on -mmt=$THREADS archive.7z directory/ The output is split in 100MB blocks (change it with the -v switch). The only real downside is that 7z does not retain ...


3

I right clicked the folder and clicked "create archive" and selected the .tar.gz option. The directory structure is deep, over 500,000 directories Yeah, good luck getting that to package up. And the GUI tool will try to do that on the same volume, which means a) you need another 1Tb of free space and b) the head thrashing of reading one file and ...


3

It would depend on what kind of stats you want, but if you're writing a program in C running on Linux, you'd definitely better know about Valgrind. Valgrind can, not only profile detailed memory usage of your program, but also detect memory access violations which are common in C and possibly very hard to debug. For your profiling purpose, take a look at ...


3

You don't have to do anything special, apart from providing swap space and mounting it. For processors with a MMU (memory management unit, i.e. most processors since the early 90's) the full address space can be used, although there doesn't have to be real memory (RAM) at all address range locations. If some memory location is addressed, RAM will be ...



Only top voted, non community-wiki answers of a minimum length are eligible