Tag Info

Hot answers tagged

27

Additional information provided in the comments reveals that the OP is using a GUI method to create the .tar.gz file. GUI software often includes a lot more bloat than the equivalent command line equivalent software, or performs additional unnecessary tasks for the sake of some "extra" feature such as a progress bar. It wouldn't surprise me if the GUI ...


12

"five million" files, and 1TB in total? Your files must be very small, then. I'd simply try rsync: rsync -alPEmivvz /source/dir remote.host.tld:/base/dir If you don't have that - or your use-case doesn't allow for using rsync, I'd at least check if 7z works with your data. It might not, but I think it's still worth a try: 7z a archive.7z /source/dir Or ...


7

Unless you can do better than 25:1 compression you are unlikely to gain anything from compressing this before snail-mailing, unless you have some hardware tape format that you can exchange the the third party. The largest common storage is blue ray and that will roughly get you 40Gb. You would need 25 to 1 compression on your data to get it to fit on that. ...


4

Did you consider torrent? Peer-to-Peer might be your best option for an over-the-internet transfer: At least as fast as other internet transfers: your upload speed will determine the transfer speed No data corruption Choose which files to transfer first No extra local/cloud storage space needed Free You didn't tell which OS you were using, but as you're ...


3

I figured out that a large portion of the memory usage was, in fact, attributable to inactive memory used by exited processes. The most accurate way to determine how much memory is available post-January 2014 is to look at MemAvailable in /proc/meminfo. You can also see the amount of inactive memory is this file.


3

It would depend on what kind of stats you want, but if you're writing a program in C running on Linux, you'd definitely better know about Valgrind. Valgrind can, not only profile detailed memory usage of your program, but also detect memory access violations which are common in C and possibly very hard to debug. For your profiling purpose, take a look at ...


2

If you are only interested in the memory used after the fact, then use GNU time: command time -v myprogram (the above uses the bash way of invoking the external time command rather than the bash builtin, your shell may vary). Or, GNU memusage: memusage -T ./myprogram If you are interested in the memory used on an ongoing basis (i.e. during a long ...


2

It seems like the -o flag will take the actual column name. So if the top command shows only "mem" then the command should be "top -o mem". For the ubuntu machine I am testing with, the column is called "%MEM". On the OSX Yosemite I tried, it is "mem".


2

The Linux kernel has options to restrict what physical address ranges it will use as RAM, but that won't prevent buggy drivers or access through /dev/mem to escape those ranges. You won't gain anything from modifying the MMU configuration during the boot stage, because the kernel will take control of the MMU after that anyway. If you want to be absolutely ...


2

I right clicked the folder and clicked "create archive" and selected the .tar.gz option. The directory structure is deep, over 500,000 directories Yeah, good luck getting that to package up. And the GUI tool will try to do that on the same volume, which means a) you need another 1Tb of free space and b) the head thrashing of reading one file and ...


2

7z would be my choice. It allows auto-splitting of archives and supports multi-threaded compression. No, xz doesn't, despite what the help message says. Try with: 7za a -v100m -m0=lzma2 -mx=9 -ms=on -mmt=$THREADS archive.7z directory/ The output is split in 100MB blocks (change it with the -v switch). The only real downside is that 7z does not retain ...


1

One handy way is to use atop. In atop, you can type P to see per program statistics like this: PAUSED NPROCS SYSCPU USRCPU VSIZE RSIZE RDDSK WRDSK SNET MEM CMD 1/4 17 1.14s 8.06s 14.3G 2.7G ? ? ? 35% chrome 1 0.30s 0.30s 2.6G ...


1

Here's the resident set size and virtual memory size of all sshd processes on one system: ulric@qvp2:~$ ps -eo rss,vsz,args|grep sshd|grep -v grep 448 55292 /usr/sbin/sshd -D 5176 147460 sshd: ulric [priv] 2776 149704 sshd: ulric@pts/3 Or perhaps easier: ulric@qvp2:~$ ps aux|head -n 1&&ps aux|grep sshd|grep -v grep USER PID %CPU %MEM ...


1

Huge pages would be useful in a situation where you needed a huge amount of information to be written in the same block. It can relate to the strategy for disk writes and can be significant for caching. Like all configuration options, it makes no sense if your use case does not fit. So the answer is, workload that actually needs a huge ton of data in the ...



Only top voted, non community-wiki answers of a minimum length are eligible