Tag Info

Hot answers tagged

27

Additional information provided in the comments reveals that the OP is using a GUI method to create the .tar.gz file. GUI software often includes a lot more bloat than the equivalent command line equivalent software, or performs additional unnecessary tasks for the sake of some "extra" feature such as a progress bar. It wouldn't surprise me if the GUI ...


12

"five million" files, and 1TB in total? Your files must be very small, then. I'd simply try rsync: rsync -alPEmivvz /source/dir remote.host.tld:/base/dir If you don't have that - or your use-case doesn't allow for using rsync, I'd at least check if 7z works with your data. It might not, but I think it's still worth a try: 7z a archive.7z /source/dir Or ...


7

Unless you can do better than 25:1 compression you are unlikely to gain anything from compressing this before snail-mailing, unless you have some hardware tape format that you can exchange the the third party. The largest common storage is blue ray and that will roughly get you 40Gb. You would need 25 to 1 compression on your data to get it to fit on that. ...


4

Did you consider torrent? Peer-to-Peer might be your best option for an over-the-internet transfer: At least as fast as other internet transfers: your upload speed will determine the transfer speed No data corruption Choose which files to transfer first No extra local/cloud storage space needed Free You didn't tell which OS you were using, but as you're ...


3

I figured out that a large portion of the memory usage was, in fact, attributable to inactive memory used by exited processes. The most accurate way to determine how much memory is available post-January 2014 is to look at MemAvailable in /proc/meminfo. You can also see the amount of inactive memory is this file.


2

I right clicked the folder and clicked "create archive" and selected the .tar.gz option. The directory structure is deep, over 500,000 directories Yeah, good luck getting that to package up. And the GUI tool will try to do that on the same volume, which means a) you need another 1Tb of free space and b) the head thrashing of reading one file and ...


2

7z would be my choice. It allows auto-splitting of archives and supports multi-threaded compression. No, xz doesn't, despite what the help message says. Try with: 7za a -v100m -m0=lzma2 -mx=9 -ms=on -mmt=$THREADS archive.7z directory/ The output is split in 100MB blocks (change it with the -v switch). The only real downside is that 7z does not retain ...


2

It seems like the -o flag will take the actual column name. So if the top command shows only "mem" then the command should be "top -o mem". For the ubuntu machine I am testing with, the column is called "%MEM". On the OSX Yosemite I tried, it is "mem".


2

The Linux kernel has options to restrict what physical address ranges it will use as RAM, but that won't prevent buggy drivers or access through /dev/mem to escape those ranges. You won't gain anything from modifying the MMU configuration during the boot stage, because the kernel will take control of the MMU after that anyway. If you want to be absolutely ...


1

One handy way is to use atop. In atop, you can type P to see per program statistics like this: PAUSED NPROCS SYSCPU USRCPU VSIZE RSIZE RDDSK WRDSK SNET MEM CMD 1/4 17 1.14s 8.06s 14.3G 2.7G ? ? ? 35% chrome 1 0.30s 0.30s 2.6G ...


1

Huge pages would be useful in a situation where you needed a huge amount of information to be written in the same block. It can relate to the strategy for disk writes and can be significant for caching. Like all configuration options, it makes no sense if your use case does not fit. So the answer is, workload that actually needs a huge ton of data in the ...



Only top voted, non community-wiki answers of a minimum length are eligible