Tag Info

Hot answers tagged

61

This behaviour can be configured by setting the value of /proc/sys/vm/swappiness. The default value is 60, setting it to 0 means “never use swap when there is still RAM left“ and 100 is swapping out memory as soon as possible. To change the value temporarily (lost on reboot): sudo sysctl vm.swappiness=10 To change the value permanently: Edit the file ...


61

Emptying the buffers cache If you ever want to empty it you can use this chain of commands. $ free && sync && echo 3 > /proc/sys/vm/drop_caches && free total used free shared buffers cached Mem: 1018916 980832 38084 0 46924 355764 -/+ buffers/cache: 578144 ...


61

I came across this diagram which shows exactly this.     In the above you can see where tools such as strace, netstat, etc. interact with the Linux kernel's subsystems. I like this diagram because it succinctly shows where each tool latches on to the Linux kernel, which can be extremely helpful when you're first learning about all the tools ...


59

It is normal for Linux systems to use some swap even if there is still RAM free. The Linux kernel will move to swap memory pages that are very seldom used (e.g., the getty instances when you only use X11, and some other inactive daemon). Swap space usage becomes an issue only when there is not enough RAM available, and the kernel is forced to continuously ...


43

When you run ls without arguments, it will just open a directory, read all the contents, sort them and print them out. When you run ls *, first the shell expands *, which is effectively the same as what the simple ls did, builds an argument vector with all the files in the current directory and calls ls. ls then has to process that argument vector and for ...


31

Linux starts swapping before the RAM is filled up. This is done to improve performance and responsiveness: Performance is increased because sometimes RAM is better used for disk cache than to store program memory. So it's better to swap out a program that's been inactive for a while, and instead keep often-used files in cache. Responsiveness is improved by ...


30

cpipe is probably better for these purposes, but another related program is pv (Pipe Viewer): If you give it the --rate flag it will show the transfer rate


28

Two potential problems: grep -R (except for the modified GNU grep found on OS/X 10.8 and above) follows symlinks, so even if there's only 100GB of files in ~/Documents, there might still be a symlink to / for instance and you'll end up scanning the whole file system including files like /dev/zero. Use grep -r with newer GNU grep, or use the standard ...


27

The "definitive" answer is of course brought to you by The Useless Use of cat Award. The purpose of cat is to concatenate (or "catenate") files. If it's only one file, concatenating it with nothing at all is a waste of time, and costs you a process. Instantiating cat just so your code reads differently makes for just one more process and one more set ...


27

It doesn't get much faster than using the System Request (SysRq) functionality and then triggering an immediate reboot. This is a key combination understood by the kernel. Enable SysRq: echo 1 > /proc/sys/kernel/sysrq Now, send it into reboot. echo b > /proc/sysrq-trigger b - Immediately reboot the system, without unmounting or syncing ...


26

There's but one way to determine the optimal block size, and that's a benchmark. I've just made a quick benchmark. The test machine is a PC running Debian GNU/Linux, with kernel 2.6.32 and coreutils 8.5. Both filesystems involved are ext3 on LVM volumes on a hard disk partition. The source file is 2GB (2040000kB to be precise). Caching and buffering are ...


26

I've done the following test and on my system the resulting difference is about 100 times longer for the second script. My file is a strace output called bigfile $ wc -l bigfile.log 1617000 bigfile.log Scripts xtian@clafujiu:~/tmp$ cat p1.sh tail -n 1000000 bigfile.log | grep '"success": true' | wc -l tail -n 1000000 bigfile.log | grep '"success": ...


25

Apart from not getting detailed information about your test setup the main problem seems to be, that you use a message size of 64 byte. This is far away from the usual MTU of 1500 bytes and makes UDP highly inefficient: while TCP merges multiple sends into a single packet on the wire (except if TCP_NODELAY is set) to make efficient use of the link, each UDP ...


24

Python imports a large number of files at startup: % python -c 'import sys; print len(sys.modules)' 39 Each of these requires an even greater number of attempts at opening a Python file, because there are many ways to define a module: % python -vv -c 'pass' # installing zipimport hook import zipimport # builtin # installed zipimport hook # trying ...


21

So I went to the source, and it looks like the slowness is in handling double byte characters. Essentially, for every character read in, it needs to call mbrtowc() to try to convert it to a wide character, then that wide character is tested to see if it's a word separator, line separator, etc. Indeed, if I change my locale LANG variable from the default ...


21

As you say, use rsync: rsync -azP /var/www/html/txt/ username@ip-address:/var/www/html/txt The options are: -a : enables archive mode, which preserves symbolic links and works recursively -z : compress the data transfer to minimise network usage -P : to display a progress bar and enables you to resume partial transfers As @aim says in his answer, make ...


20

I think you are looking for pbzip2: PBZIP2 is a parallel implementation of the bzip2 block-sorting file compressor that uses pthreads and achieves near-linear speedup on SMP machines. Have a look at the project homepage or check your favorite package repository.


19

That's exactly the difference between /dev/random and /dev/urandom -- random uses the entropy pool, which gathers noise from a bunch of sources and keeps track of "how much" noise is currently in the pool, so random knows how much high-quality randomness it can generate. Since the entropy pool has a finite amount of noise, reading from random might need to ...


18

The most obvious answer is just to use the diff command and it is probably a good idea to add the --speed-large-files parameter to it. diff --speed-large-files a.file b.file You mention unsorted files so maybe you need to sort the files first sort a.file > a.file.sorted sort b.file > b.file.sorted diff --speed-large-files a.file.sorted ...


18

Improving disk cache performance in general is more than just increasing the file system cache size unless your whole system fits in RAM in which case you should use RAM drive (tmpfs is good because it allows falling back to disk if you need the RAM in some case) for runtime storage (and perhaps an initrd script to copy system from storage to RAM drive at ...


18

I tried this on a directory with 2259 entries, and used the time command. The output of time for f in *; do echo "$f"; done (minus the files!) is: real 0m0.062s user 0m0.036s sys 0m0.012s The output of time find * -prune | while read f; do echo "$f"; done (minus the files!) is: real 0m0.131s user 0m0.056s sys 0m0.060s I ran each ...


17

Performance I wrote a small Benchmark (source), to find out, what file system performs best with hundred thousands of small files: create 300000 files (512B to 1536B) with data from /dev/urandom rewrite 30000 random files and change the size read 30000 sequential files read 30000 random files delete all files sync and drop cache after every step ...


16

You need a utility called cpipe. Usage: tar cCf / - usr | cpipe -vr -vw -vt > /dev/null Output: ... in: 19.541ms at 6.4MB/s ( 4.7MB/s avg) 2.0MB out: 0.004ms at 30.5GB/s ( 27.1GB/s avg) 2.0MB thru: 19.865ms at 6.3MB/s ( 4.6MB/s avg) 2.0MB ...


15

Izkata's comment revealed the answer: locale-specific comparisons. The sort command uses the locale indicated by the environment, whereas Python defaults to a byte order comparison. Comparing UTF-8 strings is harder than comparing byte strings. $ time (LC_ALL=C sort <numbers.txt >s2.txt) real 0m5.485s user 0m14.028s sys 0m0.404s How about ...


14

Did you have a look at some other "lighterweight" ;-) window managers? I'm completly happy with i3 for example: http://i3wm.org/ It's just a tiling windowmanger with dmenu for launching applications. No desktop, no other special features and the binary is just some KBs. There are a lot others in this range: evilwm - http://www.6809.org.uk/evilwm/ dwm - ...


14

No, it doesn't. The issue isn't with the type of disk (spinning/non-spinning), it's with committing disk buffers from RAM to disk. If the power goes out suddenly, some of these buffers may never get committed to disk, and having barriers enabled improves your chances of recovering the filesystem. There's also an additional issue with the disk's on-board ...


13

Most software build processes use make. Make sure you make make use the -j argument with a number usually about twice the number of CPUs you have, so make -j 8 would be appropriate for your case.


12

The Wikipedia page on Comparison of X Window Managers sorts the various Window Managers into four categories: Heavyweight, Middleweight, Lightweight, and minimal. You'd probably be interested in those in the minimal category. Right now, those include Matchbox, sithWM, evilwm, dwm, WMFS, wmii, and scrotwm. (i3 gets put into Lightweight; Xfwm (used by ...


12

This has been a known issue for awhile. Using an SSD-tuned FS like Btrfs might help, but it might not. Ultimately, it is a bug in the IO scheduler/memory management systems. Recently, there have been some patches that aim to address this issue. See Fixed: The Linux Desktop Responsiveness Problem? These patches may eventually make their way into the ...


12

I would give iotop a try (Linux only). It is pretty good at giving you the processes with high I/O transfers, but iotop is more for displaying the current status than logging. For logging, sar is always a good tool for performance aspects.



Only top voted, non community-wiki answers of a minimum length are eligible