Tag Info

Hot answers tagged

61

I came across this diagram which shows exactly this.     In the above you can see where tools such as strace, netstat, etc. interact with the Linux kernel's subsystems. I like this diagram because it succinctly shows where each tool latches on to the Linux kernel, which can be extremely helpful when you're first learning about all the tools ...


59

This behaviour can be configured by setting the value of /proc/sys/vm/swappiness. The default value is 60, setting it to 0 means “never use swap when there is still RAM left“ and 100 is swapping out memory as soon as possible. To change the value temporarily (lost on reboot): sudo sysctl vm.swappiness=10 To change the value permanently: Edit the file ...


58

It is normal for Linux systems to use some swap even if there is still RAM free. The Linux kernel will move to swap memory pages that are very seldom used (e.g., the getty instances when you only use X11, and some other inactive daemon). Swap space usage becomes an issue only when there is not enough RAM available, and the kernel is forced to continuously ...


45

Emptying the buffers cache If you ever want to empty it you can use this chain of commands. $ free && sync && echo 3 > /proc/sys/vm/drop_caches && free total used free shared buffers cached Mem: 1018916 980832 38084 0 46924 355764 -/+ buffers/cache: 578144 ...


43

When you run ls without arguments, it will just open a directory, read all the contents, sort them and print them out. When you run ls *, first the shell expands *, which is effectively the same as what the simple ls did, builds an argument vector with all the files in the current directory and calls ls. ls then has to process that argument vector and for ...


30

Linux starts swapping before the RAM is filled up. This is done to improve performance and responsiveness: Performance is increased because sometimes RAM is better used for disk cache than to store program memory. So it's better to swap out a program that's been inactive for a while, and instead keep often-used files in cache. Responsiveness is improved by ...


29

cpipe is probably better for these purposes, but another related program is pv (Pipe Viewer): If you give it the --rate flag it will show the transfer rate


28

Two potential problems: grep -R (except for the modified GNU grep found on OS/X 10.8 and above) follows symlinks, so even if there's only 100GB of files in ~/Documents, there might still be a symlink to / for instance and you'll end up scanning the whole file system including files like /dev/zero. Use grep -r with newer GNU grep, or use the standard ...


26

I've done the following test and on my system the resulting difference is about 100 times longer for the second script. My file is a strace output called bigfile $ wc -l bigfile.log 1617000 bigfile.log Scripts xtian@clafujiu:~/tmp$ cat p1.sh tail -n 1000000 bigfile.log | grep '"success": true' | wc -l tail -n 1000000 bigfile.log | grep '"success": ...


26

It doesn't get much faster than using the System Request (SysRq) functionality and then triggering an immediate reboot. This is a key combination understood by the kernel. Enable SysRq: echo 1 > /proc/sys/kernel/sysrq Now, send it into reboot. echo b > /proc/sysrq-trigger b - Immediately reboot the system, without unmounting or syncing ...


25

The "definitive" answer is of course brought to you by The Useless Use of cat Award. The purpose of cat is to concatenate (or "catenate") files. If it's only one file, concatenating it with nothing at all is a waste of time, and costs you a process. Instantiating cat just so your code reads differently makes for just one more process and one more set ...


24

Python imports a large number of files at startup: % python -c 'import sys; print len(sys.modules)' 39 Each of these requires an even greater number of attempts at opening a Python file, because there are many ways to define a module: % python -vv -c 'pass' # installing zipimport hook import zipimport # builtin # installed zipimport hook # trying ...


24

Apart from not getting detailed information about your test setup the main problem seems to be, that you use a message size of 64 byte. This is far away from the usual MTU of 1500 bytes and makes UDP highly inefficient: while TCP merges multiple sends into a single packet on the wire (except if TCP_NODELAY is set) to make efficient use of the link, each UDP ...


23

There's but one way to determine the optimal block size, and that's a benchmark. I've just made a quick benchmark. The test machine is a PC running Debian GNU/Linux, with kernel 2.6.32 and coreutils 8.5. Both filesystems involved are ext3 on LVM volumes on a hard disk partition. The source file is 2GB (2040000kB to be precise). Caching and buffering are ...


21

So I went to the source, and it looks like the slowness is in handling double byte characters. Essentially, for every character read in, it needs to call mbrtowc() to try to convert it to a wide character, then that wide character is tested to see if it's a word separator, line separator, etc. Indeed, if I change my locale LANG variable from the default ...


20

I think you are looking for pbzip2: PBZIP2 is a parallel implementation of the bzip2 block-sorting file compressor that uses pthreads and achieves near-linear speedup on SMP machines. Have a look at the project homepage or check your favorite package repository.


19

That's exactly the difference between /dev/random and /dev/urandom -- random uses the entropy pool, which gathers noise from a bunch of sources and keeps track of "how much" noise is currently in the pool, so random knows how much high-quality randomness it can generate. Since the entropy pool has a finite amount of noise, reading from random might need to ...


18

The most obvious answer is just to use the diff command and it is probably a good idea to add the --speed-large-files parameter to it. diff --speed-large-files a.file b.file You mention unsorted files so maybe you need to sort the files first sort a.file > a.file.sorted sort b.file > b.file.sorted diff --speed-large-files a.file.sorted ...


18

I tried this on a directory with 2259 entries, and used the time command. The output of time for f in *; do echo "$f"; done (minus the files!) is: real 0m0.062s user 0m0.036s sys 0m0.012s The output of time find * -prune | while read f; do echo "$f"; done (minus the files!) is: real 0m0.131s user 0m0.056s sys 0m0.060s I ran each ...


15

You need a utility called cpipe. Usage: tar cCf / - usr | cpipe -vr -vw -vt > /dev/null Output: ... in: 19.541ms at 6.4MB/s ( 4.7MB/s avg) 2.0MB out: 0.004ms at 30.5GB/s ( 27.1GB/s avg) 2.0MB thru: 19.865ms at 6.3MB/s ( 4.6MB/s avg) 2.0MB ...


15

Performance I wrote a small Benchmark (source), to find out, what file system performs best with hundred thousands of small files: create 300000 files (512B to 1536B) with data from /dev/urandom rewrite 30000 random files and change the size read 30000 sequential files read 30000 random files delete all files sync and drop cache after every step ...


14

No, it doesn't. The issue isn't with the type of disk (spinning/non-spinning), it's with committing disk buffers from RAM to disk. If the power goes out suddenly, some of these buffers may never get committed to disk, and having barriers enabled improves your chances of recovering the filesystem. There's also an additional issue with the disk's on-board ...


13

Did you have a look at some other "lighterweight" ;-) window managers? I'm completly happy with i3 for example: http://i3wm.org/ It's just a tiling windowmanger with dmenu for launching applications. No desktop, no other special features and the binary is just some KBs. There are a lot others in this range: evilwm - http://www.6809.org.uk/evilwm/ dwm - ...


13

Most software build processes use make. Make sure you make make use the -j argument with a number usually about twice the number of CPUs you have, so make -j 8 would be appropriate for your case.


12

The Wikipedia page on Comparison of X Window Managers sorts the various Window Managers into four categories: Heavyweight, Middleweight, Lightweight, and minimal. You'd probably be interested in those in the minimal category. Right now, those include Matchbox, sithWM, evilwm, dwm, WMFS, wmii, and scrotwm. (i3 gets put into Lightweight; Xfwm (used by ...


12

This has been a known issue for awhile. Using an SSD-tuned FS like Btrfs might help, but it might not. Ultimately, it is a bug in the IO scheduler/memory management systems. Recently, there have been some patches that aim to address this issue. See Fixed: The Linux Desktop Responsiveness Problem? These patches may eventually make their way into the ...


12

I would give iotop a try (Linux only). It is pretty good at giving you the processes with high I/O transfers, but iotop is more for displaying the current status than logging. For logging, sar is always a good tool for performance aspects.


12

I suggest reading SwapFAQ , in particular the swapiness parameter.


12

7zip can run on multiple threads when given the -mmt flag, but only when compressing into 7z-archives which offer great compression but are generally slower to create then zip archives. Do something like this. 7z a -mmt foo.7z /opt/myhugefile.dat


12

The obvious way to keep a bunch of files in the cache is to access them often. Linux is pretty good at arbitrating between swapping and caching, so I suspect that the speed difference you observe is actually not due to the OS not keeping things in the cache, but to some other difference between your usage of tmpfs and your other attempts. Try observing what ...



Only top voted, non community-wiki answers of a minimum length are eligible