Tag Info

Hot answers tagged

56

This behaviour can be configured by setting the value of /proc/sys/vm/swappiness. The default value is 60, setting it to 0 means “never use swap when there is still RAM left“ and 100 is swapping out memory as soon as possible. To change the value temporarily (lost on reboot): sudo sysctl vm.swappiness=10 To change the value permanently: Edit the file ...


54

It is normal for Linux systems to use some swap even if there is still RAM free. The Linux kernel will move to swap memory pages that are very seldom used (e.g., the getty instances when you only use X11, and some other inactive daemon). Swap space usage becomes an issue only when there is not enough RAM available, and the kernel is forced to continuously ...


50

I came across this diagram which shows exactly this.     In the above you can see where tools such as strace, netstat, etc. interact with the Linux kernel's subsystems. I like this diagram because it succinctly shows where each tool latches on to the Linux kernel, which can be extremely helpful when you're first learning about all the tools ...


42

When you run ls without arguments, it will just open a directory, read all the contents, sort them and print them out. When you run ls *, first the shell expands *, which is effectively the same as what the simple ls did, builds an argument vector with all the files in the current directory and calls ls. ls then has to process that argument vector and for ...


26

cpipe is probably better for these purposes, but another related program is pv (Pipe Viewer): If you give it the --rate flag it will show the transfer rate


26

Linux starts swapping before the RAM is filled up. This is done to improve performance and responsiveness: Performance is increased because sometimes RAM is better used for disk cache than to store program memory. So it's better to swap out a program that's been inactive for a while, and instead keep often-used files in cache. Responsiveness is improved by ...


26

Two potential problems: grep -R (except for the modified GNU grep found on OS/X 10.8 and above) follows symlinks, so even if there's only 100GB of files in ~/Documents, there might still be a symlink to / for instance and you'll end up scanning the whole file system including files like /dev/zero. Use grep -r with newer GNU grep, or use the standard ...


26

I've done the following test and on my system the resulting difference is about 100 times longer for the second script. My file is a strace output called bigfile $ wc -l bigfile.log 1617000 bigfile.log Scripts xtian@clafujiu:~/tmp$ cat p1.sh tail -n 1000000 bigfile.log | grep '"success": true' | wc -l tail -n 1000000 bigfile.log | grep '"success": ...


23

Apart from not getting detailed information about your test setup the main problem seems to be, that you use a message size of 64 byte. This is far away from the usual MTU of 1500 bytes and makes UDP highly inefficient: while TCP merges multiple sends into a single packet on the wire (except if TCP_NODELAY is set) to make efficient use of the link, each UDP ...


21

The "definitive" answer is of course brought to you by The Useless Use of cat Award. The purpose of cat is to concatenate (or "catenate") files. If it's only one file, concatenating it with nothing at all is a waste of time, and costs you a process. Instantiating cat just so your code reads differently makes for just one more process and one more set ...


20

I think you are looking for pbzip2: PBZIP2 is a parallel implementation of the bzip2 block-sorting file compressor that uses pthreads and achieves near-linear speedup on SMP machines. Have a look at the project homepage or check your favorite package repository.


19

There's but one way to determine the optimal block size, and that's a benchmark. I've just made a quick benchmark. The test machine is a PC running Debian GNU/Linux, with kernel 2.6.32 and coreutils 8.5. Both filesystems involved are ext3 on LVM volumes on a hard disk partition. The source file is 2GB (2040000kB to be precise). Caching and buffering are ...


19

That's exactly the difference between /dev/random and /dev/urandom -- random uses the entropy pool, which gathers noise from a bunch of sources and keeps track of "how much" noise is currently in the pool, so random knows how much high-quality randomness it can generate. Since the entropy pool has a finite amount of noise, reading from random might need to ...


19

So I went to the source, and it looks like the slowness is in handling double byte characters. Essentially, for every character read in, it needs to call mbrtowc() to try to convert it to a wide character, then that wide character is tested to see if it's a word separator, line separator, etc. Indeed, if I change my locale LANG variable from the default ...


18

I tried this on a directory with 2259 entries, and used the time command. The output of time for f in *; do echo "$f"; done (minus the files!) is: real 0m0.062s user 0m0.036s sys 0m0.012s The output of time find * -prune | while read f; do echo "$f"; done (minus the files!) is: real 0m0.131s user 0m0.056s sys 0m0.060s I ran each ...


16

The most obvious answer is just to use the diff command and it is probably a good idea to add the --speed-large-files parameter to it. diff --speed-large-files a.file b.file You mention unsorted files so maybe you need to sort the files first sort a.file > a.file.sorted sort b.file > b.file.sorted diff --speed-large-files a.file.sorted ...


14

You need a utility called cpipe. Usage: tar cCf / - usr | cpipe -vr -vw -vt > /dev/null Output: ... in: 19.541ms at 6.4MB/s ( 4.7MB/s avg) 2.0MB out: 0.004ms at 30.5GB/s ( 27.1GB/s avg) 2.0MB thru: 19.865ms at 6.3MB/s ( 4.6MB/s avg) 2.0MB ...


14

Performance I wrote a small Benchmark (source), to find out, what file system performs best with hundred thousands of small files: create 300000 files (512B to 1536B) with data from /dev/urandom rewrite 30000 random files and change the size read 30000 sequential files read 30000 random files delete all files sync and drop cache after every step ...


14

Emptying the buffers cache If you ever want to empty it you can use this chain of commands. $ free && sync && echo 3 > /proc/sys/vm/drop_caches && free total used free shared buffers cached Mem: 1018916 980832 38084 0 46924 355764 -/+ buffers/cache: 578144 ...


12

The Wikipedia page on Comparison of X Window Managers sorts the various Window Managers into four categories: Heavyweight, Middleweight, Lightweight, and minimal. You'd probably be interested in those in the minimal category. Right now, those include Matchbox, sithWM, evilwm, dwm, WMFS, wmii, and scrotwm. (i3 gets put into Lightweight; Xfwm (used by ...


12

Did you have a look at some other "lighterweight" ;-) window managers? I'm completly happy with i3 for example: http://i3wm.org/ It's just a tiling windowmanger with dmenu for launching applications. No desktop, no other special features and the binary is just some KBs. There are a lot others in this range: evilwm - http://www.6809.org.uk/evilwm/ dwm - ...


12

7zip can run on multiple threads when given the -mmt flag, but only when compressing into 7z-archives which offer great compression but are generally slower to create then zip archives. Do something like this. 7z a -mmt foo.7z /opt/myhugefile.dat


12

The obvious way to keep a bunch of files in the cache is to access them often. Linux is pretty good at arbitrating between swapping and caching, so I suspect that the speed difference you observe is actually not due to the OS not keeping things in the cache, but to some other difference between your usage of tmpfs and your other attempts. Try observing what ...


12

No, it doesn't. The issue isn't with the type of disk (spinning/non-spinning), it's with committing disk buffers from RAM to disk. If the power goes out suddenly, some of these buffers may never get committed to disk, and having barriers enabled improves your chances of recovering the filesystem. There's also an additional issue with the disk's on-board ...


11

This has been a known issue for awhile. Using an SSD-tuned FS like Btrfs might help, but it might not. Ultimately, it is a bug in the IO scheduler/memory management systems. Recently, there have been some patches that aim to address this issue. See Fixed: The Linux Desktop Responsiveness Problem? These patches may eventually make their way into the ...


10

Some pointers: Don't run so many GUI programs at once. Make sure that any programs running in the background that you don't need e.g. Apache are stopped. Use a distro aimed at low-memory situations (e.g. for a netbook) Buy more memory. Buy a faster HD (or SSD) for your swap partition. :)


10

I think for the most part it has been solved. My performance under heavy IO has improved in 2.6.36 and I expect it to improve more in 2.6.37. See these phoronix Articles. Wu Fengguang and KOSAKI Motohiro have published patches this week that they believe will address some of these responsiveness issues, for which they call the "system goes unresponsive ...


10

Reniceing the process group to -20 is a bad idea. This niceness level should be used only by the top-priority system-critical tasks. Otherwise you can loose responsiveness or even freeze the system. And the potential compilation-time benefit would be marginal. Apart from what Caleb already suggested, if you compile a lot, you can also speed up builds using ...


10

By using strace, I saw that the line number=$(expr $number + 1) causes a fork, path search, and exec of expr. (I'm using bash 4.2.45 on Ubuntu). That filesystem, disk, and process overhead led to bash only getting around 28% of the CPU. When I changed that line to use only shell builtin operations ((number = number + 1)) bash used around 98% of the ...


9

The kernel documentation provides a general coverage of cgroups with examples. The cgroups-bin package (which depends on libcgroup1) already provided by the distribution should be fine. Configuration is done by editing the following two files: /etc/cgconfig.conf Used by libcgroup to define control groups, their parameters and mount points. ...



Only top voted, non community-wiki answers of a minimum length are eligible