The tag has no wiki summary.

learn more… | top users | synonyms

79
votes
7answers
22k views

Why use swap when there is more than enough free space in RAM?

Using swap space instead of RAM can drastically slow down a PC. So why, when I have more than enough RAM available, does my Linux system (Arch) use the swap? Checkout my conky output below: Also, ...
46
votes
5answers
25k views

Can I configure my Linux system for more aggressive file system caching?

I am neither concerned about RAM usage (as I've got enough) nor about losing data in case of an accidental shut-down (as my power is backed, the system is reliable and the data are not critical). But ...
44
votes
1answer
2k views

Diagram of Linux kernel vs. performance tools?

Is there a diagram that shows how the various performance tools such as ip, netstat, perf, top, ps, etc. interact with the various subsystems within the Linux kernel?
33
votes
1answer
36k views

How do you empty the buffers and cache on a Linux system?

Prior to doing some benchmarking work how would one free up the memory (RAM) that the Linux Kernel is consuming for its buffers and cache?
28
votes
4answers
11k views

Is there a way to determine the optimal value for the bs parameter to dd?

On occasion I've seen comments online along the lines of "make sure you set 'bs=' because the default value will take too long," and my own extremely-unscientific experiences of, "well that seemed to ...
26
votes
4answers
2k views

python vs bc in evaluating 6^6^6

I am evaluating the expression 6^6^6 using python and bc separately. The content of the python file is print 6**6**6. When I execute time python test.py, I get the output as real 0m0.067s ...
26
votes
3answers
1k views

Should I care about unnecessary cats?

A lot of command-line utilities can take their input either from a pipe or as a filename argument. For long shell scripts, I find starting the chain off with a cat makes it more readable, especially ...
25
votes
1answer
2k views

Why does “ls *” take so much longer than “ls”?

I have a couple of files in a directory: $ ls | wc -l 9376 Can anybody explain why there is such a huge time difference in using ls * and ls? $ time ls > /dev/null real 0m0.118s user ...
25
votes
12answers
3k views

How can I tweak my Linux desktop to be more responsive?

I have the experience that Linux works fine until the physical memory is exhausted. As soon as swap space is used the performance is severely degraded and the GUI becomes unresponsive. This problem ...
24
votes
5answers
3k views

Why is iterating over a file twice faster than reading it into memory and computing twice?

I'm comparing the following tail -n 1000000 stdout.log | grep -c '"success": true' tail -n 1000000 stdout.log | grep -c '"success": false' with the following log=$(tail -n 1000000 stdout.log) echo ...
23
votes
9answers
1k views

for vs find in Bash

When looping through files there are two ways: 1. use a for-loop for f in *; do echo "$f" done 2. use find find * -prune | while read f; do echo "$f" done Assuming these two loops will ...
22
votes
7answers
6k views

Force directory to always be in cache

I've been testing out different methods to improve the time it takes to compile my entire c++ project. Currently it takes ~5 minutes. I experimented with distcc, ccache, and others. Recently, I ...
22
votes
4answers
2k views

How to limit resource usage to save CPU+RAM for a certain process?

I have a dev server on which sshd sometimes stops working because the machine runs out of RAM. Yes we are running low on memory and upgrading is not a feasible choice right now. What I want to do is ...
21
votes
11answers
8k views

the lightest way to have a GUI in Linux?

I have been using Gnome and KDE for a while, and I am now running Compiz standalone... Regarding system resource, I think the most heavy ones are feature-rich desktop environments (Gnome, KDE), then ...
21
votes
3answers
4k views

grep: memory exhausted

I was doing a very simple search: grep -R Milledgeville ~/Documents And after some time this error appeared: grep: memory exhausted How can I avoid this? I have 10GB of RAM on my system and few ...
21
votes
4answers
14k views

How to speed up X over SSH on a slow network connection?

Are there any specific recommendations on speeding up X applications over ssh on a slow network connection? In this specific case, I am accessing a server located in west coast from a laptop in east ...
18
votes
4answers
2k views

Is there a fastest way to shutdown the system?

Or is shutdown -h now the fastest it can get? I look for some syscall or similar that will allow to skip lots of the stuff done prior to shutdown (in particular to care about the running proccesses). ...
18
votes
3answers
16k views

What is the most high-performance Linux filesystem for storing a lot of small files (HDD, not SSD)?

I have a directory tree that contains many small files, and a small number of larger files. The average size of a file is about 1 kilobyte. There are 210158 files and directories in the tree (this ...
18
votes
2answers
1k views

Is dash or some other shell “faster” than bash?

I always thought that the only benefit of using dash instead of bash was that dash was smaller, and therefore many instances of dash would start faster at boot time. But I have done some research, ...
17
votes
2answers
3k views

Measure pipe throughput in the shell

There is a shell command that allows to measure how fast the data goes through it, so you can measure the speed of output of commands in a pipe. So instead of: $ somecommand | anothercommand you ...
15
votes
5answers
8k views

Is there a parallel file archiver (like tar)?

Is there something out there for parallel archiving of files? Tar is great, but I don't use tape archives, and it's more important to me that the archiving happens quickly (with compression like ...
14
votes
3answers
952 views

Why is coreutils sort slower than Python?

I wrote the following script to test the speed of Python's sort functionality: from sys import stdin, stdout lines = list(stdin) lines.sort() stdout.writelines(lines) I then compared this to the ...
14
votes
2answers
5k views

Why does high disk I/O reduce system responsiveness/performance?

I never quite understood why high disk I/O slowed the system so much. It's strange to me because I would expect the slow-down to affect only those processes dependent on the hard/optical drive data, ...
13
votes
1answer
2k views

Is disabling barriers for ext4 safe on a laptop with battery?

The manual page says about the barrier option for ext4: Write barriers enforce proper on-disk ordering of journal commits, making volatile disk write caches safe to use, at some performance ...
12
votes
1answer
3k views

Why is my TCP throughput much greater than UDP throughput?

I haven't done anything unusual to my hardware or kernel configurations (all default settings, fresh OS install, Linux kernel 3.11 TCP/IP stack) and I'm averaging about 3.83 million messages per ...
11
votes
4answers
8k views

Use of O_DIRECT on Linux

If this question is too programmer oriented, let me know. I wonder if there are people familiar with the O_DIRECT flag for the open() system call on Linux 2.6? Linus disparages its use, however high ...
11
votes
1answer
531 views

How do I go about tracking down IO usage in Linux and other *nix OS'?

Occasionally, something on my system starts hammering the disk. I would like to log data that would let me know about I/O usage per process, then I could go back to it later and see what keeps doing ...
11
votes
1answer
5k views

How can I configure cgroups to fairly share resources between users?

There used to be a kernel config option called sched_user or similar under cgroups. This allowed (to my knowledge) all users to fairly share system resources. In 2.6.35 it is not available. Is there a ...
11
votes
1answer
5k views

Using Linux cgroups to balance CPU performance

I have two dual-core Linux systems installed using Linux cgroups with relatively recent kernels; one is running Debian Squeeze, the other Ubuntu 11.04 Natty Narwhal. I've gotten CPU load balancing ...
10
votes
2answers
880 views

Why is wc so slow?

Why is the wc utility so slow? When I run it on a large file, it takes about 20 times longer than md5sum: MyDesktop:/tmp$ dd if=/dev/zero bs=1024k count=1024 of=/tmp/bigfile 1024+0 records in 1024+0 ...
10
votes
2answers
1k views

Heavy write activity on SSD nukes system performance

I've noticed that when I do heavy write applications, the whole system slows down. To test this further I ran this to do a (relatively) low-CPU, high disk activity: john -incremental > file_on_SSD ...
9
votes
4answers
11k views

Why is kworker consuming so many resources on Linux 3.0.0-12-server?

Last Friday I upgraded my Ubuntu server to 11.10, which now runs with a 3.0.0-12-server kernel. Since then the overall performance has dropped dramatically. Before the upgrade the system load was ...
9
votes
1answer
1k views

What's the progress regarding improving system performance/responsiveness during high disk I/O?

Whenever there is high disk I/O, the system tends to be much slower and less responsive than usual. What's the progress on Linux kernel regarding this? Is this problem actively being worked on?
9
votes
1answer
2k views

How to make dpkg faster?

Package (un)installation on a Debian system is horribly slow, partly because it works with a whole bunch of (small) files. Short of getting faster storage, how do I speed it up? I'm thinking maybe ...
9
votes
2answers
2k views

echo vs <<<, or Useless Use of echo in Bash Award?

By now the Useless Use of cat Award is very well known, and there's also a mention of a Useless Use of echo (not relevant for this question). I'm wondering if there should be a "Useless Use of echo in ...
9
votes
1answer
3k views

Defragmentation of LVM2 logical volumes

Question: Is there a tool supporting de-fragmentation of LVM2 logical volumes ? (Making their extents into consistent sequences) Preferably by defining desired order of extents (something like ...
9
votes
1answer
124 views

Limit on the number of btrfs snapshots?

I am considering using btrfs on my data drive so that I can use snapper, or something like snapper, to take time based snapshots. I believe this will let me browse old versions of my data. This would ...
9
votes
7answers
3k views

Processor usage increases with 4GB RAM installed

I have an Intel Atom D2700 (Synology NAS DS412+) with 4GB RAM running kernel 3.2.30 x86_64. This unit has a single DIMM slot. One thing I, and other's have found, is that when adding a 4GB DIMM ...
8
votes
2answers
6k views

Performance of unix sockets vs TCP ports

For example on php-fpm: #listen = 127.0.0.1:9000 listen = /var/run/php-fpm/php-fpm.sock Is there any major performance differences between using unix socket-based listeners over TCP ports? (Not ...
8
votes
2answers
945 views

What keeps one side of an rsync so busy?

I've got a Debian machine on my LAN serving as a backup server for the others. It has four HDDs combined into a software RAID 5 md device, on that an LVM, and on that btrfs. Backups are made using ...
7
votes
7answers
964 views

Make disk/disk copy slower

Is there a method of slowing down the copy process on Linux? I have a big file, say 10GB, and I'd like to copy it to another directory, but I don't want to copy it with full speed. Let's say I'd like ...
7
votes
2answers
7k views

Diffing two big text files

I have two big files (6GB each). They are unsorted, with linefeeds (\n) as separators. How can I diff them? It should take under 24h.
7
votes
2answers
535 views

Why is `tac file | grep foo' (piping) faster than `grep foo < <(tac file)' (process substitution)?

This question was motivated by "Reverse grepping", about grepping a huge file from bottom up. @chaos said: tac file | grep whatever Or a bit more effective: grep whatever < <(tac file) ...
7
votes
6answers
4k views

Sync writes very slow. Ubuntu 10.10, 32 bits, ext4

I am running ActiveMQ on my Macbook Pro that runs Ubuntu 10.10, 32 bits with an ext4 partition. Linux iker-laptop 2.6.35-23-generic-pae #40-Ubuntu SMP Wed Nov 17 22:32:51 UTC 2010 i686 GNU/Linux If ...
7
votes
2answers
922 views

Has anyone got any performance numbers comparing IIS and .NET to Cherokee and Mono?

I am setting up a development server and want to set it up to serve ASP.NET pages using Mono. I am planning on using Cherokee and Mono (http://www.cherokee-project.com/doc/cookbook_mono.html) and ...
7
votes
3answers
3k views

What are the exact differences between awk and cut with grep? [closed]

We know that we can get the second column of the line we want from a file using these two techniques: awk '/WORD/ { print $2 }' filename or grep WORD filename| cut -f 2 -d ' ' My questions are: ...
7
votes
2answers
92 views

How can other processes affect measurements made with `time`?

If I time a process using the time command, I get output for 'real', 'user', and 'sys'. My understanding from this discussion is that 'real' is wall time, whereas 'user' and 'sys' are process time. ...
7
votes
2answers
989 views

Replace require statements by autoload in .emacs file to improve performance

According to these posts: How can I make emacs start-up faster a-nickels-worth.blogspot.de/effective emacs one can somehow replace require and load statements in the .emacs file to speed up the ...
7
votes
1answer
2k views

Throttling web crawlers

My website is being DoS'ed by Google webspiders. Google is welcome to index my site, but sometimes it is querying a tagcloud on my site faster than my webserver can produce the results, making my ...
7
votes
3answers
791 views

How to average time commands?

I am timeing some of my commands. Now to average the measures, I'd like to run and time my commands a specified number of times, and get the results with a calculated mean and standard deviation. The ...