Questions tagged [performance]

The speed at which a program runs

Filter by
Sorted by
Tagged with
308
votes
1answer
706k views

How do you empty the buffers and cache on a Linux system?

Prior to doing some benchmarking work how would one free up the memory (RAM) that the Linux Kernel is consuming for its buffers and cache? Note that this is mostly useful for benchmarking. Emptying ...
124
votes
8answers
83k views

Why use swap when there is more than enough free space in RAM?

Using swap space instead of RAM can drastically slow down a PC. So why, when I have more than enough RAM available, does my Linux system (Arch) use the swap? Checkout my conky output below: Also, ...
117
votes
6answers
98k views

Can I configure my Linux system for more aggressive file system caching?

I am neither concerned about RAM usage (as I've got enough) nor about losing data in case of an accidental shut-down (as my power is backed, the system is reliable and the data are not critical). But ...
69
votes
5answers
44k views

Is there a way to determine the optimal value for the bs parameter to dd?

On occasion I've seen comments online along the lines of "make sure you set 'bs=' because the default value will take too long," and my own extremely-unscientific experiences of, "well that seemed to ...
59
votes
1answer
6k views

Diagram of Linux kernel vs. performance tools?

Is there a diagram that shows how the various performance tools such as ip, netstat, perf, top, ps, etc. interact with the various subsystems within the Linux kernel?
55
votes
4answers
21k views

Is dash or some other shell “faster” than bash?

I always thought that the only benefit of using dash instead of bash was that dash was smaller, and therefore many instances of dash would start faster at boot time. But I have done some research, ...
50
votes
2answers
5k views

Why is egrep [wW][oO][rR][dD] faster than grep -i word?

I've been using grep -i more often and I found out that it is slower than its egrep equivalent, where I match against the upper or lower case of each letter: $ time grep -iq "thats" testfile real ...
49
votes
4answers
5k views

Should I care about unnecessary cats?

A lot of command-line utilities can take their input either from a pipe or as a filename argument. For long shell scripts, I find starting the chain off with a cat makes it more readable, especially ...
47
votes
1answer
5k views

Why is GNU find so fast in comparison with graphical file search utilities?

I'm trying to find a file that doesn't exist in my home directory and all subdirectories. find ~/ -name "bogus" gives me that information after few seconds, yet KDE's dolphin file manager needed ...
46
votes
2answers
36k views

Performance of unix sockets vs TCP ports

For example on php-fpm: #listen = 127.0.0.1:9000 listen = /var/run/php-fpm/php-fpm.sock Is there any major performance differences between using unix socket-based listeners over TCP ports? (Not just ...
44
votes
4answers
12k views

Reverse grepping

Let's say, I have a really big text file (about 10.000.000 lines). I need to grep it from the end and save result to a file. What's the most efficient way to accomplish task?
40
votes
4answers
68k views

What is the most high-performance Linux filesystem for storing a lot of small files (HDD, not SSD)?

I have a directory tree that contains many small files, and a small number of larger files. The average size of a file is about 1 kilobyte. There are 210158 files and directories in the tree (this ...
40
votes
4answers
33k views

grep: memory exhausted

I was doing a very simple search: grep -R Milledgeville ~/Documents And after some time this error appeared: grep: memory exhausted How can I avoid this? I have 10GB of RAM on my system and few ...
38
votes
8answers
40k views

Is there a parallel file archiver (like tar)?

Is there something out there for parallel archiving of files? Tar is great, but I don't use tape archives, and it's more important to me that the archiving happens quickly (with compression like ...
37
votes
3answers
30k views

VirtualBox: Is it a bad idea to assign more virtual CPU cores than number of physical CPU cores

As I have a Hyper-Threading capable CPU, I wonder, is it a bad idea to assign more virtual CPU cores than number of physical CPU cores as the following warning suggests: Transcript: More virtual ...
35
votes
7answers
16k views

Force directory to always be in cache

I've been testing out different methods to improve the time it takes to compile my entire c++ project. Currently it takes ~5 minutes. I experimented with distcc, ccache, and others. Recently, I ...
34
votes
5answers
14k views

Measure pipe throughput in the shell

There is a shell command that allows you to measure how fast the data goes through it, so you can measure the speed of output of commands in a pipe. So instead of: $ somecommand | anothercommand you ...
31
votes
5answers
40k views

How to speed up X over SSH on a slow network connection?

Are there any specific recommendations on speeding up X applications over ssh on a slow network connection? In this specific case, I am accessing a server located in west coast from a laptop in east ...
30
votes
2answers
38k views

Diffing two big text files

I have two big files (6GB each). They are unsorted, with linefeeds (\n) as separators. How can I diff them? It should take under 24h.
30
votes
3answers
39k views

What are the exact differences between awk and cut with grep? [closed]

We know that we can get the second column of the line we want from a file using these two techniques: awk '/WORD/ { print $2 }' filename or grep WORD filename| cut -f 2 -d ' ' My questions are: ...
30
votes
12answers
5k views

How can I tweak my Linux desktop to be more responsive?

I have the experience that Linux works fine until the physical memory is exhausted. As soon as swap space is used the performance is severely degraded and the GUI becomes unresponsive. This problem ...
29
votes
4answers
3k views

python vs bc in evaluating 6^6^6

I am evaluating the expression 6^6^6 using python and bc separately. The content of the python file is print 6**6**6. When I execute time python test.py, I get the output as real 0m0.067s ...
28
votes
11answers
6k views

Why doesn't shell automatically fix “useless use of cat”? [closed]

Many people use oneliners and scripts containing code along the lines cat "$MYFILE" | command1 | command2 > "$OUTPUT" The first cat is often called "useless use of cat" because technically it ...
28
votes
1answer
3k views

Why does “ls *” take so much longer than “ls”?

I have a couple of files in a directory: $ ls | wc -l 9376 Can anybody explain why there is such a huge time difference in using ls * and ls? $ time ls > /dev/null real 0m0.118s user 0m0....
28
votes
5answers
19k views

How do I minimize disk space usage

One of my machines is the 2GB EeePC Surf, a neat netbook with very limited resources. So limited that right now, I have 22MB free space left. On it, I'm running Arch Linux with the Openbox DE and a ...
27
votes
9answers
5k views

for vs find in Bash

When looping through files there are two ways: use a for-loop: for f in *; do echo "$f" done use find: find * -prune | while read f; do echo "$f" done Assuming these two loops will find ...
26
votes
8answers
16k views

Make disk/disk copy slower

Is there a method of slowing down the copy process on Linux? I have a big file, say 10GB, and I'd like to copy it to another directory, but I don't want to copy it with full speed. Let's say I'd like ...
26
votes
5answers
5k views

Why is iterating over a file twice faster than reading it into memory and computing twice?

I'm comparing the following tail -n 1000000 stdout.log | grep -c '"success": true' tail -n 1000000 stdout.log | grep -c '"success": false' with the following log=$(tail -n 1000000 stdout.log) echo "...
26
votes
2answers
5k views

Is it possible to trigger OOM-killer on forced swapping?

Is it possible to have the system preemptively swap out inactive pages (vm.swappiness), but invoke the oom-killer when the system runs out of RAM (as opposed to running out of memory) and is forced to ...
25
votes
8answers
3k views

Is piping, shifting, or parameter expansion more efficient?

I'm trying to find the most efficient way to iterate through certain values that are a consistent number of values away from each other in a space separated list of words(I don't want to use an array)....
25
votes
11answers
19k views

Lightest way to run GUI applications in Linux? [closed]

I realized I can use a window manager without a desktop environment and don't lose any functionality I care about. Applications still work fine, I can still arrange application windows as I please. ...
25
votes
4answers
3k views

How to limit resource usage to save CPU+RAM for a certain process?

I have a dev server on which sshd sometimes stops working because the machine runs out of RAM. Yes we are running low on memory and upgrading is not a feasible choice right now. What I want to do is ...
24
votes
5answers
19k views

Can I just disable updatedb?

Is updatedb necessary at all? I never use locate and my servers tend to have dozens of millions of files which usually makes updatedb to run for a long time and consume I/O needed by MySQL and/or ...
23
votes
4answers
2k views

Are Linux utilities smart when running piped commands?

I was just running a few commands in a terminal and I started wondering, does Unix/Linux take shortcuts when running piped commands? For example, let's say I have a file with one million lines, the ...
23
votes
4answers
30k views

Benchmark ssd on linux: How to measure the same things as crystaldiskmark does in windows

I want to benchmark a ssd (possibly with encrypted filesystems) and compare it to benchmarks done by crystaldiskmark on windows. So how can I measure approximately the same things as crystaldiskmark ...
23
votes
1answer
8k views

How to ensure exclusive CPU availability for a running process?

At first, the question seems to be a little bit silly/confusing as the OS does the job of managing process execution. However, I want to measure how much some processes are CPU/IO-bound and I feel ...
22
votes
5answers
24k views

Use of O_DIRECT on Linux

If this question is too programmer oriented, let me know. I wonder if there are people familiar with the O_DIRECT flag for the open() system call on Linux 2.6? Linus disparages its use, however high ...
22
votes
2answers
3k views

Why does the `updatedb` program run so fast?

Usually when I have programs that are doing a full disk scan and going over all files in the system they take a very long time to run. Why does updatedb run so fast in comparison?
22
votes
3answers
5k views

Practical limit on the number of btrfs snapshots?

I am considering using btrfs on my data drive so that I can use snapper, or something like snapper, to take time based snapshots. I believe this will let me browse old versions of my data. This would ...
21
votes
3answers
16k views

Why is rm slow on an external storage drive (USB-connected, type fuseblk) with 50Gb of files?

I have been trying to use rsnapshot for making backups, but I'm finding it unusable. While it is able to diff a directory (50gb) and duplicate it (hardlinking every file) in a few minutes, and I can ...
21
votes
3answers
20k views

Is there an open source tool to measure cpu performance?

I have used many times ab for measuring web performance, hdparm for measuring hard disk performance and netperf for measuring network performance. But I didn't find any tools to measure cpu ...
20
votes
4answers
7k views

Is there a fastest way to shutdown the system?

Or is shutdown -h now the fastest it can get? I look for some syscall or similar that will allow to skip lots of the stuff done prior to shutdown (in particular to care about the running proccesses). ...
20
votes
3answers
4k views

Why is coreutils sort slower than Python?

I wrote the following script to test the speed of Python's sort functionality: from sys import stdin, stdout lines = list(stdin) lines.sort() stdout.writelines(lines) I then compared this to the ...
20
votes
1answer
7k views

Abysmal general dm-crypt (LUKS) write performance

I am investigating a problem where encrypting a block device imposes a huge performance penalty when writing to it. Hours of Internet reading and experiments did not provide me with a proper ...
19
votes
3answers
12k views

Why load is high despite the fact that neither CPU or disk is overused

I'm getting the following output from top: Cpu(s): 43.8%us, 32.5%sy, 4.8%ni, 2.0%id, 15.6%wa, 0.2%hi, 1.2%si, 0.0%st Mem: 16331504k total, 15759412k used, 572092k free, 4575980k buffers Swap:...
19
votes
2answers
3k views

What does grep do when it's not running the CPU?

When looking for matches with grep, I often notice that the subsequent search takes significantly less time than the first-- e.g. 25s vs. 2s. Obviously, it's not by reusing the data structures from ...
19
votes
2answers
9k views

Why does high disk I/O reduce system responsiveness/performance?

I never quite understood why high disk I/O slowed the system so much. It's strange to me because I would expect the slow-down to affect only those processes dependent on the hard/optical drive data, ...
19
votes
3answers
7k views

What is the IPsec bottleneck in Linux?

I am trying to compare the performance of several network security protocols between two hosts connected in Gigabit Ethernet. My goal here is to see if I can saturate my bandwidth, and if not, what ...
18
votes
4answers
42k views

Optimizing logical sector size for physical sector size 4096 HDD

With many new hard drive disks the physical sector size is 4096. Would it be possible to make the system use a logical sector size of the same size, rather than the default logical sector size of 512? ...
18
votes
4answers
26k views

Why is kworker consuming so many resources on Linux 3.0.0-12-server?

Last Friday I upgraded my Ubuntu server to 11.10, which now runs with a 3.0.0-12-server kernel. Since then the overall performance has dropped dramatically. Before the upgrade the system load was ...