The speed at which a program runs

learn more… | top users | synonyms (1)

92
votes
7answers
31k views

Why use swap when there is more than enough free space in RAM?

Using swap space instead of RAM can drastically slow down a PC. So why, when I have more than enough RAM available, does my Linux system (Arch) use the swap? Checkout my conky output below: Also, ...
75
votes
3answers
142k views

How do you empty the buffers and cache on a Linux system?

Prior to doing some benchmarking work how would one free up the memory (RAM) that the Linux Kernel is consuming for its buffers and cache?
55
votes
5answers
35k views

Can I configure my Linux system for more aggressive file system caching?

I am neither concerned about RAM usage (as I've got enough) nor about losing data in case of an accidental shut-down (as my power is backed, the system is reliable and the data are not critical). But ...
50
votes
1answer
3k views

Diagram of Linux kernel vs. performance tools?

Is there a diagram that shows how the various performance tools such as ip, netstat, perf, top, ps, etc. interact with the various subsystems within the Linux kernel?
39
votes
4answers
18k views

Is there a way to determine the optimal value for the bs parameter to dd?

On occasion I've seen comments online along the lines of "make sure you set 'bs=' because the default value will take too long," and my own extremely-unscientific experiences of, "well that seemed to ...
36
votes
4answers
4k views

Reverse grepping

Let's say, I have a really big text file (about 10.000.000 lines). I need to grep it from the end and save result to a file. What's the most efficient way to accomplish task?
29
votes
3answers
2k views

Should I care about unnecessary cats?

A lot of command-line utilities can take their input either from a pipe or as a filename argument. For long shell scripts, I find starting the chain off with a cat makes it more readable, especially ...
28
votes
4answers
2k views

python vs bc in evaluating 6^6^6

I am evaluating the expression 6^6^6 using python and bc separately. The content of the python file is print 6**6**6. When I execute time python test.py, I get the output as real 0m0.067s ...
27
votes
7answers
7k views

Force directory to always be in cache

I've been testing out different methods to improve the time it takes to compile my entire c++ project. Currently it takes ~5 minutes. I experimented with distcc, ccache, and others. Recently, I ...
26
votes
1answer
2k views

Why does “ls *” take so much longer than “ls”?

I have a couple of files in a directory: $ ls | wc -l 9376 Can anybody explain why there is such a huge time difference in using ls * and ls? $ time ls > /dev/null real 0m0.118s user ...
26
votes
12answers
3k views

How can I tweak my Linux desktop to be more responsive?

I have the experience that Linux works fine until the physical memory is exhausted. As soon as swap space is used the performance is severely degraded and the GUI becomes unresponsive. This problem ...
25
votes
5answers
3k views

Why is iterating over a file twice faster than reading it into memory and computing twice?

I'm comparing the following tail -n 1000000 stdout.log | grep -c '"success": true' tail -n 1000000 stdout.log | grep -c '"success": false' with the following log=$(tail -n 1000000 stdout.log) echo ...
25
votes
3answers
7k views

grep: memory exhausted

I was doing a very simple search: grep -R Milledgeville ~/Documents And after some time this error appeared: grep: memory exhausted How can I avoid this? I have 10GB of RAM on my system and few ...
24
votes
9answers
1k views

for vs find in Bash

When looping through files there are two ways: 1. use a for-loop for f in *; do echo "$f" done 2. use find find * -prune | while read f; do echo "$f" done Assuming these two loops will ...
23
votes
4answers
18k views

How to speed up X over SSH on a slow network connection?

Are there any specific recommendations on speeding up X applications over ssh on a slow network connection? In this specific case, I am accessing a server located in west coast from a laptop in east ...
23
votes
4answers
2k views

How to limit resource usage to save CPU+RAM for a certain process?

I have a dev server on which sshd sometimes stops working because the machine runs out of RAM. Yes we are running low on memory and upgrading is not a feasible choice right now. What I want to do is ...
22
votes
4answers
1k views

Are Linux utilities smart when running piped commands?

I was just running a few commands in a terminal and I started wondering, does Unix/Linux take shortcuts when running piped commands? For example, let's say I have a file with one million lines, the ...
21
votes
11answers
10k views

the lightest way to have a GUI in Linux? [closed]

I have been using Gnome and KDE for a while, and I am now running Compiz standalone... Regarding system resource, I think the most heavy ones are feature-rich desktop environments (Gnome, KDE), then ...
21
votes
3answers
3k views

Is dash or some other shell “faster” than bash?

I always thought that the only benefit of using dash instead of bash was that dash was smaller, and therefore many instances of dash would start faster at boot time. But I have done some research, ...
19
votes
3answers
4k views

Measure pipe throughput in the shell

There is a shell command that allows to measure how fast the data goes through it, so you can measure the speed of output of commands in a pipe. So instead of: $ somecommand | anothercommand you ...
19
votes
3answers
23k views

What is the most high-performance Linux filesystem for storing a lot of small files (HDD, not SSD)?

I have a directory tree that contains many small files, and a small number of larger files. The average size of a file is about 1 kilobyte. There are 210158 files and directories in the tree (this ...
18
votes
5answers
10k views

Is there a parallel file archiver (like tar)?

Is there something out there for parallel archiving of files? Tar is great, but I don't use tape archives, and it's more important to me that the archiving happens quickly (with compression like ...
18
votes
4answers
2k views

Is there a fastest way to shutdown the system?

Or is shutdown -h now the fastest it can get? I look for some syscall or similar that will allow to skip lots of the stuff done prior to shutdown (in particular to care about the running proccesses). ...
18
votes
3answers
2k views

Why is coreutils sort slower than Python?

I wrote the following script to test the speed of Python's sort functionality: from sys import stdin, stdout lines = list(stdin) lines.sort() stdout.writelines(lines) I then compared this to the ...
16
votes
2answers
6k views

Why does high disk I/O reduce system responsiveness/performance?

I never quite understood why high disk I/O slowed the system so much. It's strange to me because I would expect the slow-down to affect only those processes dependent on the hard/optical drive data, ...
15
votes
2answers
10k views

Performance of unix sockets vs TCP ports

For example on php-fpm: #listen = 127.0.0.1:9000 listen = /var/run/php-fpm/php-fpm.sock Is there any major performance differences between using unix socket-based listeners over TCP ports? (Not ...
15
votes
3answers
8k views

How do I minimize disk space usage

One of my machines is the 2GB EeePC Surf, a neat netbook with very limited resources. So limited that right now, I have 22MB free space left. On it, I'm running Arch Linux with the Openbox DE and a ...
14
votes
3answers
431 views

What is the IPsec bottleneck in Linux?

I am trying to compare the performance of several network security protocols between two hosts connected in Gigabit Ethernet. My goal here is to see if I can saturate my bandwidth, and if not, what ...
13
votes
1answer
2k views

Is disabling barriers for ext4 safe on a laptop with battery?

The manual page says about the barrier option for ext4: Write barriers enforce proper on-disk ordering of journal commits, making volatile disk write caches safe to use, at some performance ...
12
votes
1answer
6k views

Why is my TCP throughput much greater than UDP throughput?

I haven't done anything unusual to my hardware or kernel configurations (all default settings, fresh OS install, Linux kernel 3.11 TCP/IP stack) and I'm averaging about 3.83 million messages per ...
12
votes
2answers
1k views

Why is wc so slow?

Why is the wc utility so slow? When I run it on a large file, it takes about 20 times longer than md5sum: MyDesktop:/tmp$ dd if=/dev/zero bs=1024k count=1024 of=/tmp/bigfile 1024+0 records in 1024+0 ...
12
votes
4answers
14k views

Why is kworker consuming so many resources on Linux 3.0.0-12-server?

Last Friday I upgraded my Ubuntu server to 11.10, which now runs with a 3.0.0-12-server kernel. Since then the overall performance has dropped dramatically. Before the upgrade the system load was ...
11
votes
4answers
10k views

Use of O_DIRECT on Linux

If this question is too programmer oriented, let me know. I wonder if there are people familiar with the O_DIRECT flag for the open() system call on Linux 2.6? Linus disparages its use, however high ...
11
votes
2answers
2k views

Heavy write activity on SSD nukes system performance

I've noticed that when I do heavy write applications, the whole system slows down. To test this further I ran this to do a (relatively) low-CPU, high disk activity: john -incremental > file_on_SSD ...
11
votes
1answer
6k views

How can I configure cgroups to fairly share resources between users?

There used to be a kernel config option called sched_user or similar under cgroups. This allowed (to my knowledge) all users to fairly share system resources. In 2.6.35 it is not available. Is there a ...
11
votes
1answer
551 views

How do I go about tracking down IO usage in Linux and other *nix OS'?

Occasionally, something on my system starts hammering the disk. I would like to log data that would let me know about I/O usage per process, then I could go back to it later and see what keeps doing ...
11
votes
1answer
2k views

How to make dpkg faster?

Package (un)installation on a Debian system is horribly slow, partly because it works with a whole bunch of (small) files. Short of getting faster storage, how do I speed it up? I'm thinking maybe ...
11
votes
1answer
4k views

Defragmentation of LVM2 logical volumes

Question: Is there a tool supporting de-fragmentation of LVM2 logical volumes ? (Making their extents into consistent sequences) Preferably by defining desired order of extents (something like ...
11
votes
1answer
6k views

Using Linux cgroups to balance CPU performance

I have two dual-core Linux systems installed using Linux cgroups with relatively recent kernels; one is running Debian Squeeze, the other Ubuntu 11.04 Natty Narwhal. I've gotten CPU load balancing ...
10
votes
1answer
543 views

Limit on the number of btrfs snapshots?

I am considering using btrfs on my data drive so that I can use snapper, or something like snapper, to take time based snapshots. I believe this will let me browse old versions of my data. This would ...
9
votes
1answer
1k views

What's the progress regarding improving system performance/responsiveness during high disk I/O?

Whenever there is high disk I/O, the system tends to be much slower and less responsive than usual. What's the progress on Linux kernel regarding this? Is this problem actively being worked on?
9
votes
2answers
3k views

echo vs <<<, or Useless Use of echo in Bash Award?

By now the Useless Use of cat Award is very well known, and there's also a mention of a Useless Use of echo (not relevant for this question). I'm wondering if there should be a "Useless Use of echo in ...
9
votes
7answers
4k views

Processor usage increases with 4GB RAM installed

I have an Intel Atom D2700 (Synology NAS DS412+) with 4GB RAM running kernel 3.2.30 x86_64. This unit has a single DIMM slot. One thing I, and other's have found, is that when adding a 4GB DIMM ...
9
votes
2answers
1k views

What keeps one side of an rsync so busy?

I've got a Debian machine on my LAN serving as a backup server for the others. It has four HDDs combined into a software RAID 5 md device, on that an LVM, and on that btrfs. Backups are made using ...
8
votes
7answers
2k views

Make disk/disk copy slower

Is there a method of slowing down the copy process on Linux? I have a big file, say 10GB, and I'd like to copy it to another directory, but I don't want to copy it with full speed. Let's say I'd like ...
8
votes
3answers
2k views

Transfering millions of files from one server to another

I have two servers. One of them has 15 million text files (about 40 GB). I am trying to transfer them to another server. I considered zipping them and transferring the archive. But I realized that ...
8
votes
3answers
8k views

What are the exact differences between awk and cut with grep? [closed]

We know that we can get the second column of the line we want from a file using these two techniques: awk '/WORD/ { print $2 }' filename or grep WORD filename| cut -f 2 -d ' ' My questions are: ...
8
votes
4answers
4k views

Can I just disable updatedb?

Is updatedb necessary at all? I never use locate and my servers tend to have dozens of millions of files which usually makes updatedb to run for a long time and consume I/O needed by MySQL and/or ...
8
votes
5answers
12k views

Why is this network connection so slow?

I am having some problems with network performance speed on a Linux server running Ubuntu 9.10. Transfer speeds on all types of traffic are around 1.5MB/s on a 1000mbit/s wired ethernet connection. ...
8
votes
1answer
3k views

Throttling web crawlers

My website is being DoS'ed by Google webspiders. Google is welcome to index my site, but sometimes it is querying a tagcloud on my site faster than my webserver can produce the results, making my ...