Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [performance]

The speed at which a program runs

Filter by
Sorted by
Tagged with
71
votes
5answers
45k views

Is there a way to determine the optimal value for the bs parameter to dd?

On occasion I've seen comments online along the lines of "make sure you set 'bs=' because the default value will take too long," and my own extremely-unscientific experiences of, "well that seemed to ...
316
votes
1answer
718k views

How do you empty the buffers and cache on a Linux system?

Prior to doing some benchmarking work how would one free up the memory (RAM) that the Linux Kernel is consuming for its buffers and cache? Note that this is mostly useful for benchmarking. Emptying ...
125
votes
8answers
85k views

Why use swap when there is more than enough free space in RAM?

Using swap space instead of RAM can drastically slow down a PC. So why, when I have more than enough RAM available, does my Linux system (Arch) use the swap? Checkout my conky output below: Also, ...
49
votes
4answers
6k views

Should I care about unnecessary cats?

A lot of command-line utilities can take their input either from a pipe or as a filename argument. For long shell scripts, I find starting the chain off with a cat makes it more readable, especially ...
24
votes
1answer
9k views

How to ensure exclusive CPU availability for a running process?

At first, the question seems to be a little bit silly/confusing as the OS does the job of managing process execution. However, I want to measure how much some processes are CPU/IO-bound and I feel ...
42
votes
4answers
34k views

grep: memory exhausted

I was doing a very simple search: grep -R Milledgeville ~/Documents And after some time this error appeared: grep: memory exhausted How can I avoid this? I have 10GB of RAM on my system and few ...
20
votes
3answers
12k views

Why load is high despite the fact that neither CPU or disk is overused

I'm getting the following output from top: Cpu(s): 43.8%us, 32.5%sy, 4.8%ni, 2.0%id, 15.6%wa, 0.2%hi, 1.2%si, 0.0%st Mem: 16331504k total, 15759412k used, 572092k free, 4575980k buffers Swap:...
5
votes
1answer
189 views

System lags when doing large R/W operations on external disks

I am having some issues with system-wide latency/lagging when doing large disk imaging operations on an Ubuntu 18.04 system. Here's the system specs: Processor: Intel Core i7 (never near capacity on ...
35
votes
5answers
15k views

Measure pipe throughput in the shell

There is a shell command that allows you to measure how fast the data goes through it, so you can measure the speed of output of commands in a pipe. So instead of: $ somecommand | anothercommand you ...
25
votes
11answers
20k views

Lightest way to run GUI applications in Linux? [closed]

I realized I can use a window manager without a desktop environment and don't lose any functionality I care about. Applications still work fine, I can still arrange application windows as I please. ...
31
votes
5answers
41k views

How to speed up X over SSH on a slow network connection?

Are there any specific recommendations on speeding up X applications over ssh on a slow network connection? In this specific case, I am accessing a server located in west coast from a laptop in east ...
25
votes
4answers
3k views

How to limit resource usage to save CPU+RAM for a certain process?

I have a dev server on which sshd sometimes stops working because the machine runs out of RAM. Yes we are running low on memory and upgrading is not a feasible choice right now. What I want to do is ...
13
votes
4answers
20k views

Grep in couple thousands files

I have directory with cca 26 000 files and I need to grep in all these files. Problem is, that I need it as fast as possible, so it's not ideal to make script where grep will take name of one file ...
7
votes
1answer
2k views

Performance differences between pipelines and process substitution

I tend to use pipelines in my bash scripts over process substitution in most situations, especially in cases of using multiple sets of commands as it seems more readable to do ... | ... | ... over ... ...
23
votes
3answers
5k views

Practical limit on the number of btrfs snapshots?

I am considering using btrfs on my data drive so that I can use snapper, or something like snapper, to take time based snapshots. I believe this will let me browse old versions of my data. This would ...
23
votes
4answers
2k views

Are Linux utilities smart when running piped commands?

I was just running a few commands in a terminal and I started wondering, does Unix/Linux take shortcuts when running piped commands? For example, let's say I have a file with one million lines, the ...
1
vote
1answer
343 views

Why is `sync + drop_caches` not dropping caches?

I have a test case for journalctl where it spends several seconds reading from the disk. But if I try to benchmark multiple runs of the test case, I find that it's impossibly fast after the first run....
119
votes
6answers
100k views

Can I configure my Linux system for more aggressive file system caching?

I am neither concerned about RAM usage (as I've got enough) nor about losing data in case of an accidental shut-down (as my power is backed, the system is reliable and the data are not critical). But ...
59
votes
1answer
6k views

Diagram of Linux kernel vs. performance tools?

Is there a diagram that shows how the various performance tools such as ip, netstat, perf, top, ps, etc. interact with the various subsystems within the Linux kernel?
56
votes
4answers
22k views

Is dash or some other shell “faster” than bash?

I always thought that the only benefit of using dash instead of bash was that dash was smaller, and therefore many instances of dash would start faster at boot time. But I have done some research, ...
42
votes
4answers
70k views

What is the most high-performance Linux filesystem for storing a lot of small files (HDD, not SSD)?

I have a directory tree that contains many small files, and a small number of larger files. The average size of a file is about 1 kilobyte. There are 210158 files and directories in the tree (this ...
26
votes
2answers
5k views

Is it possible to trigger OOM-killer on forced swapping?

Is it possible to have the system preemptively swap out inactive pages (vm.swappiness), but invoke the oom-killer when the system runs out of RAM (as opposed to running out of memory) and is forced to ...
19
votes
4answers
43k views

Optimizing logical sector size for physical sector size 4096 HDD

With many new hard drive disks the physical sector size is 4096. Would it be possible to make the system use a logical sector size of the same size, rather than the default logical sector size of 512? ...
31
votes
2answers
39k views

Diffing two big text files

I have two big files (6GB each). They are unsorted, with linefeeds (\n) as separators. How can I diff them? It should take under 24h.
26
votes
5answers
5k views

Why is iterating over a file twice faster than reading it into memory and computing twice?

I'm comparing the following tail -n 1000000 stdout.log | grep -c '"success": true' tail -n 1000000 stdout.log | grep -c '"success": false' with the following log=$(tail -n 1000000 stdout.log) echo "...
9
votes
1answer
1k views

What's the progress regarding improving system performance/responsiveness during high disk I/O?

Whenever there is high disk I/O, the system tends to be much slower and less responsive than usual. What's the progress on Linux kernel regarding this? Is this problem actively being worked on?
7
votes
2answers
2k views

Why is `tac file | grep foo' (piping) faster than `grep foo < <(tac file)' (process substitution)?

This question was motivated by "Reverse grepping", about grepping a huge file from bottom up. @chaos said: tac file | grep whatever Or a bit more effective: grep whatever < <(tac file) ...
6
votes
3answers
2k views

Fast way to build a test file with every second listed in YYYY-mm-dd HH:MM:SS format

I want to create a large test file with lines containg dates listed by the second, but my method is taking inordinately long... (or at least, that's how it feels :) ... 43 minutes to create only ...
0
votes
1answer
86 views

Why does rename() take longer when fsync() is called first?

In this test, why does rename() take longer when fsync() is called first? Environment: btrfs, mechanical HDD, Debian 9 container, running on kernel 5.0.17-200.fc29.x86_64. Test command: dpkg -r ...
4
votes
3answers
4k views

Linux, SSD and swap

This article suggests not to create a swap space on an SSD drive in order to reduce its wear. On the other hand, it suggested here that swap space positively affects performance. Would it be a better ...
4
votes
4answers
864 views

Grep of many keywords over many files: speeding it up

I am currently facing a "performance problem" while using grep. I am trying to locate the occurrences of many (10,000+) keywords in many (think Linux kernel repository size) files. The objective is to ...
3
votes
1answer
852 views

What are the pros and cons of badblock's two non-destructive tests?

I use badblocks to test my 32GB class-10 microSD card that I use to boot my RPi. I already have a functioning file system on it, so I don't want to scan it with the -w option (destructive read-write ...
20
votes
4answers
7k views

Is there a fastest way to shutdown the system?

Or is shutdown -h now the fastest it can get? I look for some syscall or similar that will allow to skip lots of the stuff done prior to shutdown (in particular to care about the running proccesses). ...
25
votes
4answers
32k views

Benchmark ssd on linux: How to measure the same things as crystaldiskmark does in windows

I want to benchmark a ssd (possibly with encrypted filesystems) and compare it to benchmarks done by crystaldiskmark on windows. So how can I measure approximately the same things as crystaldiskmark ...
44
votes
4answers
12k views

Reverse grepping

Let's say, I have a really big text file (about 10.000.000 lines). I need to grep it from the end and save result to a file. What's the most efficient way to accomplish task?
30
votes
12answers
5k views

How can I tweak my Linux desktop to be more responsive?

I have the experience that Linux works fine until the physical memory is exhausted. As soon as swap space is used the performance is severely degraded and the GUI becomes unresponsive. This problem ...
9
votes
1answer
6k views

What are Kernel PMU event-s in perf_events list?

Searching for what one can monitor with perf_events on Linux, I cannot find what Kernel PMU event are? Namely, with perf version 3.13.11-ckt39 the perf list shows events like: branch-instructions OR ...
30
votes
3answers
41k views

What are the exact differences between awk and cut with grep? [closed]

We know that we can get the second column of the line we want from a file using these two techniques: awk '/WORD/ { print $2 }' filename or grep WORD filename| cut -f 2 -d ' ' My questions are: ...
10
votes
3answers
27k views

Bash script; optimization of processing speed

I was wondering if there are general guidelines for optimizing Bash scripts. For example, it is more convenient to write loops than lines of commands, but is it also faster to process for the system? ...
5
votes
1answer
3k views

Efficient way to print lines from a massive file using awk, sed, or something else? [duplicate]

If I had a plain-text file containing 8 million lines and I want to print out lines 4,000,010 to 4,000,000 to the screen, which would be more efficient: awk or sed? There is no pattern to the text, ...
11
votes
1answer
883 views

Why doesn't `strace` show this process is waiting for something?

The mighty strace has let me down. How is this possible? time foo shows that foo takes several seconds to run ("real"), but uses negligible cpu time, both in userspace ("user") and in the kernel ("...
5
votes
4answers
26k views

Authentication very slow (both with ssh and su)

I'm having a problem on one of our production servers where the login attempts become progressively slower as time passes. After roughly 5 days it becomes so slow that some critical processes/cron ...
3
votes
3answers
3k views

Optimal SSD configuration

I am thinking about buying an SSD for my Debian system. However, because I don't know what is the optimal arrangement (i.e. what goes on the SSD), I am not sure what is the optimal size. Obviously I ...
15
votes
1answer
1k views

How do I go about tracking down IO usage in Linux and other *nix OS'?

Occasionally, something on my system starts hammering the disk. I would like to log data that would let me know about I/O usage per process, then I could go back to it later and see what keeps doing ...
7
votes
2answers
600 views

How can other processes affect measurements made with `time`?

If I time a process using the time command, I get output for 'real', 'user', and 'sys'. My understanding from this discussion is that 'real' is wall time, whereas 'user' and 'sys' are process time. ...
29
votes
5answers
20k views

How do I minimize disk space usage

One of my machines is the 2GB EeePC Surf, a neat netbook with very limited resources. So limited that right now, I have 22MB free space left. On it, I'm running Arch Linux with the Openbox DE and a ...
22
votes
5answers
24k views

Use of O_DIRECT on Linux

If this question is too programmer oriented, let me know. I wonder if there are people familiar with the O_DIRECT flag for the open() system call on Linux 2.6? Linus disparages its use, however high ...
28
votes
8answers
17k views

Make disk/disk copy slower

Is there a method of slowing down the copy process on Linux? I have a big file, say 10GB, and I'd like to copy it to another directory, but I don't want to copy it with full speed. Let's say I'd like ...
21
votes
3answers
16k views

Why is rm slow on an external storage drive (USB-connected, type fuseblk) with 50Gb of files?

I have been trying to use rsnapshot for making backups, but I'm finding it unusable. While it is able to diff a directory (50gb) and duplicate it (hardlinking every file) in a few minutes, and I can ...
20
votes
3answers
4k views

Why is coreutils sort slower than Python?

I wrote the following script to test the speed of Python's sort functionality: from sys import stdin, stdout lines = list(stdin) lines.sort() stdout.writelines(lines) I then compared this to the ...