I/O is an acronym for Input/Output, which refers to the mechanism or process for the transfer of information between one system or component and another.

learn more… | top users | synonyms

42
votes
5answers
3k views

Redirecting stdout to a file you don't have write permission on

When you attempt to modify a file without having write permissions on it, you get an error: > touch /tmp/foo && sudo chown root /tmp/foo > echo test > /tmp/foo zsh: permission ...
25
votes
12answers
3k views

How can I tweak my Linux desktop to be more responsive?

I have the experience that Linux works fine until the physical memory is exhausted. As soon as swap space is used the performance is severely degraded and the GUI becomes unresponsive. This problem ...
23
votes
5answers
3k views

Why is iterating over a file twice faster than reading it into memory and computing twice?

I'm comparing the following tail -n 1000000 stdout.log | grep -c '"success": true' tail -n 1000000 stdout.log | grep -c '"success": false' with the following log=$(tail -n 1000000 stdout.log) echo ...
21
votes
3answers
1k views

Can the “du” program be made less aggressive?

We have a regular job that does du summaries of a number of subdirectories, picking out worst offenders, and use the output to find if there are things that are rapidly rising to spot potential ...
21
votes
3answers
733 views

How can I limit the output speed of stdout?

I'm running CentOS 5.7 and I have a backup utility that has the option of dumping its backup file to stdout. The backup file is rather large (multiple gigabytes). The target is an SSHFS filesystem. To ...
20
votes
2answers
4k views

When is dd suitable for copying data? (or, when are read() and write() partial)

Short version: In what circumstances is dd safe to use for copying data, safe meaning that there is no risk of corruption due to a partial read or write? Long version — preamble: dd is often used to ...
16
votes
1answer
3k views

Meaning of colours in htop

Given the option: [ ] Detailed CPU time (System/IO-Wait/Hard-IRQ/Soft-IRQ/Steal/Guest) In htop (Version 1.0.1) what are the meanings of the colours? In the following image, I'm seeing rather a lot ...
15
votes
2answers
3k views

Measure pipe throughput in the shell

There is a shell command that allows to measure how fast the data goes through it, so you can measure the speed of output of commands in a pipe. So instead of: $ somecommand | anothercommand you ...
14
votes
2answers
1k views

Portability of “> /dev/stdout”

Occasionally I need to specify a "path-equivalent" of one of the standard IO streams (stdin, stdout, stderr). Since 99% of the time I work with Linux, I just prepend /dev/ to get /dev/stdin, etc., ...
14
votes
3answers
2k views

Determining Specific File Responsible for High I/O

This is a simple problem but the first time I've ever had to actually fix it: finding which specific files/inodes are the targets of the most I/O. I'd like to be able to get a general system overview, ...
13
votes
2answers
3k views

Can I watch the progress of a `sync` operation?

I've copied a large file to a USB disk mounted on a Linux system with async. This returns to a command prompt relatively quickly, but when I type sync, of course, it all has to go to disk, and that ...
13
votes
2answers
5k views

Why does high disk I/O reduce system responsiveness/performance?

I never quite understood why high disk I/O slowed the system so much. It's strange to me because I would expect the slow-down to affect only those processes dependent on the hard/optical drive data, ...
12
votes
5answers
36k views

How can I read line by line from a variable in bash?

I have a variable which contains multiline output of a command. What's the most effecient way to read the output line by line from the variable? For example: jobs="$(jobs)" if [ "$jobs" ]; then ...
12
votes
2answers
1k views

Weight-based block IO control for VMs

I use KVM to manage my virtual machines. I am trying to limit the resources provided for VMs. I am able to limit the CPU and memory allocated for VMs using libvirt/cgroups. Now I want to control the ...
11
votes
1answer
529 views

How do I go about tracking down IO usage in Linux and other *nix OS'?

Occasionally, something on my system starts hammering the disk. I would like to log data that would let me know about I/O usage per process, then I could go back to it later and see what keeps doing ...
10
votes
2answers
5k views

How to Throttle per process I/O to a max limit?

I'm looking for a way to limit a processes disk io to a set speed limit. Ideally the program would work similar to this: $ limitio --pid 32423 --write-limit 1M Limiting process 32423 to 1 megabyte ...
10
votes
1answer
155 views

Moving a file to a file that is being read

When a file is moved and the destination file already exists and is currently being read, will the operation succeed? Will it depend on how long the reading operation on the destination file lasts?
9
votes
4answers
555 views

Utility to buffer an unbounded amount of data in a pipeline?

Is there a utility that I can stick in a pipeline to decouple read and write speeds? $ producer | buf | consumer Basically, I want a utility buf that reads its input as fast as possible, storing it ...
9
votes
1answer
1k views

What's the progress regarding improving system performance/responsiveness during high disk I/O?

Whenever there is high disk I/O, the system tends to be much slower and less responsive than usual. What's the progress on Linux kernel regarding this? Is this problem actively being worked on?
9
votes
1answer
1k views

Measuring disk I/O usage of a program

time is a brilliant command if you want to figure out how much CPU time a given command takes. I am looking for something similar that can measure the disk I/O of the program and any children. ...
8
votes
2answers
7k views

Run process with higher priority

I want to run mplayer with higher priority than any other processes, including the IO-processes. How can I do that?
8
votes
2answers
1k views

Heavy write activity on SSD nukes system performance

I've noticed that when I do heavy write applications, the whole system slows down. To test this further I ran this to do a (relatively) low-CPU, high disk activity: john -incremental > file_on_SSD ...
8
votes
2answers
870 views

What keeps one side of an rsync so busy?

I've got a Debian machine on my LAN serving as a backup server for the others. It has four HDDs combined into a software RAID 5 md device, on that an LVM, and on that btrfs. Backups are made using ...
7
votes
3answers
10k views

Find out what processes are writing to hard drive

On my Lenovo T400 and Ubuntu, the light for hard drive writing keeps flashing. I was wondering if in Linux it is possible to find out what processes are doing I/O to the hard drive? Just like by top, ...
7
votes
3answers
3k views

How can I monitor disk I/O in a particular directory?

I've got a few processes with a known name that all write to files in a single directory. I'd like to log the number of disk block reads and writes over a period (not just file access) to test whether ...
7
votes
2answers
2k views

How to stop xargs from badly merging output from multiple processes?

I'm using xargs with the option --max-args=0 (alternatively -P 0). However, the output of the processes is merged into the stdout stream without regard for proper line separation. So I'll often end ...
7
votes
3answers
310 views

Harddrive I/O per Pid

is there any way to see the disk I/O per pid/process ? Monitorix shows me the following and correct me if I´m wrong, but it looks like that the harddrive is slowing down the whole system: UPDATE #1 ...
6
votes
1answer
1k views

How do I view the IO priority of a process?

How can I view the IO priority of a process? like to see for example if something has been ionice-ed.
6
votes
2answers
127 views

Reduce bad block retry / wait times in Ubuntu

How can I reduce the IO wait time and retry times so that the OS doesn't continually try to write to a failing drive? I have an system that I use to make copies of demo content that gets loaned out ...
6
votes
3answers
264 views

Why is I/O uninterruptible?

What is the rationale behind making I/O uninterruptible? What would be the negative consequences if a process handling I/O were allowed to handle a signal?
6
votes
2answers
378 views

Mysterious write IO

I am just reviewing the disk IO of a XEN VM server. I stumbled across a DomU that has constant write traffic. To narrow that traffic further down I logged in to the DomU (CentOS 5, 64 Bit) and ...
6
votes
2answers
634 views

Piping output from a segfaulting program

I have a script that calls a program (specifically, ttf2afm, part of tetex 3.0) that sometimes segfaults and sometimes doesn't. The information I need is always printed out before it segfaults, but ...
6
votes
3answers
759 views

Deleting lots of files

I accidentally created 8 million files and every time I'm trying to delete them the server almost dies because of the rm process eating all disk IO (the server is remote without console). Should ...
6
votes
5answers
533 views

copy consumed output to stdout when stdout is redirected to a file

Given a command the output of which is being processed by, for example, grep I also want the actual output to be included in the log file for debugging purposes. For example if I have the following ...
5
votes
4answers
655 views

Can I just disable updatedb?

Is updatedb necessary at all? I never use locate and my servers tend to have dozens of millions of files which usually makes updatedb to run for a long time and consume I/O needed by MySQL and/or ...
5
votes
3answers
414 views

File size of 60.0 PB is wrong. Can deleting it cause data loss?

While backing up some data (a 200 GB home directory) with rsync, I got an I/0 error for a particular file, after which rsync continued on "normally" with its backup. The problem source file showed as ...
5
votes
2answers
4k views

root user denied access to .gvfs in rsnapshot?

I was running rsnapshot as root and I got the following error. Why would this happen? what is .gvfs? rsnapshot weekly slave-iv ...
5
votes
1answer
300 views

`unbuffer` or `stdbuf` for removing stdout buffering?

Is there a difference between unbuffer(1) and stdbuf(1)? From what I gather, unbuffer makes more than the "best effort" of calling the libc function set(X)buf at the beginning, and then letting things ...
5
votes
2answers
49 views

How can I see i/o stats for a briefly running process?

For long running processes like init, I can do things like $ cat /proc/[pid]/io What can I do if I want to see stats for a briefly running process such as a command line utility like ls? I don't ...
5
votes
2answers
346 views

How to improve IO when there is a lot of random read and write?

My scripts uses tons of read and writes. Most of them are small and random. I want Linux to wait 10 seconds before writing any data to the disk. How would this be achieved? I think Linux can already ...
5
votes
1answer
836 views

SysBench highlighting abysmal disk write performance on white box vs Tier1 server

Have been testing white box system with SuperMicro X8DTL motherboard with various SATA hard disks such as 7,200RPM Seagate Constellation ES and 10,000 RPM Western Digital VelociRaptor. Have tested ...
4
votes
2answers
2k views

Hight iowait while the wMB/s is low

I was indexing a MySQL table. It makes a high load on that computer. It seems due to a high iowait. But it also shows that the wMB/s is only 2.87. Isn't even a common SATA HDD able to handle more ...
4
votes
5answers
412 views

performance monitoring

Is there some performace monitoring tool which would run in background gathering info about all system activity? Somethimes my system (Arch linux, 32 bit) slows down terribly and the top utility ...
4
votes
1answer
540 views

Does IO prioritise by the very nature of renicing a task?

The CFQ IO scheduler supports priorities though I am not sure that Deadline does (I believe not). The premise is that when I renice a task it gets a larger share of CPU under the Completely Fair ...
4
votes
2answers
155 views

Determine which process is taking most of disk bandwidth?

When the HDD indicator is blinking (for a long period), how could I know which process is taking most disk bandwidth?
4
votes
3answers
1k views

Intercept input and output from specific executable

I have a certain Linux executable that is called from a bigger application to perform some calculations. I want to modify the way the calculations are performed, for that I first need to understand ...
4
votes
1answer
1k views

Why do backgrounded processes sometimes stop spontaneously?

Sometimes, some time after I've backgrounded a process with bg in bash, when I press Enter in the same shell to redisplay the prompt (just to check that I'm still in bash when some output from the ...
4
votes
1answer
169 views

Execution time and resources after processes that has been running for so long

This might be a basic question, but I am getting hard time finding an answer to it. I am using RHEL6. After running any process that takes significant cpu time, I get a line that tells (I guess) the ...
4
votes
4answers
922 views

Special File that causes I/O error

I want to automatically test if a piece of software reacts as expected if an essential SQLite DB file fails to be read (causing an I/O error). Exactly that happened some days ago at a client. We ...
4
votes
3answers
272 views

Poor IO due to LUKS/Software RAID/LVM ordering?

I'm trying to determine if I should re-setup my RAID array due to poor IO performance. First off, the system: i7 920 4 4TB WD 5400 Green drives CentOS 6.3 host Secondly, the disk setup: ...