I/O is an acronym for Input/Output, which refers to the mechanism or process for the transfer of information between one system or component and another.

learn more… | top users | synonyms

35
votes
5answers
3k views

Redirecting stdout to a file you don't have write permission on

When you attempt to modify a file without having write permissions on it, you get an error: > touch /tmp/foo && sudo chown root /tmp/foo > echo test > /tmp/foo zsh: permission ...
25
votes
12answers
2k views

How can I tweak my Linux desktop to be more responsive?

I have the experience that Linux works fine until the physical memory is exhausted. As soon as swap space is used the performance is severely degraded and the GUI becomes unresponsive. This problem ...
22
votes
5answers
2k views

Why is iterating over a file twice faster than reading it into memory and computing twice?

I'm comparing the following tail -n 1000000 stdout.log | grep '"success": true' | wc -l tail -n 1000000 stdout.log | grep '"success": false' | wc -l with the following log=$(tail -n 1000000 ...
21
votes
3answers
684 views

How can I limit the output speed of stdout?

I'm running CentOS 5.7 and I have a backup utility that has the option of dumping its backup file to stdout. The backup file is rather large (multiple gigabytes). The target is an SSHFS filesystem. To ...
20
votes
3answers
1k views

Can the “du” program be made less aggressive?

We have a regular job that does du summaries of a number of subdirectories, picking out worst offenders, and use the output to find if there are things that are rapidly rising to spot potential ...
18
votes
2answers
3k views

When is dd suitable for copying data? (or, when are read() and write() partial)

Short version: In what circumstances is dd safe to use for copying data, safe meaning that there is no risk of corruption due to a partial read or write? Long version — preamble: dd is often used to ...
14
votes
2answers
2k views

Measure pipe throughput in the shell

There is a shell command that allows to measure how fast the data goes through it, so you can measure the speed of output of commands in a pipe. So instead of: $ somecommand | anothercommand you ...
14
votes
2answers
943 views

Portability of “> /dev/stdout”

Occasionally I need to specify a "path-equivalent" of one of the standard IO streams (stdin, stdout, stderr). Since 99% of the time I work with Linux, I just prepend /dev/ to get /dev/stdin, etc., ...
13
votes
3answers
935 views

Determining Specific File Responsible for High I/O

This is a simple problem but the first time I've ever had to actually fix it: finding which specific files/inodes are the targets of the most I/O. I'd like to be able to get a general system overview, ...
13
votes
2answers
4k views

Why does high disk I/O reduce system responsiveness/performance?

I never quite understood why high disk I/O slowed the system so much. It's strange to me because I would expect the slow-down to affect only those processes dependent on the hard/optical drive data, ...
13
votes
1answer
2k views

Meaning of colours in htop

Given the option: [ ] Detailed CPU time (System/IO-Wait/Hard-IRQ/Soft-IRQ/Steal/Guest) In htop (Version 1.0.1) what are the meanings of the colours? In the following image, I'm seeing rather a lot ...
12
votes
5answers
28k views

How can I read line by line from a variable in bash?

I have a variable which contains multiline output of a command. What's the most effecient way to read the output line by line from the variable? For example: jobs="$(jobs)" if [ "$jobs" ]; then ...
11
votes
2answers
2k views

Can I watch the progress of a `sync` operation?

I've copied a large file to a USB disk mounted on a Linux system with async. This returns to a command prompt relatively quickly, but when I type sync, of course, it all has to go to disk, and that ...
11
votes
1answer
521 views

How do I go about tracking down IO usage in Linux and other *nix OS'?

Occasionally, something on my system starts hammering the disk. I would like to log data that would let me know about I/O usage per process, then I could go back to it later and see what keeps doing ...
11
votes
2answers
1k views

Weight-based block IO control for VMs

I use KVM to manage my virtual machines. I am trying to limit the resources provided for VMs. I am able to limit the CPU and memory allocated for VMs using libvirt/cgroups. Now I want to control the ...
10
votes
1answer
152 views

Moving a file to a file that is being read

When a file is moved and the destination file already exists and is currently being read, will the operation succeed? Will it depend on how long the reading operation on the destination file lasts?
9
votes
2answers
4k views

How to Throttle per process I/O to a max limit?

I'm looking for a way to limit a processes disk io to a set speed limit. Ideally the program would work similar to this: $ limitio --pid 32423 --write-limit 1M Limiting process 32423 to 1 megabyte ...
9
votes
1answer
1k views

What's the progress regarding improving system performance/responsiveness during high disk I/O?

Whenever there is high disk I/O, the system tends to be much slower and less responsive than usual. What's the progress on Linux kernel regarding this? Is this problem actively being worked on?
8
votes
3answers
434 views

Utility to buffer an unbounded amount of data in a pipeline?

Is there a utility that I can stick in a pipeline to decouple read and write speeds? $ producer | buf | consumer Basically, I want a utility buf that reads its input as fast as possible, storing it ...
8
votes
2answers
1k views

Heavy write activity on SSD nukes system performance

I've noticed that when I do heavy write applications, the whole system slows down. To test this further I ran this to do a (relatively) low-CPU, high disk activity: john -incremental > file_on_SSD ...
8
votes
1answer
866 views

Measuring disk I/O usage of a program

time is a brilliant command if you want to figure out how much CPU time a given command takes. I am looking for something similar that can measure the disk I/O of the program and any children. ...
7
votes
3answers
8k views

Find out what processes are writing to hard drive

On my Lenovo T400 and Ubuntu, the light for hard drive writing keeps flashing. I was wondering if in Linux it is possible to find out what processes are doing I/O to the hard drive? Just like by top, ...
7
votes
3answers
2k views

How can I monitor disk I/O in a particular directory?

I've got a few processes with a known name that all write to files in a single directory. I'd like to log the number of disk block reads and writes over a period (not just file access) to test whether ...
7
votes
2answers
1k views

How to stop xargs from badly merging output from multiple processes?

I'm using xargs with the option --max-args=0 (alternatively -P 0). However, the output of the processes is merged into the stdout stream without regard for proper line separation. So I'll often end ...
7
votes
3answers
160 views

Harddrive I/O per Pid

is there any way to see the disk I/O per pid/process ? Monitorix shows me the following and correct me if I´m wrong, but it looks like that the harddrive is slowing down the whole system: UPDATE #1 ...
6
votes
2answers
6k views

Run process with higher priority

I want to run mplayer with higher priority than any other processes, including the IO-processes. How can I do that?
6
votes
3answers
205 views

Why is I/O uninterruptible?

What is the rationale behind making I/O uninterruptible? What would be the negative consequences if a process handling I/O were allowed to handle a signal?
6
votes
2answers
346 views

Mysterious write IO

I am just reviewing the disk IO of a XEN VM server. I stumbled across a DomU that has constant write traffic. To narrow that traffic further down I logged in to the DomU (CentOS 5, 64 Bit) and ...
6
votes
2answers
530 views

Piping output from a segfaulting program

I have a script that calls a program (specifically, ttf2afm, part of tetex 3.0) that sometimes segfaults and sometimes doesn't. The information I need is always printed out before it segfaults, but ...
6
votes
3answers
689 views

Deleting lots of files

I accidentally created 8 million files and every time I'm trying to delete them the server almost dies because of the rm process eating all disk IO (the server is remote without console). Should ...
6
votes
5answers
453 views

copy consumed output to stdout when stdout is redirected to a file

Given a command the output of which is being processed by, for example, grep I also want the actual output to be included in the log file for debugging purposes. For example if I have the following ...
5
votes
3answers
382 views

File size of 60.0 PB is wrong. Can deleting it cause data loss?

While backing up some data (a 200 GB home directory) with rsync, I got an I/0 error for a particular file, after which rsync continued on "normally" with its backup. The problem source file showed as ...
5
votes
1answer
1k views

How do I view the IO priority of a process?

How can I view the IO priority of a process? like to see for example if something has been ionice-ed.
5
votes
2answers
4k views

root user denied access to .gvfs in rsnapshot?

I was running rsnapshot as root and I got the following error. Why would this happen? what is .gvfs? rsnapshot weekly slave-iv ...
5
votes
1answer
185 views

`unbuffer` or `stdbuf` for removing stdout buffering?

Is there a difference between unbuffer(1) and stdbuf(1)? From what I gather, unbuffer makes more than the "best effort" of calling the libc function set(X)buf at the beginning, and then letting things ...
5
votes
1answer
752 views

What keeps one side of an rsync so busy?

I've got a debian machine in my lan serving as a backup server for the others. It has four HDDs combined into a software RAID 5 md device, on that an lvm, and on that a btrfs. Backups are made using ...
5
votes
2answers
276 views

How to improve IO when there is a lot of random read and write?

My scripts uses tons of read and writes. Most of them are small and random. I want Linux to wait 10 seconds before writing any data to the disk. How would this be achieved? I think Linux can already ...
5
votes
1answer
771 views

SysBench highlighting abysmal disk write performance on white box vs Tier1 server

Have been testing white box system with SuperMicro X8DTL motherboard with various SATA hard disks such as 7,200RPM Seagate Constellation ES and 10,000 RPM Western Digital VelociRaptor. Have tested ...
4
votes
2answers
2k views

Hight iowait while the wMB/s is low

I was indexing a MySQL table. It makes a high load on that computer. It seems due to a high iowait. But it also shows that the wMB/s is only 2.87. Isn't even a common SATA HDD able to handle more ...
4
votes
5answers
375 views

performance monitoring

Is there some performace monitoring tool which would run in background gathering info about all system activity? Somethimes my system (Arch linux, 32 bit) slows down terribly and the top utility ...
4
votes
1answer
522 views

Does IO prioritise by the very nature of renicing a task?

The CFQ IO scheduler supports priorities though I am not sure that Deadline does (I believe not). The premise is that when I renice a task it gets a larger share of CPU under the Completely Fair ...
4
votes
2answers
150 views

Determine which process is taking most of disk bandwidth?

When the HDD indicator is blinking (for a long period), how could I know which process is taking most disk bandwidth?
4
votes
3answers
1k views

Intercept input and output from specific executable

I have a certain Linux executable that is called from a bigger application to perform some calculations. I want to modify the way the calculations are performed, for that I first need to understand ...
4
votes
1answer
830 views

Why do backgrounded processes sometimes stop spontaneously?

Sometimes, some time after I've backgrounded a process with bg in bash, when I press Enter in the same shell to redisplay the prompt (just to check that I'm still in bash when some output from the ...
4
votes
1answer
125 views

Execution time and resources after processes that has been running for so long

This might be a basic question, but I am getting hard time finding an answer to it. I am using RHEL6. After running any process that takes significant cpu time, I get a line that tells (I guess) the ...
4
votes
3answers
74 views

Poor IO due to LUKS/Software RAID/LVM ordering?

I'm trying to determine if I should re-setup my RAID array due to poor IO performance. First off, the system: i7 920 4 4TB WD 5400 Green drives CentOS 6.3 host Secondly, the disk setup: ...
4
votes
1answer
249 views

how can I retrieve disk IO utilization in percents on HP-UX

I need to collect data about disk utilization for selected disks. I can use glance-plus monitoring tool to display the current data in percents (it looks similar to top), but I need to collect these ...
4
votes
4answers
707 views

script for logging all the stats for a particular process

I want to monitor CPU usage, disk read/write usage for a particular process, say ./myprocess. To monitor CPU top command seems to be a nice option and for read and write iotop seems to be a handy ...
4
votes
1answer
521 views

“Split I/O”s on a Linux VM, disk alignment

I'm trying to figure out if there's some way I can tell if a system is being adversely affected by disk misalignment -- i.e. when I do a disk I/O operation, it actually needs to request extra blocks ...
4
votes
3answers
785 views

How to get output from a remote shell

I have an application running on a server, started from the command line. From time to time, I need to connect to the server via SSH and get the output messages written to stdout from the application. ...