I/O is an acronym for Input/Output, which refers to the mechanism or process for the transfer of information between one system or component and another.
24
votes
12answers
2k views
How can I tweak my Linux desktop to be more responsive?
I have the experience that Linux works fine until the physical memory is exhausted. As soon as swap space is used the performance is severely degraded and the GUI becomes unresponsive.
This problem ...
23
votes
4answers
2k views
Redirecting stdout to a file you don't have write permission on
When you attempt to modify a file without having write permissions on it, you get an error:
> touch /tmp/foo && sudo chown root /tmp/foo
> echo test > /tmp/foo
zsh: permission ...
18
votes
3answers
524 views
How can I limit the output speed of stdout?
I'm running CentOS 5.7 and I have a backup utility that has the option of dumping its backup file to stdout. The backup file is rather large (multiple gigabytes). The target is an SSHFS filesystem. To ...
13
votes
2answers
3k views
Why does high disk I/O reduce system responsiveness/performance?
I never quite understood why high disk I/O slowed the system so much. It's strange to me because I would expect the slow-down to affect only those processes dependent on the hard/optical drive data, ...
13
votes
2answers
2k views
When is dd suitable for copying data? (or, when are read() and write() partial)
Short version: In what circumstances is dd safe to use for copying data, safe meaning that there is no risk of corruption due to a partial read or write?
Long version — preamble: dd is often used to ...
11
votes
2answers
1k views
Measure pipe throughput in the shell
There is a shell command that allows to measure how fast the data goes through it, so you can measure the speed of output of commands in a pipe. So instead of:
$ somecommand | anothercommand
you ...
11
votes
2answers
511 views
Portability of “> /dev/stdout”
Occasionally I need to specify a "path-equivalent" of one of the standard IO streams (stdin, stdout, stderr). Since 99% of the time I work with Linux, I just prepend /dev/ to get /dev/stdin, etc., ...
11
votes
2answers
721 views
Weight-based block IO control for VMs
I use KVM to manage my virtual machines. I am trying to limit the resources provided for VMs. I am able to limit the CPU and memory allocated for VMs using libvirt/cgroups. Now I want to control the ...
10
votes
1answer
477 views
How do I go about tracking down IO usage in Linux and other *nix OS'?
Occasionally, something on my system starts hammering the disk. I would like to log data that would let me know about I/O usage per process, then I could go back to it later and see what keeps doing ...
10
votes
1answer
692 views
Meaning of colours in htop
Given the option:
[ ] Detailed CPU time (System/IO-Wait/Hard-IRQ/Soft-IRQ/Steal/Guest)
In htop (Version 1.0.1) what are the meanings of the colours?
In the following image, I'm seeing rather a lot ...
9
votes
6answers
14k views
How can I read line by line from a variable in bash?
I have a variable which contains multiline output of a command. What's the most effecient way to read the output line by line from the variable?
For example:
jobs="$(jobs)"
if [ "$jobs" ]; then
...
8
votes
2answers
1k views
Heavy write activity on SSD nukes system performance
I've noticed that when I do heavy write applications, the whole system slows down. To test this further I ran this to do a (relatively) low-CPU, high disk activity:
john -incremental > file_on_SSD
...
8
votes
1answer
845 views
What's the progress regarding improving system performance/responsiveness during high disk I/O?
Whenever there is high disk I/O, the system tends to be much slower and less responsive than usual. What's the progress on Linux kernel regarding this? Is this problem actively being worked on?
8
votes
1answer
537 views
Measuring disk I/O usage of a program
time is a brilliant command if you want to figure out how much CPU time a given command takes.
I am looking for something similar that can measure the disk I/O of the program and any children. ...
6
votes
2answers
661 views
Can I watch the progress of a `sync` operation?
I've copied a large file to a USB disk mounted on a Linux system with async. This returns to a command prompt relatively quickly, but when I type sync, of course, it all has to go to disk, and that ...
6
votes
3answers
1k views
How can I monitor disk I/O in a particular directory?
I've got a few processes with a known name that all write to files in a single directory. I'd like to log the number of disk block reads and writes over a period (not just file access) to test whether ...
6
votes
2answers
263 views
Mysterious write IO
I am just reviewing the disk IO of a XEN VM server. I stumbled across a DomU that has constant write traffic.
To narrow that traffic further down I logged in to the DomU (CentOS 5, 64 Bit) and ...
6
votes
2answers
404 views
Piping output from a segfaulting program
I have a script that calls a program (specifically, ttf2afm, part of tetex 3.0) that sometimes segfaults and sometimes doesn't. The information I need is always printed out before it segfaults, but ...
6
votes
3answers
519 views
Deleting lots of files
I accidentally created 8 million files and every time I'm trying to delete them the server almost dies because of the rm process eating all disk IO (the server is remote without console).
Should ...
6
votes
4answers
348 views
copy consumed output to stdout when stdout is redirected to a file
Given a command the output of which is being processed by, for example, grep I also want the actual output to be included in the log file for debugging purposes.
For example if I have the following
...
5
votes
2answers
3k views
Run process with higher priority
I want to run mplayer with higher priority than any other processes, including the IO-processes. How can I do that?
5
votes
3answers
217 views
Utility to buffer an unbounded amount of data in a pipeline?
Is there a utility that I can stick in a pipeline to decouple read and write speeds?
$ producer | buf | consumer
Basically, I want a utility buf that reads its input as fast as possible, storing it ...
5
votes
2answers
796 views
How to stop xargs from badly merging output from multiple processes?
I'm using xargs with the option --max-args=0 (alternatively -P 0).
However, the output of the processes is merged into the stdout stream without regard for proper line separation. So I'll often end ...
5
votes
3answers
4k views
Find out what processes are writing to hard drive
On my Lenovo T400 and Ubuntu, the light for hard drive writing keeps flashing. I was wondering if in Linux it is possible to find out what processes are doing I/O to the hard drive? Just like by top, ...
5
votes
3answers
256 views
File size of 60.0 PB is wrong. Can deleting it cause data loss?
While backing up some data (a 200 GB home directory) with rsync, I got an I/0 error for a particular file, after which rsync continued on "normally" with its backup. The problem source file showed as ...
5
votes
2answers
3k views
root user denied access to .gvfs in rsnapshot?
I was running rsnapshot as root and I got the following error. Why would this happen? what is .gvfs?
rsnapshot weekly slave-iv
...
5
votes
1answer
418 views
What keeps one side of an rsync so busy?
I've got a debian machine in my lan serving as a backup server for the others. It has four HDDs combined into a software RAID 5 md device, on that an lvm, and on that a btrfs. Backups are made using ...
5
votes
2answers
128 views
How to improve IO when there is a lot of random read and write?
My scripts uses tons of read and writes. Most of them are small and random. I want Linux to wait 10 seconds before writing any data to the disk.
How would this be achieved?
I think Linux can already ...
5
votes
1answer
575 views
SysBench highlighting abysmal disk write performance on white box vs Tier1 server
Have been testing white box system with SuperMicro X8DTL motherboard with various SATA hard disks such as 7,200RPM Seagate Constellation ES and 10,000 RPM Western Digital VelociRaptor.
Have tested ...
4
votes
2answers
1k views
Hight iowait while the wMB/s is low
I was indexing a MySQL table. It makes a high load on that computer.
It seems due to a high iowait. But it also shows that the wMB/s is only 2.87.
Isn't even a common SATA HDD able to handle more ...
4
votes
4answers
293 views
performance monitoring
Is there some performace monitoring tool which would run in background gathering info about all system activity? Somethimes my system (Arch linux, 32 bit) slows down terribly and the top utility ...
4
votes
1answer
414 views
Does IO prioritise by the very nature of renicing a task?
The CFQ IO scheduler supports priorities though I am not sure that Deadline does (I believe not). The premise is that when I renice a task it gets a larger share of CPU under the Completely Fair ...
4
votes
2answers
95 views
Determine which process is taking most of disk bandwidth?
When the HDD indicator is blinking (for a long period), how could I know which process is taking most disk bandwidth?
4
votes
3answers
758 views
Intercept input and output from specific executable
I have a certain Linux executable that is called from a bigger application to perform some calculations. I want to modify the way the calculations are performed, for that I first need to understand ...
4
votes
2answers
1k views
How to Throttle per process I/O to a max limit?
I'm looking for a way to limit a processes disk io to a set speed limit. Ideally the program would work similar to this:
$ limitio --pid 32423 --write-limit 1M
Limiting process 32423 to 1 megabyte ...
4
votes
1answer
762 views
How do I view the IO priority of a process?
How can I view the IO priority of a process? like to see for example if something has been ionice-ed.
4
votes
2answers
116 views
Why is I/O uninterruptible?
What is the rationale behind making I/O uninterruptible? What would be the negative consequences if a process handling I/O were allowed to handle a signal?
4
votes
4answers
422 views
script for logging all the stats for a particular process
I want to monitor CPU usage, disk read/write usage for a particular process, say ./myprocess.
To monitor CPU top command seems to be a nice option and for read and write iotop seems to be a handy ...
4
votes
1answer
443 views
“Split I/O”s on a Linux VM, disk alignment
I'm trying to figure out if there's some way I can tell if a system is being adversely affected by disk misalignment -- i.e. when I do a disk I/O operation, it actually needs to request extra blocks ...
4
votes
3answers
703 views
How to get output from a remote shell
I have an application running on a server, started from the command line. From time to time, I need to connect to the server via SSH and get the output messages written to stdout from the application. ...
4
votes
1answer
154 views
Repeatably generate a read error for testing?
I'm trying to test the error handling in some software, in particular what happens when an error occurs reading from a file or pipe. Is there a simple way to send a certain amount of data to stdout ...
4
votes
2answers
2k views
Write inside a socket open by another process in Linux
Is it possible on Linux for a process to write inside a socket open by another one?
Let's say I open a connection to google.com using netcat:
myuser@linux:~$ nc google.com 80
Now I can lookup for ...
3
votes
1answer
247 views
Removing redirection operator does not change output. Why?
I saw this usage of redirection somewhere, and thought it was a typo:
grep root < /etc/passwd
But after I run it, I saw that it gives the same output with
grep root /etc/passwd:
$ grep root ...
3
votes
3answers
154 views
Changing the disk scheduler on the fly
Can the Linux disk scheduler be changed on the fly by writing to /sys/block/sda/queue/scheduler? Should applications (e.g. Mysql) be stopped / started when changing?
3
votes
2answers
157 views
Storing thousands of files in one directory
I have a web-site that I'm checking for performance issues and bugs, and i came across a caching code that caches thousands of files into one single directory.
I understand that this is not good and ...
3
votes
2answers
992 views
100.0%sy with poor disk performance
I have RHEL 5.4 kernel 2.6.18-164.el5 and I am experiencing very bad disk performance that happens randomly during working with Oracle.
Nothing appears in the logs.
When it happens, I see one CPU ...
3
votes
2answers
81 views
dealing with un-executable commands inside a shell-script
I am given an assignment like this: Write a Bash script that takes a *.tex file as command-line argument and checks if the file executes correctly without any errors. If it executes without any error ...
3
votes
2answers
452 views
I/O statistics per user
I have access to an academic supercomputer where there is an NFS filesystem where our lab keeps our data.
When I run iostats -nh I believe I am getting statistics for all users accessing all storage ...
3
votes
1answer
98 views
Copying a file that is written at the same time
This related question can't solve my exact problem, so I'm writing a similar question.
My scenario is the following: a program (run by wine) writes to a file sized about 10-500 KB. About 1 KB ...
3
votes
2answers
200 views
Running jobs in parallel on Ubuntu - I/O contention differences between Perl and Java
Apologies if this is off topic - it concerns the relative efficiencies of running I/O-heavy Perl/Java scripts in parallel on a Ubuntu system.
I have written two simple versions of a file copy script ...
