Hot answers tagged

162

Looking at /proc/meminfo will show the Dirty number shrinking over time as all the data spools out; some of it may spill into Writeback as well. That will be a summary against all devices, but in the cases where one device on the system is much slower than the rest you'll usually end up where everything in that queue is related to it. You'll probably find ...


59

It simply gives an illusion of speed to programs that don't actually have to wait until a write is complete. Mount your filesystems in sync mode (which gives you your instant writes) and see how slow everything is. Sometimes files exist only temporarily... a program does some bit of work and deletes the file right after the work is done. If you delayed ...


57

There are several aspects to this question which have been addressed partially through other tools, but there doesn't appear to be a single tool that provides all the features you're looking for. iotop This tools shows which processes are consuming the most I/O. But it lacks options to show specific file names. $ sudo iotop Total DISK READ: 0.00 B/s ...


55

They do count as I/O, but not of the type measured by the fields you’re looking at. In htop, IO_RBYTES and IO_WBYTES show the read_bytes and write_bytes fields from /proc/<pid>/io, and those fields measure bytes which go through the block layer. /dev/zero doesn’t involve the block layer, so reads from it don’t show up there. To see I/O from /dev/zero,...


47

What's the philosophy behind such an approach? Efficiency (better usage of the disk characteristics) and performance (allows the application to continue immediately after a write). Why isn't the data written at once? The main advantage is the OS is free to reorder and merge contiguous write operations to improve their bandwidth usage (less operations ...


45

iostat is part of the sysstat package, which is able to show overall iops if desired, or show them separated by reads/writes. Run iostat with the -d flag to only show the device information page, and -x for detailed information (separate read/write stats). You can specify the device you want information for by simply adding it afterwards on the command ...


41

Can a file be saved not sequentially on disk? I mean, part of the file is located under physical address X and the other part under physical address Y which isn't close to X + offset). Yes; this is known as file fragmentation and is not uncommon, especially with larger files. Most file systems allocate space as it's needed, more or less sequentially, but ...


41

Problem is that when you redirect your output, it's not available anymore for the next redirect. You can pipe to tee in a subshell to keep the output for the second redirection: ( cmd | tee -a file2 ) >> file1 2>&1 or if you like to see the output in terminal: ( cmd | tee -a file2 ) 2>&1 | tee -a file1 To avoid adding the stderr of ...


35

It's been available on Linux back into its prehistory. It is not POSIX, although many actual shells (including AT&T ksh and bash) will simulate it if it's not present in the OS; note that this simulation only works at the shell level (i.e. redirection or command line parameter, not as explicit argument to e.g. open()). That said, it should be available ...


31

That is certainly not trivial task that can't be done in userspace. Fortunately, it is possible to do on Linux, using cgroup mechanizm and its blkio controller. Setting up cgroup is somehow distribution specific as it may already be mounted or even used somewhere. Here's general idea, however (assuming you have proper kernel configuration): mount -t tmpfs ...


28

You can add a rate limiting tool to your pipeline. For example there is pv which has a rate-limiting option: -L RATE, --rate-limit RATE Limit the transfer to a maximum of RATE bytes per second. A suffix of "k", "m", "g", or "t" can be added to denote kilobytes (*1024), megabytes, and so on An alternative is the tool buffer which has: -u ...


28

Take a look at ionice. From man ionice: This program sets or gets the io scheduling class and priority for a program. If no arguments or just -p is given, ionice will query the current io scheduling class and priority for that process. To run du with the "idle" I/O class, which is the lowest priority available, you can do something like this: ionice -c ...


27

After some more investigation, it looks like this issue is less kernel related and more about how rsync and CIFS interact. As far as I can make out, what is happening is that when rsync closes the destination file, CIFS (and probably any network filesystem) ensures the file is completely flushed and written to the remote disk before the close syscall ...


26

F1 or h will show you the legend. It looks like in this color scheme: CPU: blue is for low priority threads green is normal priority threads black is for io-wait see below for more. Memory: green is memory in use blue is buffer orange is cache


26

I've done the following test and on my system the resulting difference is about 100 times longer for the second script. My file is a strace output called bigfile $ wc -l bigfile.log 1617000 bigfile.log Scripts xtian@clafujiu:~/tmp$ cat p1.sh tail -n 1000000 bigfile.log | grep '"success": true' | wc -l tail -n 1000000 bigfile.log | grep '"success": false'...


26

512 byte is not really the default sector size. It depends on your hardware. You can display what physical/logical sector sizes your disk reports via the /sys pseudo filesystem, for instance: # cat /sys/block/sda/queue/physical_block_size 4096 # cat /sys/block/sda/queue/logical_block_size 512 What is the difference between those two values? The ...


26

Asynchronous, buffered I/O was in use before Linux and even before Unix. Unix had it, and so have all its offshoots. Here is what Ritchie and Thompson wrote in their CACM paper The UNIX Time-Sharing System: To the user, both reading and writing of files appear to be synchronous and unbuffered. That is immediately after return from a read call the ...


25

Yes you can disable it in the crons or remove the package that provides updatedb. On a Red Hat system you'd go about the steps in determining if anything requires it prior to removal. First find out where the program is located on disk. $ type updatedb updatedb is /usr/bin/updatedb Next find out what package provides updatedb. $ rpm -qf /usr/bin/updatedb ...


25

Obviously closing stdout does not fail, on the contrary, it succeeds because writing on it fails, as can be seen from the error message. Edit: to clarify my answer, what happens is that you first tell the shell to close the file descriptor, then the ls program tries to write to it. This is where the error message comes from.


24

You can look at the /sys/block/<device>/stat file for the appropriate device while you're syncing. The 9th column will indicate the number of in-flight requests on the device, which should go down to zero when the sync is done. Don't know of a way to translate that to a number of bytes, but it should give you a rough idea of how much "stuff" is still ...


24

Using nvme-cli, I can get temperature from a Samsung 950 Pro with this command: nvme smart-log /dev/nvme0 | grep "^temperature" You can get other informations too: nvme smart-log /dev/nvme0 Smart Log for NVME device:nvme0 namespace-id:ffffffff critical_warning : 0 temperature : 45 C available_spare ...


22

ionice from the util-linux does something similar to what you want. It doesn't set absolute IO limits, it sets IO priority and 'niceness' - similar to what nice does for a process' CPU priority. From the man page: ionice - set or get process I/O scheduling class and priority DESCRIPTION This program sets or gets the I/O scheduling class and priority ...


21

the /dev/std{in,out,err} files are normally just symlinks to /proc/self/fd/{0,1,2} (respectively). As such theres nothing gained over using methods that are POSIX defined. If you want to be POSIX compliant, the best way to do this is to use output redirection. Shell output redirection is defined in the POSIX standard. Additionally the STDIN, STDOUT, STDERR ...


20

Use: curl http://mysite.com/myfile.jpg > myfile.jpg


19

The information shown by iotop isn't gathered in the same way for individual processes and for the system as a whole. The “actual” global figures are not the sum of the per-process figures (that's what “total” is). All information is gathered from the proc filesystem. For each process, iotop reads data from /proc/PID/io, specifically the rchar and wchar ...


18

To reference a few more tools. htop http://hisham.hm/htop/ https://github.com/hishamhm/htop Command line tool, packaged in most distributions, is able to show the I/O without root privileges but only for your processes. run htop(1), you'll find an interface similar to top(1) hit F2 to enter the configuration use ↓ to select "Columns" use → to select "...


17

The command filefrag will tell you how your file is physically stored on your device: # filefrag -v /var/log/messages.1 Filesystem type is: ef53 File size of /var/log/messages.1 is 41733 (11 blocks, blocksize 4096) ext logical physical expected length flags 0 0 2130567 1 1 1 15907576 2130568 1 2 2 15910400 ...


16

The file in /proc/<pid>/io represent what you need. It's a bit scripting work to get an output similar to iotop. See the linux kernel documentation to the /proc/<pid>/io file values: rchar I/O counter: chars read The number of bytes which this task has caused to be read from storage. This is simply the sum of bytes which this process ...


16

With zsh: cmd >& out+err.log > out.log In append mode: cmd >>& out+err.log >> out.log In zsh, and provided the mult_ios option has not been disabled, when a file descriptor (here 1) is redirected several times for writing, then the shell implements a built-in tee to duplicate the output to all targets.


15

Many good answers, but let me add one other thing... Remember that Unix is a multi-process and multi-users system, so potentially many users would be trying to do file-operations (esp. writes) at (almost) the same time. With old slow hard-disks - perhaps mounted over the network - this would not only take time (for which the programs would basically lock-...


Only top voted, non community-wiki answers of a minimum length are eligible