Tag Info

Hot answers tagged

57

Yes, using tee. So echo test > /tmp/foo becomes echo test | sudo tee /tmp/foo You can also append (>>) echo test | sudo tee -a /tmp/foo


53

It simply gives an illusion of speed to programs that don't actually have to wait until a write is complete. Mount your filesystems in sync mode (which gives you your instant writes) and see how slow everything is. Sometimes files exist only temporarily... a program does some bit of work and deletes the file right after the work is done. If you delayed ...


37

What's the philosophy behind such an approach? Efficiency (better usage of the disk characteristics) and performance (allows the application to continue immediately after a write). Why isn't the data written at once? The main advantage is the OS is free to reorder and merge contiguous write operations to improve their bandwidth usage (less ...


34

cpipe is probably better for these purposes, but another related program is pv (Pipe Viewer): If you give it the --rate flag it will show the transfer rate


29

Looking at /proc/meminfo will show the "Dirty" number shrinking over time as all the data spools out; some of it may spill into "Writeback" as well. That will be a summary against all devices, but in the cases where one device on the system is much slower than the rest you'll usually end up where everything in that queue is related to it. You'll probably ...


28

Take a look at ionice. From man ionice: This program sets or gets the io scheduling class and priority for a program. If no arguments or just -p is given, ionice will query the current io scheduling class and priority for that process. To run du with the "idle" I/O class, which is the lowest priority available, you can do something like this: ionice ...


26

I've done the following test and on my system the resulting difference is about 100 times longer for the second script. My file is a strace output called bigfile $ wc -l bigfile.log 1617000 bigfile.log Scripts xtian@clafujiu:~/tmp$ cat p1.sh tail -n 1000000 bigfile.log | grep '"success": true' | wc -l tail -n 1000000 bigfile.log | grep '"success": ...


25

You can use a while loop with process substitution: while read -r line do echo "$line" done < <(jobs) To read a multiline variable, a simple way is: # You need printf '%s\n' "$var" here because if you use printf '%s' "$var" # on a variable that doesn't end with a newline then the while loop will # completely miss the last line of the variable. ...


23

You can add a rate limiting tool to your pipeline. For example there is pv which has a rate-limiting option: -L RATE, --rate-limit RATE Limit the transfer to a maximum of RATE bytes per second. A suffix of "k", "m", "g", or "t" can be added to denote kilobytes (*1024), megabytes, and so on An alternative is the tool buffer which has: ...


23

Asynchronous, buffered I/O was in use before Linux and even before Unix. Unix had it, and so have all its offshoots. Here is what Ritchie and Thompson wrote in their CACM paper The UNIX Time-Sharing System: To the user, both reading and writing of files appear to be synchronous and unbuffered. That is immediately after return from a read call the ...


21

Iotop is a good tool for what you want. It also allows one to display the accumulated amount of I/O on any of the DISK READ, DISK WRITE, SWAPIN, and IO (overall). This is through a nifty interface: You just press a on the keyboard, and it will sort the hungriest processes on top. Reversing the order, you just press r. If you want to sort by other colums, ...


20

There are several aspects to this question which have been addressed partially through other tools, but there doesn't appear to be a single tool that provides all the features you're looking for. iotop This tools shows which processes are consuming the most I/O. But it lacks options to show specific file names. $ sudo iotop Total DISK READ: 0.00 B/s ...


19


16

You need a utility called cpipe. Usage: tar cCf / - usr | cpipe -vr -vw -vt > /dev/null Output: ... in: 19.541ms at 6.4MB/s ( 4.7MB/s avg) 2.0MB out: 0.004ms at 30.5GB/s ( 27.1GB/s avg) 2.0MB thru: 19.865ms at 6.3MB/s ( 4.6MB/s avg) 2.0MB ...


16

F1 or h will show you the legend. It looks like in this color scheme: CPU: blue is for low priority threads green is normal priority threads black is for io-wait see below for more. Memory: green is memory in use blue is buffer orange is cache


15

The difference between cat <input >output and dd if=input of=output is in the block size they will use. The default for dd if no bs parameter is given is 512 bytes (the size of a disk sector), whereas cat will use whatever blocksize it is coded to work with, which based on a quick inspection of the source code is either 32 KB or the st_blksize value ...


15

It's been available on Linux back into its prehistory. It is not POSIX, although many actual shells (including AT&T ksh and bash) will simulate it if it's not present in the OS; note that this simulation only works at the shell level (i.e. redirection or command line parameter, not as explicit argument to e.g. open()). That said, it should be available ...


14

To process the output of a command line by line (explanation): jobs | while IFS= read -r line; do process "$line" done If you have the data in a variable already: printf %s "$foo" | … printf %s "$foo" is almost identical to echo "$foo", but prints $foo literally, whereas echo "$foo" might interpret $foo as an option to the echo command if it begins ...


13

Many good answers, but let me add one other thing... Remember that Unix is a multi-process and multi-users system, so potentially many users would be trying to do file-operations (esp. writes) at (almost) the same time. With old slow hard-disks - perhaps mounted over the network - this would not only take time (for which the programs would basically ...


12

ionice [-p] <pids/> For example: $ ionice -p `pidof X` none: prio 0 This means X is using the none scheduling class (best effort) with priority 0 (highest priority out of 7). Read more with man ionice.


12

This has been a known issue for awhile. Using an SSD-tuned FS like Btrfs might help, but it might not. Ultimately, it is a bug in the IO scheduler/memory management systems. Recently, there have been some patches that aim to address this issue. See Fixed: The Linux Desktop Responsiveness Problem? These patches may eventually make their way into the ...


12

I would give iotop a try (Linux only). It is pretty good at giving you the processes with high I/O transfers, but iotop is more for displaying the current status than logging. For logging, sar is always a good tool for performance aspects.


12

I suggest reading SwapFAQ , in particular the swapiness parameter.


12

To set niceness (CPU bound) use nice. To set IO niceness (IO bound) use ionice. Refer to the respective man pages for more information. You can use them together as follow: ionice -c 2 -n 0 nice -n -20 mplayer Note: the lowest level of niceness (lower means more favorable) you can define is determined by limits.conf. On my computer the file is located at ...


12

That is certainly not trivial task that can't be done in userspace. Fortunately, it is possible to do on Linux, using cgroup mechanizm and its blkio controller. Setting up cgroup is somehow distribution specific as it may already be mounted or even used somewhere. Here's general idea, however (assuming you have proper kernel configuration): mount -t tmpfs ...


11

I think for the most part it has been solved. My performance under heavy IO has improved in 2.6.36 and I expect it to improve more in 2.6.37. See these phoronix Articles. Wu Fengguang and KOSAKI Motohiro have published patches this week that they believe will address some of these responsiveness issues, for which they call the "system goes unresponsive ...


10

I realize this is going to sound both simplistic and absurd, but if you have control over the apps in question (maybe in a test environment) you could mount ONLY that directory on a partition of its own, then iostat, etc. would tell you only about it, and nothing else on that spot. If there are physical drives involved you could fake it up with a loopback ...


10

Some pointers: Don't run so many GUI programs at once. Make sure that any programs running in the background that you don't need e.g. Apache are stopped. Use a distro aimed at low-memory situations (e.g. for a netbook) Buy more memory. Buy a faster HD (or SSD) for your swap partition. :)


10

tee is surely a good choice, but how about this? sudo -i eval "echo test > /tmp/foo" EDIT: as per hop's comment, there is a better way sudo sh -c "echo test > /tmp/foo"


10

Disk and memory scheduling are entirely different. In the absence of an IO priority scheduler, IO will be handled on a first come first served basis. If the system is IO bound, then all processes run in a more or less round robin basis until all are waiting for I/O. The nice priority of a process will have little impact on its scheduling frequency. ...



Only top voted, non community-wiki answers of a minimum length are eligible