Tag Info

Hot answers tagged

49

Yes, using tee. So echo test > /tmp/foo becomes echo test | sudo tee /tmp/foo You can also append (>>) echo test | sudo tee -a /tmp/foo


28

cpipe is probably better for these purposes, but another related program is pv (Pipe Viewer): If you give it the --rate flag it will show the transfer rate


28

Take a look at ionice. From man ionice: This program sets or gets the io scheduling class and priority for a program. If no arguments or just -p is given, ionice will query the current io scheduling class and priority for that process. To run du with the "idle" I/O class, which is the lowest priority available, you can do something like this: ionice ...


26

I've done the following test and on my system the resulting difference is about 100 times longer for the second script. My file is a strace output called bigfile $ wc -l bigfile.log 1617000 bigfile.log Scripts xtian@clafujiu:~/tmp$ cat p1.sh tail -n 1000000 bigfile.log | grep '"success": true' | wc -l tail -n 1000000 bigfile.log | grep '"success": ...


22

You can add a rate limiting tool to your pipeline. For example there is pv which has a rate-limiting option: -L RATE, --rate-limit RATE Limit the transfer to a maximum of RATE bytes per second. A suffix of "k", "m", "g", or "t" can be added to denote kilobytes (*1024), megabytes, and so on An alternative is the tool buffer which has: ...


19

You can use a while loop with process substitution: while read -r line do echo "$line" done < <(jobs) To read a multiline variable, a simple way is: printf %s "$var" | while IFS= read -r line do echo "$line" done Also, please don't call your variable jobs because that is a shell command and may cause confusion.


16

Iotop fits well with what you want. What I also love about it is that it also allows one to display the accumulated amount of I/O on any of the DISK READ, DISK WRITE, SWAPIN, and IO (overall). This is through a nifty interface: You just press a on the keyboard, and it will sort the hungriest processes on top. Reversing the order, you just press r. If you ...


16

Looking at /proc/meminfo will show the "Dirty" number shrinking over time as all the data spools out; some of it may spill into "Writeback" as well. That will be a summary against all devices, but in the cases where one device on the system is much slower than the rest you'll usually end up where everything in that queue is related to it. You'll probably ...


15

You need a utility called cpipe. Usage: tar cCf / - usr | cpipe -vr -vw -vt > /dev/null Output: ... in: 19.541ms at 6.4MB/s ( 4.7MB/s avg) 2.0MB out: 0.004ms at 30.5GB/s ( 27.1GB/s avg) 2.0MB thru: 19.865ms at 6.3MB/s ( 4.6MB/s avg) 2.0MB ...


15


12

I would give iotop a try (Linux only). It is pretty good at giving you the processes with high I/O transfers, but iotop is more for displaying the current status than logging. For logging, sar is always a good tool for performance aspects.


12

I suggest reading SwapFAQ , in particular the swapiness parameter.


12

It's been available on Linux back into its prehistory. It is not POSIX, although many actual shells (including AT&T ksh and bash) will simulate it if it's not present in the OS; note that this simulation only works at the shell level (i.e. redirection or command line parameter, not as explicit argument to e.g. open()). That said, it should be available ...


11

This has been a known issue for awhile. Using an SSD-tuned FS like Btrfs might help, but it might not. Ultimately, it is a bug in the IO scheduler/memory management systems. Recently, there have been some patches that aim to address this issue. See Fixed: The Linux Desktop Responsiveness Problem? These patches may eventually make their way into the ...


11

F1 or h will show you the legend. It looks like in this color scheme: CPU: blue is for low priority threads green is normal priority threads black is for io-wait see below for more. Memory: green is memory in use blue is buffer orange is cache


10

Some pointers: Don't run so many GUI programs at once. Make sure that any programs running in the background that you don't need e.g. Apache are stopped. Use a distro aimed at low-memory situations (e.g. for a netbook) Buy more memory. Buy a faster HD (or SSD) for your swap partition. :)


10

To set niceness (CPU bound) use nice. To set IO niceness (IO bound) use ionice. Refer to the respective man pages for more information. You can use them together as follow: ionice -c 2 -n 0 nice -n -20 mplayer Note: the lowest level of niceness (lower means more favorable) you can define is determined by limits.conf. On my computer the file is located at ...


10

I think for the most part it has been solved. My performance under heavy IO has improved in 2.6.36 and I expect it to improve more in 2.6.37. See these phoronix Articles. Wu Fengguang and KOSAKI Motohiro have published patches this week that they believe will address some of these responsiveness issues, for which they call the "system goes unresponsive ...


10

To process the output of a command line by line (explanation): jobs | while IFS= read -r line; do process "$line" done If you have the data in a variable already: printf %s "$foo" | … printf %s "$foo" is almost identical to echo "$foo", but prints $foo literally, whereas echo "$foo" might interpret $foo as an option to the echo command if it begins ...


10

Disk and memory scheduling are entirely different. In the absence of an IO priority scheduler, IO will be handled on a first come first served basis. If the system is IO bound, then all processes run in a more or less round robin basis until all are waiting for I/O. The nice priority of a process will have little impact on its scheduling frequency. ...


10

There are several aspects to this question which have been addressed partially through other tools, but there doesn't appear to be a single tool that provides all the features you're looking for. iotop This tools shows which processes are consuming the most I/O. But it lacks options to show specific file names. $ sudo iotop Total DISK READ: 0.00 B/s ...


10

The reading operation will succeed, regardless from the time it takes to complete the reading operation. Why and how does this work? When the reading operation starts, the file's Inode is used as a handle from which the file's content is read. When moving another file to the target file, the result will be a new inode, which means the physical content of ...


9

I realize this is going to sound both simplistic and absurd, but if you have control over the apps in question (maybe in a test environment) you could mount ONLY that directory on a partition of its own, then iostat, etc. would tell you only about it, and nothing else on that spot. If there are physical drives involved you could fake it up with a loopback ...


9

tee is surely a good choice, but how about this? sudo -i eval "echo test > /tmp/foo" EDIT: as per hop's comment, there is a better way sudo sh -c "echo test > /tmp/foo"


9

.gvfs directories are mount points (sometimes). You may want to use the one_fs option in your rsnapshot configuration (so that it passes --one-file-system to rsync). Gvfs is a library-level filesystem implementation, implemented in libraries written by the Gnome project (in particular libgvfscommon). Applications linked with this library can use a ...


9

You're doing small random writes, which is pretty much the slowest thing you can do on a spinning disk, so I would say your throughput meets (my) expectations. Your avgrq-sz size is 15.35, which means your average request is 15.35 x the sector size of your SATA disk (most commonly 512 bytes, but possibly 4096 bytes on a very new SATA disk), so you're ...


9

The pv (pipe viewer) utility can do this (with the -B option) and a lot more, including giving you progress reports.


9

> writes to a file, overwriting any existing contents. >> appends to a file. From man bash: Redirecting Output Redirection of output causes the file whose name results from the expansion of word to be opened for writing on file descriptor n, or the standard output (file descriptor 1) if n is not specified. If the file does not exist ...


9

On the one hand, the first method calls tail twice, so it has to do more work than the second method which only does this once. On the other hand, the second method has to copy the data into the shell and then back out, so it has to do more work than the first version where tail is directly piped into grep. The first method has an extra advantage on a ...


8

ionice [-p] <pids/> For example: $ ionice -p `pidof X` none: prio 0 This means X is using the none scheduling class (best effort) with priority 0 (highest priority out of 7). Read more with man ionice.



Only top voted, non community-wiki answers of a minimum length are eligible