Tag Info

Hot answers tagged

50

Yes, using tee. So echo test > /tmp/foo becomes echo test | sudo tee /tmp/foo You can also append (>>) echo test | sudo tee -a /tmp/foo


30

cpipe is probably better for these purposes, but another related program is pv (Pipe Viewer): If you give it the --rate flag it will show the transfer rate


28

Take a look at ionice. From man ionice: This program sets or gets the io scheduling class and priority for a program. If no arguments or just -p is given, ionice will query the current io scheduling class and priority for that process. To run du with the "idle" I/O class, which is the lowest priority available, you can do something like this: ionice ...


26

I've done the following test and on my system the resulting difference is about 100 times longer for the second script. My file is a strace output called bigfile $ wc -l bigfile.log 1617000 bigfile.log Scripts xtian@clafujiu:~/tmp$ cat p1.sh tail -n 1000000 bigfile.log | grep '"success": true' | wc -l tail -n 1000000 bigfile.log | grep '"success": ...


22

You can add a rate limiting tool to your pipeline. For example there is pv which has a rate-limiting option: -L RATE, --rate-limit RATE Limit the transfer to a maximum of RATE bytes per second. A suffix of "k", "m", "g", or "t" can be added to denote kilobytes (*1024), megabytes, and so on An alternative is the tool buffer which has: ...


20

You can use a while loop with process substitution: while read -r line do echo "$line" done < <(jobs) To read a multiline variable, a simple way is: printf %s "$var" | while IFS= read -r line do echo "$line" done Also, please don't call your variable jobs because that is a shell command and may cause confusion.


18

Iotop is a good tool for what you want. It also allows one to display the accumulated amount of I/O on any of the DISK READ, DISK WRITE, SWAPIN, and IO (overall). This is through a nifty interface: You just press a on the keyboard, and it will sort the hungriest processes on top. Reversing the order, you just press r. If you want to sort by other colums, ...


18

Looking at /proc/meminfo will show the "Dirty" number shrinking over time as all the data spools out; some of it may spill into "Writeback" as well. That will be a summary against all devices, but in the cases where one device on the system is much slower than the rest you'll usually end up where everything in that queue is related to it. You'll probably ...


16

You need a utility called cpipe. Usage: tar cCf / - usr | cpipe -vr -vw -vt > /dev/null Output: ... in: 19.541ms at 6.4MB/s ( 4.7MB/s avg) 2.0MB out: 0.004ms at 30.5GB/s ( 27.1GB/s avg) 2.0MB thru: 19.865ms at 6.3MB/s ( 4.6MB/s avg) 2.0MB ...


15


13

It's been available on Linux back into its prehistory. It is not POSIX, although many actual shells (including AT&T ksh and bash) will simulate it if it's not present in the OS; note that this simulation only works at the shell level (i.e. redirection or command line parameter, not as explicit argument to e.g. open()). That said, it should be available ...


13

F1 or h will show you the legend. It looks like in this color scheme: CPU: blue is for low priority threads green is normal priority threads black is for io-wait see below for more. Memory: green is memory in use blue is buffer orange is cache


12

This has been a known issue for awhile. Using an SSD-tuned FS like Btrfs might help, but it might not. Ultimately, it is a bug in the IO scheduler/memory management systems. Recently, there have been some patches that aim to address this issue. See Fixed: The Linux Desktop Responsiveness Problem? These patches may eventually make their way into the ...


12

I would give iotop a try (Linux only). It is pretty good at giving you the processes with high I/O transfers, but iotop is more for displaying the current status than logging. For logging, sar is always a good tool for performance aspects.


12

I suggest reading SwapFAQ , in particular the swapiness parameter.


12

There are several aspects to this question which have been addressed partially through other tools, but there doesn't appear to be a single tool that provides all the features you're looking for. iotop This tools shows which processes are consuming the most I/O. But it lacks options to show specific file names. $ sudo iotop Total DISK READ: 0.00 B/s ...


11

To set niceness (CPU bound) use nice. To set IO niceness (IO bound) use ionice. Refer to the respective man pages for more information. You can use them together as follow: ionice -c 2 -n 0 nice -n -20 mplayer Note: the lowest level of niceness (lower means more favorable) you can define is determined by limits.conf. On my computer the file is located at ...


11

To process the output of a command line by line (explanation): jobs | while IFS= read -r line; do process "$line" done If you have the data in a variable already: printf %s "$foo" | … printf %s "$foo" is almost identical to echo "$foo", but prints $foo literally, whereas echo "$foo" might interpret $foo as an option to the echo command if it begins ...


10

I think for the most part it has been solved. My performance under heavy IO has improved in 2.6.36 and I expect it to improve more in 2.6.37. See these phoronix Articles. Wu Fengguang and KOSAKI Motohiro have published patches this week that they believe will address some of these responsiveness issues, for which they call the "system goes unresponsive ...


10

Some pointers: Don't run so many GUI programs at once. Make sure that any programs running in the background that you don't need e.g. Apache are stopped. Use a distro aimed at low-memory situations (e.g. for a netbook) Buy more memory. Buy a faster HD (or SSD) for your swap partition. :)


10

tee is surely a good choice, but how about this? sudo -i eval "echo test > /tmp/foo" EDIT: as per hop's comment, there is a better way sudo sh -c "echo test > /tmp/foo"


10

Disk and memory scheduling are entirely different. In the absence of an IO priority scheduler, IO will be handled on a first come first served basis. If the system is IO bound, then all processes run in a more or less round robin basis until all are waiting for I/O. The nice priority of a process will have little impact on its scheduling frequency. ...


10

You're doing small random writes, which is pretty much the slowest thing you can do on a spinning disk, so I would say your throughput meets (my) expectations. Your avgrq-sz size is 15.35, which means your average request is 15.35 x the sector size of your SATA disk (most commonly 512 bytes, but possibly 4096 bytes on a very new SATA disk), so you're ...


10

That is certainly not trivial task that can't be done in userspace. Fortunately, it is possible to do on Linux, using cgroup mechanizm and its blkio controller. Setting up cgroup is somehow distribution specific as it may already be mounted or even used somewhere. Here's general idea, however (assuming you have proper kernel configuration): mount tmpfs ...


10

The reading operation will succeed, regardless from the time it takes to complete the reading operation. Why and how does this work? When the reading operation starts, the file's Inode is used as a handle from which the file's content is read. When moving another file to the target file, the result will be a new inode, which means the physical content of ...


10

On the one hand, the first method calls tail twice, so it has to do more work than the second method which only does this once. On the other hand, the second method has to copy the data into the shell and then back out, so it has to do more work than the first version where tail is directly piped into grep. The first method has an extra advantage on a ...


9

ionice [-p] <pids/> For example: $ ionice -p `pidof X` none: prio 0 This means X is using the none scheduling class (best effort) with priority 0 (highest priority out of 7). Read more with man ionice.


9

.gvfs directories are mount points (sometimes). You may want to use the one_fs option in your rsnapshot configuration (so that it passes --one-file-system to rsync). Gvfs is a library-level filesystem implementation, implemented in libraries written by the Gnome project (in particular libgvfscommon). Applications linked with this library can use a ...


9

I realize this is going to sound both simplistic and absurd, but if you have control over the apps in question (maybe in a test environment) you could mount ONLY that directory on a partition of its own, then iostat, etc. would tell you only about it, and nothing else on that spot. If there are physical drives involved you could fake it up with a loopback ...


9

GNU Parallel is specifcally designed to solve this problem: echo -n $IPs | parallel -d ' ' -j0 wget -q -O- http://{}/somepage.html | egrep --count '^string' If your IPs are in a file it is even prettier: cat IPs | parallel -j0 wget -q -O- http://{}/somepage.html | egrep --count '^string' To learn more watch the intro video: ...



Only top voted, non community-wiki answers of a minimum length are eligible