1

My systems go under heavy load when I run

sudo tar czf /media/masi/ntfsDisc/backup_home.tar.gz $HOME/

Fans goes to max etc. I would like to find a better balance between computing and power consumption. I cannot monitor the process enough well. You cannot slow it down during the computation when run like that. Intuition: add some sleep there but how. I really would like to have an xargs approach too, to compare it to "ready" products. My tops

  • I do top at rest

    top - 09:34:34 up 19:14,  1 user,  load average: 0.52, 0.42, 0.24
    Tasks: 236 total,   1 running, 235 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  1.5 us,  1.1 sy,  0.0 ni, 97.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 s
    KiB Mem :  8115460 total,   257036 free,  3006452 used,  4851972 buff/cache
    KiB Swap:  8326140 total,  8321852 free,     4288 used.  4369448 avail Mem 
    
  • I do top 1 minute after the nice tar czf ...

    top - 09:48:49 up 19:28,  1 user,  load average: 1.63, 0.99, 0.62
    Tasks: 244 total,   2 running, 242 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  1.4 us,  0.9 sy, 24.1 ni, 73.2 id,  0.3 wa,  0.0 hi,  0.1 si,  0.0 s
    KiB Mem :  8115460 total,   127644 free,  3237648 used,  4750168 buff/cache
    KiB Swap:  8326140 total,  8321868 free,     4272 used.  4092404 avail Mem 
    
      PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND   
    28831 root      30  10    4640   1600   1316 R  97.7  0.0   1:43.24 gzip      
     9573 root      20   0   21196   2860   1772 S   2.3  0.0  13:16.29 mount.nt+ 
      842 root      20   0  380136  63780  48568 S   1.7  0.8  23:57.16 Xorg      
    
  • I do top 10 minutes after the start

    top - 10:00:33 up 19:40,  1 user,  load average: 1.98, 2.13, 1.50
    Tasks: 253 total,   2 running, 251 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  2.6 us,  2.8 sy, 21.4 ni, 73.0 id,  0.2 wa,  0.0 hi,  0.0 si,  0.0 s
    KiB Mem :  8115460 total,   130408 free,  4432384 used,  3552668 buff/cache
    KiB Swap:  8326140 total,  8321948 free,     4192 used.  2837616 avail Mem 
    
      PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND   
    28831 root      30  10    4640   1600   1316 R  87.0  0.0  11:49.08 gzip      
     9573 root      20   0   21196   2860   1772 S  13.6  0.0  14:45.84 mount.nt+ 
      842 root      20   0  384936  66304  51092 S   2.0  0.8  24:18.44 Xorg      
    28830 root      30  10   37584   3096   2688 S   1.3  0.0   0:14.50 tar       
     1674
    

My PV tar cf - $HOME/ | pv | gzip > media/masi/ntfsDisc/testbackup.tar.gz

  • at 1 min, 13 - 22 MB/s; at 2 min, 14 - 22 MB/s; at 3 min, 5 - 7 MB/s; at 4 min, 5 - 22 MB/s; at 5 min, 15 - 17 MB/s; at 6 min, 8 - 24 MB/s; at 7 min, 16 - 20 MB/s
  • at 19 min, 18-21 MB/s and fans little/steadily on such that you can hear them

System: Ubuntu 16.04 64 bit
Hardware: Macbook Air 2013-mid

  • Can you run top and show us what's using up CPU time? If it's the NTFS FUSE program, parallelization of the tar/gzip probably won't help. – Mark Plotnick Jun 24 '16 at 0:35
  • Thanks for providing the top output. If your goal is to minimize symptoms of high CPU load such as loud fan noise, I think the best option is to separate tar from gzip with a pipeline that has pv in it to rate limit the data that is being fed to gzip, which is the largest CPU user. Do you have the pv program? – Mark Plotnick Jun 27 '16 at 19:51
  • I need one more piece of info, please. After you've installed pv, please run tar cf - $HOME/ | pv | gzip > /media/masi/ntfsDisc/testbackup.tar.gz and look at the transfer rate that pv outputs, something like [80.3MiB/s]. Let us know what that number is after a few minutes. We can then recommend an amount to try to rate-limit the data. – Mark Plotnick Jun 28 '16 at 15:48
1

The command

tar czf /media/masi/ntfsDisc/backup_home.tar.gz $HOME/

is the same as this:

tar cf - $HOME/ | gzip > /media/masi/ntfsDisc/backup_home.tar.gz

When you ran top, it showed the gzip was using up around 100% of one cpu thread. The NTFS FUSE software is using up a nonzero amount of CPU, too, but essentially you're CPU-bound because of that gzip. Your load average is around 2, and with 2 cores of 2 threads each, you're not overloading your system.

But, if your goal is to reduce the max CPU usage (because the fans are running at their max), one easy way to do this is to slow down the rate of the data that's being fed to gzip.

You ran the test

tar cf - $HOME/ | pv | gzip > /media/masi/ntfsDisc/testbackup.tar.gz

and pv indicated the peak transfer rate into gzip was 20MiB/sec. I'd recommend cutting that in half by giving pv the -L 10m option.

tar cf - $HOME/ | pv -L 10m | gzip > /media/masi/ntfsDisc/testbackup.tar.gz

Try adjusting that rate limit higher or lower until you get the CPU usage you like.

3

First of all, the the overall power consumption will be likely the same or more if you artificially slow down the backup process. Simply because the total number of operations is the same and if the process takes longer, the cpu consumes less peak power but over a longer time. For example, if the process runs for 10 s at peak power of 200W this will consume 10s*200W=2000J, if the process runs for 100s at 30W this will consume 100s*30W=3000J.

If you are mainly after improving the responsiveness of your computer during the process You could try increasing the niceness of the process (nice will lower the cpu priority, freeing cpu power for other processes, ionice will reduce the disk priority, freeing the disk i/o for other processes):

sudo nice -n19 ionice -c2 -n7 tar czf /media/masi/ntfsDisc/backup_home.tar.gz $HOME/

This will reduce the priority of the process so that it does not slow down other processes while you are working on the machine. Other than that it will still try to finish the process as fast as possible and cause your fans to spin up.

If you really want/need to reduce the peak power consumption (because your system overheats or the fans wake you up at night) you could try one of the following approaches:

A more elaborate solution would be not to compress everything at once but directory by directory (put this code into a file called backup_home.sh, make it executable and run it via sudo backup_home.sh):

#!/bin/bash
OLDIFS=$IFS
IFS='
'
for dir in $(ls -d1 $HOME/*); do
   nice tar rf /media/masi/ntfsDisc/backup_home.tar $HOME/
   sleep 10
done;
gzip /media/masi/ntfsDisc/backup_home.tar
IFS=$OLDIFS

Notice however, that the overall power consumption will not be reduced, it will simply spread over a longer time (increasing the possibility that a file is changed during the backup. Also this will not evenly distribute the load since likely not all folders have the same size. I highly recommend you use nice and leave the rest to the system.

Finally, if you really want to dive into it, you could use CPU frequency scaling to manually underclock your CPU for the duration of the backup

  • What do you think about pigz approach? How do you compare nice to it? – Léo Léopold Hertz 준영 Jun 23 '16 at 21:13
  • @Masi pigz would go faster and heat more, which is exactly the opposite of what the question asks. – Gilles 'SO- stop being evil' Jun 23 '16 at 22:54
  • Niceness won't change anything here for two reasons. One is that the operation is likely to be IO-bound (it might be CPU-bound with a more intense compression algorithm, but with gzip that's unlikely). The other is that even if the operation is IO-bound, nice only has the effect of giving other processes running at the same time a higher priority. It can never reduce the total load on the processor. – Gilles 'SO- stop being evil' Jun 23 '16 at 22:57
  • @Gilles you are totally correct on all accounts (as I also pointed out in the answer) however, increasing the niceness might still be exactly what the OP (or other people stumbling upon this question) want to achieve, therefore I put that into the answer. For the heating problems I added, the by directory and the cpufreq approach. – Thawn Jun 25 '16 at 10:23
  • @Gilles thanks for mentioning the i/o. I added ionice to account for that. – Thawn Jun 25 '16 at 10:33
2

One approach would be to use parallel compression in order to use all of the cores of your system and therefore reduce the compression time. It won't reduce the load on your system, but it will be loaded for a shortest amount of time !

You can find how to do it in this Q/A for example : utilizing-multi-core-for-targzip-bzip-compression-decompression

For example:

tar cf - paths-to-archive | pigz > archive.tar.gz
  • Yep, this command is correct. There is no z in the tar command because compression is performed by "pigz" (Parallel Implementation of GZip) in which we pipe the output of the tar command. – Dieter G Jun 23 '16 at 20:42
  • You can set more cores (in fact more threads) than your computer has cores, but it would probably wont be very efficient. It will result in permanent content switching on your CPUs and then waste precious computing time. – Dieter G Jun 23 '16 at 20:44
  • 1
    My recommendation is to use it without parameters. It will then stick to the number of cores on your system and therefore it is portable to the various system you will run it into. – Dieter G Jun 23 '16 at 20:49
  • 1
    As the others said, with you will reduce the priority of the process. It will then augment the time taken to perform the compression, whereas the parallel processing will reduce it. It is up to you knowing all these informations to decide which method works best for you specific need. – Dieter G Jun 23 '16 at 21:41
  • 1
    Heating multiple cores in parallel makes the computer heat more for a shorter time, which is exactly the opposite of what the question requests. – Gilles 'SO- stop being evil' Jun 23 '16 at 22:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.