119

When moving large directories using mv, is there a way to view the progress (%)? The cp command on gentoo had a -g switch that showed the progress.

12 Answers 12

50

You can build a patched cp and mv which then both support the -g switch to show progress. There are instructions and patches at this page. However: The page instructs you to do

$ sudo cp src/cp /usr/bin/cp
$ sudo cp src/mv /usr/bin/mv

which overwrites the original cp and mv. This has two disadvantages: Firstly, if an updated coreutils package arrives at your system, they are overwritten. Secondly, if the patched version has a problem, they might break scripts relying on standard cp and mv. I would rather do something like this:

$ sudo cp src/cp /usr/local/bin/cpg
$ sudo cp src/mv /usr/local/bin/mvg

which copies the files to /usr/local/bin which is intended for user compiled programs and gives them a different name. So when you want a progress bar, you say mvg -g bigfile /mnt/backup and use mv normally.

Also you can do alias mvg="/usr/local/mvg -g" then you only need to say mvg bigfile /mnt/backup and directly get the progress bar.

93

I don't like the idea to overwrite binaries from coreutil when there are simpler solutions, so here are mine:

rsync: Rsync copies files and has a -P switch for a progress bar. So if you have rsync installed, you could use a simple alias in your shells dotfile:

alias cp='rsync -aP'

The downside is, that rsync is a little bit slower than cp, but you should measure this with time and decide for your self, I can live with it :-)

Shell Script: A shell script can also create the progress bar. I found this a while ago on the net and I don't remember the source:

#!/bin/sh
cp_p()
{
   strace -q -ewrite cp -- "${1}" "${2}" 2>&1 \
      | awk '{
        count += $NF
            if (count % 10 == 0) {
               percent = count / total_size * 100
               printf "%3d%% [", percent
               for (i=0;i<=percent;i++)
                  printf "="
               printf ">"
               for (i=percent;i<100;i++)
                  printf " "
               printf "]\r"
            }
         }
         END { print "" }' total_size=$(stat -c '%s' "${1}") count=0
}

This will look like:

% cp_p /home/echox/foo.dat /home/echox/bar.dat
66% [===============================>                      ]

bar:

‘bar’ - ‘cat’ with ASCII progress bar

bar is a small shell script to display a process bar for all kind of operations (cp, tar, etc.). You can find examples on the project homepage.

Its also written for the bourne shell, so it will run nearby everywhere.

  • 3
    bar doesn't wrap mv yet. – Tobu Jan 2 '11 at 23:04
  • 5
    Credit for the shell script – Lamnk Jun 26 '11 at 16:42
  • 1
    Note that rsync doesn't replace mv (the OP is asking about) completely, especially when it comes to fast copies on copy-on-write filesystems like btrfs. – Karl Richter May 29 '16 at 14:40
76

There's a new tool called progress that can find any descriptor related to a running command and show progress and speed: available here

progress -w

outputs the stats for all running cp,mv etc. operations

  • 3
    WOW, I did not expect this to be so awesome! I simply typed progress after installing and it presented me with progress for the mv command I was currently waiting for. Thanks! – jredd Sep 30 '15 at 0:59
  • 3
    Thanks! When moving stuff as a root it seems you have to use sudo (same user I guess) - you can also use watch to say what is happening every 2 seconds: watch progress -w – Wilf May 30 '16 at 20:13
  • 6
    Awesome program, but when moving e.g. a folder with many files it only shows the progress of the current file, not the whole folder in total. Instead of watch progress -w you can also simply use progress -m – mxmlnkn Nov 13 '16 at 15:38
  • 7
    IMHO, this is the most unix-way to solve the problem. Instead of patching coreutils or using tools which are not designated for specific purposes (like rsync for mv) or writing overkilling scripts, just use that tool written for the subject purpose with love. – Grief Dec 28 '16 at 1:54
34

You can use pipe viewer command pv to show progress bar:

pv /original/file > /new/file

I often use this to copy a big file over a mounted network filesystem (combine with gzip and tar). The drawback is that you can only copy one file and not directory. And you must give the new file a name, you can not just give destination directory like cp does. However copying is not pv's purpose. It is a powerful tool and do much more than just copy file. See the homepage for more examples of pv.

A much better option is to use rsync -aP. If you want to mv instead, append the flag --remove-source-files. Add this to your .bashrc if you want to use the commands frequently:

alias rscp='rsync -aP'
alias rsmv='rsync -aP --remove-source-files'

The downside here is rsync only shows progress, not a progress bar.

  • tar cf - source-dir | pv | tar xf - -C out-dir will give you throughput of a whole directory. If you know the size and pass -s to pv it can give you progress as well. – arantius Feb 10 '18 at 17:34
6

If your goal is to move/copy a directory with progress bar, but avoiding non-terminal GUI, mc (Midnight Commander) is a good choice.

3

My solution is to use rsync. It can copy directories, remove the source files after a successful copy (thus "moving" them) and display progress reports, among many other features (most notably syncing partially copied directories and working over the network.)

Here is how I would move a directory with rsync, with progress report:

rsync -aP --remove-source-files $src $dst && rm -r $src

Make sure $src does not end in a backslash, otherwise you will get a different meaning than that of GNU cp.

The rm -r at the end is needed because --remove-source-files only removes the source files, after each one is successfully copied over, not the directories. The && makes sure to run it only if rsync completes successfully. If you are paranoid, you can replace it with some kind of recursive rmdir, such as find $src -d -exec rmdir {} +

The only glitch is that the -P option shows progress for each file, not for the entire copy operation. This might be useless if you are trying to move a directory with many tiny files. The latest development version of rsync has an alternate flag --info=progress2 that shows progress for the entire operation.

2

To avoid coreutil patching to add progress bar to mv and cp, gcp (Goffi's CoPier) is a new alternative: http://wiki.goffi.org/wiki/Gcp/en

To use the same way as cp with options -r for recursivity and -v for verbosity.

It uses a file queue when multiple commands are invoked concurrently. It also allows to record copied file names in a log, see Source saving man page section for details.

Here is a good tutorial to get started with gcp: http://mylinuxbook.com/gcp-advanced-command-line-file-copier-inspired-by-cp/

  • Is there gmv like gcp for cp? – CodyChan Sep 1 '16 at 6:07
  • I have found no equivalent. By the way, there is no need for progress bar when running "mv" on a single file system... it is quite fast. If move is done across two different file systems, it is implemented as cp and then remove source if succeed, or remove target if failed. A simple shell script based on gcp may do it properly (with a first check to know if source and target are really on different file systems) – Yves Martin Sep 1 '16 at 6:23
2

First off: I never copy large files without using ionice, unless I know that I will not want to use the computer for half an hour or more.

Second: all my partitions are jouranled so intrapartition copying takes not time. If it is a long copy I do a du -sm on the files and df -m | grep copy_to_partition. Then if curious how much more time it will take I do the df again and see how much of the files was copied.

  • Would you care to share your ionice tips ? what priority would you give to mv cpu-wise and disk i/o wise ? Here are my settings but the CPU load is still high even after renicing : imgur.com/a/DLMJplz – ychaouche Feb 17 at 11:55
2

On my Mint I like to use gcp on this or the curl-solution (which is really nice)

gcp SOURCE DESTINATION

You might need to install it sudo apt-get install gcp

And a very nice and excellent task for the progress bar with info about

  • % Total
  • % Received
  • % Xferd
  • Average DL
  • Speed Upload
  • Time left
  • Time spent
  • Time total
  • Current Speed

in a nice readable format is:

curl -o DESTINATION FILE://source

Also, you might need to install it first sudo apt-get install curl

  • gcp doesn't support moving of file, i.e. source files have to be removed manually after successful copying. – Karl Richter May 29 '16 at 14:31
2

This does not give you a progress bar, but hit CTRL-T and you will send a SIGINFO to mv, which will report which file it's working on and how far along in the process it is. Pretty handy to have this for large file transfers, possibly not as useful for a transfer with many small files.

Quick example:

mv 135/61 /Volumes/Jules/135/
# hit C-t
load: 1.08  cmd: cp 35470 uninterruptible 0.00u 0.04s
135/61/1/1-s2.0-S006738-main.pdf -> /Volumes/Jules/135/61/1/1-s2.0-S006738-main.pdf  68%
  • 1
    This didn't work for me: I got INFO: unknown signal; killall -l lists signals. I think SIGINFO is only on some BSDs, not in standard POSIX. – Brian Minton Aug 26 '16 at 21:42
  • 2
    Right. It does work in Mac OS X, but likely not in, say, Ubuntu. – Charles Aug 27 '16 at 9:12
1

Use mv -v to print one line per moved file and pv -l to print progress based on line count (in effect file count).

You will need pv (pipe viewer): http://www.ivarch.com/programs/pv.shtml

mv -v sourcedir targetdir | pv -l -s filecount > logfile

Use the following command to get file count:

find sourcedir | wc -l

This command will show progress information based on number of files copied. This works best if there are many smallish files. If there are only a few files which are huge then you will not have much fun.

Note that mv on the same partition should be fast. mv to another partition would be the same as cp source target followed by rm source. So here are two commands to copy directories with progress information.

to report progress based on number of files copied:

cp -av sourcedir targetdir | pv -l -s filecount > logfile

to report progress based on bytes copied:

tar c sourcedir | pv -s size | tar x -C targetdir

See here for more details: progress information via pv for directory copy

1

Invariably I end up searching for this answer after I start the 'mv' command, at which point I don't want to start it again. If you're in this situation a very simple way to monitor the progress (or to see if it's far enough along that it's not worth killing) is to run:

du -hs /directory/you're/moving/data/to

This will just print the size of the directory so you can see how much data has been copied, or you can run it a couple of times to see how fast the data is being transfered.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.