cp hidden files with glob patterns

Question

Situation :

$ mkdir foo && touch foo/.test
$ cp foo/* .
zsh: no matches found: foo/*
(or bash : cp: cannot stat ‘foo/*’: No such file or directory)

I have a directory full of hidden folders and files. What is happening and what is the solution?

Answer 1

Disclaimer: This answer deals with Bash specifically but much of it applies to the question regarding glob patterns!

The star character (*) is a wildcard. There are a certain set of characters that it will take the place of and the first character being a dot (.) isn't one of them. This is a special case just because of how the Unix filesystems work, files that start with a dot are considered "hidden". That means that tools such as cp, ls, etc. will not "see" them unless explicitly told to do so.

Examples

First let's create some sample data.

$ mkdir .dotdir{1,2} regdir{1,2}
$ touch .dotfile{1,2} regfile{1..3}

So now we have the following:

$ tree -a
.
|-- .dotdir1
|-- .dotdir2
|-- .dotfile1
|-- .dotfile2
|-- regdir1
|-- regdir2
|-- regfile1
|-- regfile2
`-- regfile3

Now let's play some games. You can use the command echo to list out what a particular wildcard (*) would be for a given command like so:

$ echo *
regdir1 regdir2 regfile1 regfile2 regfile3


$ echo reg*
regdir1 regdir2 regfile1 regfile2 regfile3

$ echo .*
. .. .dotdir1 .dotdir2 .dotfile1 .dotfile2

$ echo .* *
. .. .dotdir1 .dotdir2 .dotfile1 .dotfile2 regdir1 regdir2 regfile1 regfile2 regfile3

$ echo .dotdir*
.dotdir1 .dotdir2

Changing the behavior?

You can use the command shopt -s dotglob to change the behavior of the * so that in addition to files like regfile1 it will also match .dotfile1.

excerpt from the bash man page

dotglob If set, bash includes filenames beginning with a `.' in the results 
        of pathname expansion.

Example:

$ shopt -s dotglob
$ echo *
.dotdir1 .dotdir2 .dotfile1 .dotfile2 regdir1 regdir2 regfile1 regfile2 regfile3

You can revert this behavior with this command:

$ shopt -u dotglob
$ echo *
regdir1 regdir2 regfile1 regfile2 regfile3

Your situation?

For you you're telling cp that you want to copy all the files that match the pattern *, and there aren't any files.

$ cp foo/.* .

Or you can do this if you want everything in the foo folder:

$ cp foo .

Or you can be explicit:

$ cp foot/.* foo/* .

A more compact form using brace expansion in bash:

$ cp foo/{.,}* .

At any time you can use the echo trick to see what your proposed file patterns (that's the fancy term for what the star is a part of).

$ echo {.,}*
. .. .dotdir1 .dotdir2 .dotfile1 .dotfile2 abc regdir1 regdir2 regfile1 regfile2 regfile3

Incidentally if you're going to copy a directory of files + other directories, you typically want to do this recursively, that's the -R switch to cp:

$ cp -R foo/. .

Answer 2

With zsh, the typical way is to use the D glob qualifier (to include [D]ot files):

cp foo/*(D) .

Note that with or without D, zsh globs never include . nor .. (as you'd expect).

Answer 3

The shell is treating those files as hidden when it resolves the * character, so cp doesn't receive any of these file names as arguments.

You can copy them by explicitly specifying cp foo/.* .

Answer 4

Stéphane Chazelas's zsh answer is correct for a single glob. However, if you want to set it for all globs you can use

setopt GLOB_DOTS

Put it in your ~/.zshrc to make it permanent.

Answer 5

If what you're wanting to do is copy all files & directories from one place to another, you could use the standard rsync command. In the example you give above:

mkdir foo && touch foo/.test
rsync -a foo/ .

will recursively copy all of the contents of foo, including hidden files and hidden directories, into the current directory. The trailing slash at the end of foo/ is important to rsync; with it only the contents of foo are copied, without it rsync will copy foo as well. For example:-

mkdir src && mkdir dest && touch src/.test
rsync -a src  dest    // copies 'src' contents to 'dest/src'
rsync -a src/ dest    // copies 'src' contents to 'dest' 

There are many other options available tweak rsync, including copying between machines.