1

I'm not allowed to use rsync on the cluster I'm working on so I need to use things like cp or tar or find etc. I want to copy a large directory including all files and subfolders etc. while ignoring, i.e. not copying any files that for example have the names "file1", "file2" or "file3".

I saw this post on SO which allows me to include only certain files. I want basically that but I want to exclude certain files.

I also tried this solution but when I tried a command using tar which was suggested there I get the error tar: This does not look like a tar archive and tar: Cowardly refusing to create an empty archive

I also tried cp !(filename-to-ignore) new_dir/ from this post but this doesn't copy the subfolders.

I'm a beginner with bash so if you could explain your answer and what the flags etc. do that would really be helpfull, I've been trying forever now, on what I think shouldn't be that hard to do.

1
  • 1
    It's difficult to come up with good solutions when we don't know what's "allowed" on your cluster. Running rsync would be easiest, even if it meant installing it as a non-root user in your home directory. Contact the admin(s) of the cluster. They likely have a standard way of performing backups or can suggest a way for you to get data off of the cluster in the way you want it. Many backup tools have exclusion options, like --exclude in restic etc. It is their job to help you, the user of the system.
    – Kusalananda
    Nov 12, 2021 at 8:42

1 Answer 1

4

With GNU or libarchive's tar:

(cd src && tar --exclude='file[123]' -cf - .) |
  (cd dst && tar -xpf -)

Beware that GNU tar doesn't preserve the sparseness of files and depending on the version (or OS as some OS have some patches for that), not all extended attributes. libarchive's tar (often called bsdtar) does preserve extended attributes and the -S option upon extraction can be used to create sparse files.

That runs two subshells piped to each other. The first one generates a tar stream that represents the . (current) directory and all its contents recursively but ignoring the files whose name matches the test[123] pattern (similar to shell glob patterns, not regexps, here test followed by a character in the {1, 2, 3} set) after having changed the subshell's current working directory with cd. And the second has tar extract that stream on the fly in its own working directory (dst).

GNU tar also has a -C option to do that chdir() internally, as shown by Gilles in the Q&A you linked (there was a missing . in there which I've now added and explains your error), though that has little advantage over using the cd of your shell.

See also info -n exclude tar with GNU tar for the many different ways you can exclude files there and more generally info -n choosing tar to tune how files are selected/included in the archives it generates or extracts.

1
  • 1
    If your list of exclusions does not follow convenient patterns, or is unwieldy, there is an option --exclude-from=myFile that takes many patterns from your specified file. Nov 12, 2021 at 9:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.