There can be plenty of ways, though all require some kind of complexity in order to handle the override case.
As a one-liner, though a bit long, you could do like this for one iteration i.e. one "environments" directory:
(r=Products; e=stage; (find -- "$r" -regextype posix-extended -maxdepth 2 \( -regex '^[^/]+(/[^/]+)?' -o ! -type d \) -print0; find -- "$r" -mindepth 1 -path "$r/*/$e/*" -print0) | tar --null --no-recursion -czf "$r-$e.tgz" -T- --transform=s'%^\(\([^/]\{1,\}/\)\{2\}\)[^/]\{1,\}/%\1%')
broken down to observe it better:
(
r=Products; e=stage
(
find -- "$r" -regextype posix-extended -maxdepth 2 \( -regex '^[^/]+(/[^/]+)?' -o ! -type d \) -print0
find -- "$r" -mindepth 1 -path "$r/*/$e/*" -print0
) \
| tar --null --no-recursion -czf "$r-$e.tgz" -T- \
--transform=s'%^\(\([^/]\{1,\}/\)\{2\}\)[^/]\{1,\}/%\1%'
)
Things to note:
- it shows GNU tools' syntax. For BSD
find you must replace -regextype posix-extended with just -E and for BSD tar you must replace --no-recursion with just -n as well as --transform=s (<- note the final s) with just -s
- for simplicity of demonstration the snippet assumes to be run from the directory containing
Products, and uses the custom $e variable for the name of the "environments" directory to archive, while $r is just a short-named helper variable to contain the Products name
- it is enclosed within parentheses, making it a subshell, just so as not to pollute your shell with
$r and $e should you run it from the command-line
- it does not copy nor link/refer to the original files, it does handle any valid filename, it has no memory constraints, and it can handle any amount of names; the only assumption is about the first two levels of the directories hierarchy in that any directory directly below the first level is considered an "environments" directory and thus ignored (except the one indicated in
$e)
You could simply enclose that snippet in a for e in dev prod stage; do ...; done shell loop and just go. (possibly taking away the outermost parentheses and rather surround the entire for loop).
The upside is that it is quite short and relatively simple after all.
The downside is that it always archives also all the overridden files (i.e. the base ones), the trick being just that the double find commands feed tar with the to-be-overridden files first, and hence during extraction they will be overwritten by the overriding files (i.e. the "environments" specific ones). This leads to a bigger archive taking more time both during creation and during extraction, and might be undesirable depending on whether such "overhead" can be negligible or not.
That pipeline described in prose is:
- (besides the outermost parentheses and the helper variables)
- the first
find command produces the list of non-specific files (and leading directories as per your update) only, while the second find produces the list of all environments-specific files only
- the two
find commands are within parentheses by themselves so that both their outputs feed the pipe to tar in sequence
tar reads such pipe in order to get the names of the files, and puts those files in the archive while also --transform-ing their names by eliminating the "environments" component (if present) from the path-name of each file- the two
find commands are separated instead of being just one, and they are run one after the other, so that the non-specific files are produced (for tar to consume) before the environments-specific files, which enables the trick I described earlier
To avoid the overhead of including always all the files we need additional complexity in order to truly purge the overridden files. One way might be like below:
# still a pipeline, but this time I won't even pretend it to be a one-liner
(
r=Products; e=stage; LC_ALL=C
find -- "$r" -regextype posix-extended \( -path "$r/*/$e/*" -o \( -regex '^([^/]+/){2}[^/]+' ! -type d \) -o -regex '^[^/]+(/[^/]+)?' \) -print0 \
| sed -zE '\%^(([^/]+/){2})([^/]+/)%s%%0/\3\1%;t;s%^%1//%' \
| sort -zt/ -k 3 -k 1,1n \
| sort -zut/ -k 3 \
| sed -zE 's%^[01]/(([^/]+/)|/)(([^/]+/?){2})%\3\2%' \
| tar --null --no-recursion -czf "$r-$e.tgz" -T- \
--transform=s'%^\(\([^/]\{1,\}/\)\{2\}\)[^/]\{1,\}/%\1%'
)
Several things to note:
- everything we said earlier regarding GNU and BSD syntaxes for
find and tar applies here as well
- like the previous solution, it has no constraints whatsoever besides the assumption about the first two levels of the directories hierarchy
- I'm using GNU
sed here in order to deal with nul-delimited I/O (option -z), but you could easily replace those two sed commands with e.g. a while read ... shell loop (Bash version 3 or greater would be required) or another language you feel confident with, the only recommendation being that the tool you use is able to handle nul-delimited I/O (e.g. GNU's gawk can do it); see below for a replacement using Bash loops
- I use one single
find here, as I'm not relying on any implied behavior from tar
- The
sed commands manipulate the list of names, paving the way for the sort commands
- specifically, the first
sed moves the "environments" name at the beginning of the path, also prefixing it with a helper 0 number just to make it sort before the non-environments files, as I'm prefixing these latter with a leading 1 for the purpose of sorting
- such preparation normalizes the list of names in the "eyes" of the
sort commands, making all names without the "environments" name and all having the same amount of slash-delimited fields at the beginning, which is important for sort's keys definitions
- the first
sort applies a sorting based first on the files' names, thus putting same names adjacent to each other, and then by numeric value of 0 or 1 as marked previously by the sed command, thus guaranteeing that any "environments" specific file, when present, comes before its non-specific counterpart
- the second
sort coalesces (option -u) on the files' names leaving only the first of duplicate names, which due to the previous reordering is always an "environments" specific file when present
- finally, a second
sed undoes what has been done by the first one, thus reshaping the file names for tar to archive
If you are curious to explore the intermediate pieces of such long pipeline, keep in mind that they all work with nul-delimited names, and hence do not show well on screen. You can pipe any one of the intermediate outputs (i.e. taking away at least the tar) to a courtesy tr '\0' '\n' to show a human-friendly output, just remember that filenames with newlines will span two lines on screen.
Several improvements could be done, certainly by making it a fully parameterized function/script, or for instance by detecting automatically any arbitrary name for "environments" directories, like below:
Important: pay attention to the comments as they may not be well accepted by an interactive shell
(
export r=Products LC_ALL=C
cd -- "$r/.." || exit
# make arguments out of all directories lying at the second level of the hierarchy
set -- "$r"/*/*/
# then expand all such paths found, take their basenames only, uniquify them, and pass them along xargs down to a Bash pipeline the same as above
printf %s\\0 "${@#*/*/}" \
| sort -zu \
| xargs -0I{} sh -c '
e="${1%/}"
echo --- "$e" ---
find -- "$r" -regextype posix-extended \( -path "$r/*/$e/*" -o \( -regex '\''^([^/]+/){2}[^/]+'\'' ! -type d \) -o -regex '\''^[^/]+(/[^/]+)?'\'' \) -print0 \
| sed -zE '\''\%^(([^/]+/){2})([^/]+/)%s%%0/\3\1%;t;s%^%1//%'\'' \
| sort -zt/ -k 3 -k 1,1n \
| sort -zut/ -k 3 \
| sed -zE '\''s%^[01]/(([^/]+/)|/)(([^/]+/?){2})%\3\2%'\'' \
| tar --null --no-recursion -czf "$r-$e.tgz" -T- \
--transform=s'\''%^\(\([^/]\{1,\}/\)\{2\}\)[^/]\{1,\}/%\1%'\''
' packetizer {}
)
Example replacement for the first sed command with a Bash loop:
(IFS=/; while read -ra parts -d $'\0'; do
if [ "${#parts[@]}" -gt 3 ]; then
env="${parts[2]}"; unset parts[2]
printf 0/%s/%s\\0 "$env" "${parts[*]}"
else
printf 1//%s\\0 "${parts[*]}"
fi
done)
For the second sed command:
(IFS=/; while read -ra parts -d $'\0'; do
printf %s "${parts[*]:2:2}" "/${parts[1]:+${parts[1]}/}" "${parts[*]:4}"
printf \\0
done)
Both snippets require the surrounding parentheses in order to be drop-in replacements for their respective sed commands within the pipeline above, and of course the sh -c piece after xargs needs to be turned into bash -c.
