2

I want to generate symbolic links from multiple folders while maintaining the directory structure. I read a few answers and the best thing I can come up with now is:

find /data/DIV5/SASC/e042_ctcl/input/*/Clean_data/*/*/*.fq.gz -type f -exec ln -sf \{\} . \;

This generates symlinks all in the same directory. But what I want is to have the same directory structure as in the find command and if possible using the same structure as the input. For example: input/*/Clean_data/*/*/*.fq.gz

When files and directories are matched the symlink should follow the same structure as the absolute link found with the find command.

Any input is appreciated.

  • As a side note, cp -rs could have been used if you had to link all the files, not just a couple of them (*.fq.gz). – Cristian Ciupitu Jun 10 '14 at 9:08
2

I whould generate the commands with awk and then pipe them into bash:

find /data/DIV5/SASC/e042_ctcl/input/*/Clean_data/*/*/*.fq.gz -type f | \
awk '{printf "mkdir -p `dirname %s`\nln -s %s ./%s \n", substr($0, 27), $0, substr($0, 27)}'

This generates a list of the commands, I would first control them and then finally pipe it to bash. The complete command would be:

find /data/DIV5/SASC/e042_ctcl/input/*/Clean_data/*/*/*.fq.gz -type f | \
awk '{printf "mkdir -p `dirname %s`\nln -s %s ./%s \n", substr($0, 27), $0, substr($0, 27)}' | bash
  • I think there's a tiny bug. I did a test and 'm getting mkdir -p `dirname /usr/share/doc/xml-commons-apis/LICENSE.sac.html` and ln -s /usr/share/doc/xml-commons-apis/LICENSE.sac.html ./-apis/LICENSE.sac.html. – Cristian Ciupitu Jun 10 '14 at 9:38
  • When you use the command as above? You have search results in /usr/share/doc/ that cannot be when searching in /data/DIV5/SASC/.../.../. Please post the output of find /data/DIV5/SASC/e042_ctcl/input/*/Clean_data/*/*/*.fq.gz -type f – chaos Jun 10 '14 at 9:47
  • So that 27 is a hard-coded value for the particular directory mentioned in the question? – Cristian Ciupitu Jun 10 '14 at 9:49
  • @CristianCiupitu yes, because he wanted the directory structure from input on, that's the 26th character – chaos Jun 10 '14 at 9:51
  • Then a +1 from me. – Cristian Ciupitu Jun 10 '14 at 10:00
4

Ugly and fragile, but it should get the job done:

find /data/DIV5/SASC/e042_ctcl/input/*/Clean_data/*/*/*.fq.gz -type f -print0 | \
  xargs -0 -n 1 \
    sh -c 'mkdir -p "$PWD/`dirname $0`"; ln -s "$0" "$PWD/`dirname $0`"'

Inspired by chaos' answer, I came up with this alternative which also trims unneeded directories:

find /data/DIV5/SASC/e042_ctcl/input/*/Clean_data/*/*/*.fq.gz -type f   \
     -printf 'mkdir -p "${PWD}/%h"; ln -s "%p" \\\n\t"${PWD}/%h"\n'   | \
sed 's#/data/DIV5/SASC/e042_ctcl##'                                   | \
sh -

It generates the required mkdir and ln commands:

mkdir -p "${PWD}//data/DIV5/SASC/e042_ctcl/input/x/Clean_data/0/a"; ln -s "/data/DIV5/SASC/e042_ctcl/input/x/Clean_data/0/a/foobar.fq.gz" \
    "${PWD}//data/DIV5/SASC/e042_ctcl/input/x/Clean_data/0/a"

Then it removes the unneed directories from them with sed, resulting in:

mkdir -p "${PWD}//input/x/Clean_data/0/a"; ln -s "/data/DIV5/SASC/e042_ctcl/input/x/Clean_data/0/a/foobar.fq.gz" \
    "${PWD}//input/x/Clean_data/0/a"

The ln command starts on the same line as mkdir and continues on the next line in order to keep the source intact while trimming the destination. Printing them on the same line or each on its own distinct line would have required a more complicated sed script.

Explaination from the documentation for find, of the -printf parameters:

%p File's name (not the absolute path name, but the name of the file as it was encountered by find - that is, as a relative path from one of the starting points).

%h Leading directories of file's name (all but the last element and the slash before it). If the file's name contains no slashes (for example because it was named on the command line and is in the current working directory), then “%h” expands to “.”. This prevents “%h/%f” expanding to “/foo”, which would be surprising and probably not desirable.

  • Woo thanks for the explanation will dive into it! was busy doing some other work till now, so sorry for late reply – Sander Van der Zeeuw Jun 10 '14 at 14:55
  • Note that the commands based on … | sh break if files contain some special characters. – Gilles 'SO- stop being evil' Jun 10 '14 at 22:53
  • It would certainly fail for double quotes (") because I used them for quoting, but it should work with multiple spaces or glob characters like ? or *. – Cristian Ciupitu Jun 11 '14 at 0:58
2

I am with the impression that you are looking for a tool like GNU Stow.

GNU Stow is a symlink farm manager which takes distinct packages of software and/or data located in separate directories on the filesystem, and makes them appear to be installed in the same place. For example, /usr/local/bin could contain symlinks to files within /usr/local/stow/emacs/bin, /usr/local/stow/perl/bin etc., and likewise recursively for any other subdirectories such as .../share, .../man, and so on.

Check it out, if it matches the use-case.

1

Zsh has a convenient function zmv. First, load it (you can do this from your .zshrc, or on the command line for a one-session use):

autoload -U zmv
alias zcp='zmv -C'
alias zln='zmv -L'

To recreate a forest of symbolic links is as easy as this:

 zln -s '/data/DIV5/SASC/e042_ctcl/(input/*/Clean_data/*/*/*.fq.gz)' '$1'

If you want to traverse Clean_data directories recursively instead of going only exactly two levels deep:

 zln -s '/data/DIV5/SASC/e042_ctcl/(input/*/Clean_data/**/*.fq.gz)' '$1'

There is a big limitation: this won't create the necessary subdirectories. You can do that by defining and using a wrapper function for ln that creates directories as needed.

ln_s_mkdir () {
  mkdir -p -- ${(P)#}
  ln -s "$@"
}
 zmv -p ln_s_mkdir '/data/DIV5/SASC/e042_ctcl/(input/*/Clean_data/**/*.fq.gz)' '$1'

You can make the command run a bit faster by using zsh's internal mkdir and ln commands (which are not loaded by default because they have fewer options that the GNU utilities found on most systems).

zmodload -F zsh/files b:zf_ln b:zf_mkdir
ln_s_mkdir () {
  zf_mkdir -p -- ${(P)#}
  zf_ln -s "$@"
}
 zmv -p ln_s_mkdir '/data/DIV5/SASC/e042_ctcl/(input/*/Clean_data/**/*.fq.gz)' '$1'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.