Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I need paths to be both fully resolved and relative to a given directory. This must be done efficiently, since the number of paths is often greater than 100,000.

Situation: I have directories which contain mostly symlinks to other directories, as in

foo
 123 -> ../baz/123
 896 -> ../bar/896

(Note that foo does NOT contain only symlinks to directories, it also contains ordinary files which I have to catch too.)

These symlinked directories contain files. . I want to get a list of those files, in the form

baz/123/some.file
bar/123/other.file

That is, when 'find' finds a symlink, I want it to dereference that path when it is reporting the contents.

So I am running this command from the parent directory of foo:

find -L foo -type f

But that doesn't work.

Honestly, you would expect the -L option, which claims to 'follow symbolic links', to implement this behaviour. But, its actual behaviour is to look into the contents of those directories, but report files inside them with their non-dereferenced names, ie. the results look like

foo/baz/123/some.file
foo/bar/896/another.file

The results will be used for set operations against a list of filepaths that are all 1. fully resolved and 2. relative to foo's parent directory, so every result must also fulfil that criteria. I can guarantee for these purposes that all links are resolvable ie. none are circular or excessively deep. Most, but not all links, point at directories rather than files.

At the moment, the best I can do is a Python script that rewrites any non-dereferenced paths into resolved ones. But since the numbers of files involved is in the 100000+ range, this is not very practical (and rather ridiculous, since find already had bothered to dereference them, it just didn't return the dereferenced paths). (EDIT: See my comment on this post -- I've found a non-solution (in that it does the job efficiently but in the wrong way -- executing external commands).)

I'm convinced that I should be able to do this task just with find and no external commands, but I'm not finding the man page enlightening here -- none of -L, -H, -P, -follow have the correct behaviour, nor does -printf %l. -exec is out for obvious reasons -- it is not internal to find. Any ideas?

EDIT 2: at this point, Stephane has convinced me that there is no particularly good reason why find would have this functionality internally, so I'm willing to accept any reasonably efficient answer.

share|improve this question
    
Right now I am using find -L foo -type f -exec realpath {} "+" | sed "s,^$REALPATH_OF_TOPLEVEL_DIR,," . This is actually acceptable in speed (3.8s to process 160000 files), but does not match the criteria of the question. I still believe that there is a way to do this basic thing without resorting to -exec. –  kampu Jun 13 '13 at 3:46
    
A variant of the above avoids the non-POSIX -exec ... + form: find -L foo -type f -print0 | xargs -0 realpath --relative-to "$REALPATH_OF_TOPLEVEL_DIR". This is virtually identical to the non-posix find -L foo -type f -exec realpath --relative-to "$REALPATH_OF_TOPLEVEL_DIR" {} "+" except an extra process is spawned. –  kampu Jun 13 '13 at 11:32
1  
find doesn't dereference them, it just follows them, or ommits to ignore them when descending the directory tree if you prefer. –  Stephane Chazelas Jun 13 '13 at 11:40
    
-exec cmd {} + is POSIX -print0 and realpath are not. –  Stephane Chazelas Jun 13 '13 at 11:59
1  
POSIX has -printf '%s\0' {} +. The problem is more than its xargs is unusable (no -0, no -r, and the minimum guaranteed limits are too low to be useful). –  Stephane Chazelas Jun 13 '13 at 14:56
show 2 more comments

1 Answer

up vote 3 down vote accepted

What you're asking for doesn't make much sense in the general case, so it's not surprising that find has no provision for it.

A symlink with a relative target is relative to the path of the symlink. So for instance, if by traversing a directory by following symlinks, find encounters a/b/c/d and a, a/b, a/b/c are all relative or absolute symlinks (or symlinks to paths with symlink components), what should it do?

If you're looking for a find predicate or a GNU -printf % directive that expands to a symlink-free path to the file relative to the current directory or any directory, I'm afraid there's none.

If you're on Linux, you can get the absolute path of those files with:

find -L foo -type f -exec readlink -f {} \;

As you found out, there exists at least one realpath command which accepts more than one path argument which in combination with the standard -exec cmd {} + syntax is going to be a lot more efficient since it's running as few realpath commands as necessary:

find -L foo -type f -exec realpath {} +

find -L foo -type f -print0 | xargs -r0 realpath

might be quicker as if more than one realpath command is needed, find can keep on looking for more files while the first realpath starts working which even on a single processor system might make it more efficient.

-print0 and xargs -r0 are not standard, come from GNU but are found in a number of other implementations like most modern BSDs.

Zsh has builtin support for it:

print -rl foo/***/*(-.:A)

If you don't care about the sorting order, you can disable sorting and make it a bit more efficient with:

print -rl foo/***/*(-.oN:A)

If you want to convert those to relative paths to the current directory, you could have a look at that SO question.

If you know that all those files have an absolute canonical path (whose none of the components are symlinks) inside the current directory, you can simplify it to (still with zsh):

files=(foo/***/*(-.:A))
print -rl -- ${files#$PWD/}

Though short and convenient, and works whatever character filenames contain, I doubt it would faster than find + realpath.

With the Debian realpath and GNU tools, you can do:

cd -P .
find -L foo -type f -exec realpath -z {} + | 
  gawk -v p="$PWD" -v l="${#PWD}" -v RS='\0' -vORS='\0' '
    substr($0, 1, l+1) == p "/" {$0 = substr($0, l+2)}; 1' |
  xargs -r0 whatever you want to do with them

As I realise now, there's now a realpath in recent versions of GNU coreutils, which has the exact feature you're looking for, so it's just a matter of

find -L foo -type f -print0 |
  xargs -r0 realpath -z --relative-base . |
  xargs -r0 whatever you want to do with them

(use --relative-to . instead of --relative-base . if you want relative paths even for files whose symlink free path doesn't reside below the current working directory).

share|improve this answer
    
This is similar to what I had commented I was currently using (find -L foo -type f -exec realpath {} "+" | sed "s,^$REALPATH_OF_TOPLEVEL_DIR,,". I still believe that I must be missing something, because this (choosing between resolved and unresolved output paths) seems like basic enough functionality that it would be included in find itself. BTW, the ";" form of exec, though it's the only POSIX-specified form, is far too slow when you are dealing with 100000+ files. The "+" form is faster but nonstandard. xargs is also a possibility if portability is really needed (it isn't in my case) –  kampu Jun 13 '13 at 11:08
    
@kampu, -exec cmd {} + is POSIX and standard. But readlink accepts only one path argument. readlink comes as standard on most Linux systems since it's part of GNU coreutils. realpath not necessarily. Note that I hadn't seen your comment when replying. –  Stephane Chazelas Jun 13 '13 at 11:52
    
Re: your edit: A) Is the a/b/c/d situation a problem? Find normally (by definition, of how you invoke it) returns paths relative to the current working directory, so any dereferencing of non-circular symlinks is unambiguous (even if said references happen to fall outside of $PWD).; B) I'm fascinated to see that the print command doesn't immediately fail with 'commandline too long' error. Instead it wanders off into the depths of my CPU for 5 minutes and then finally returns; C) -exec readlink -f {} ; is portable but horrendously inefficient (>200x slower than realpath {} +) .. –  kampu Jun 13 '13 at 12:13
    
.. Well, in summary, I appreciate your effort to address this, and I want to accept this answer, I just can't reconcile this with the inefficiency of the proposed solutions. *uncertain face* –  kampu Jun 13 '13 at 12:20
    
Or to put it more positively, I'm happy to accept your answer IFF it contains at least one answer that completes in reasonable time (less than a minute for < 200,000 files, a constraint which my current solution goes well past satisfying.) –  kampu Jun 13 '13 at 12:45
show 4 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.