find -exec ls -lR {} \; returns every item twice

Question

Background: I'm currently writing an archiving script, which creates gzipped tarballs from some folders and their contents. It should be able to synchronize the gzipped archives with the sources without uncompressing the archives, or compressing the sources. For this, the sought solution is synchronizing the output of ls -l with the output of tar -ztvf.

Both of the commands return similar output, however they are slightly different. Most of those differences can be sorted out with regular expressions, or cut. One thing which I could not resolve easily is listing the files' path relative to the queried directory in maximum depth.

To overcome this problem I used find to find every file, and fed them into ls with the command:

find Webcam -exec ls -lR --time-style="+%Y-%m-%d %H:%M" {} \; | cut -f1,3- -d" " | sed "s/ /\//2" | sed "s/ \+/ /g"

where most of the pipeline serve formatting purpose, find Webcam -exec ls -lR {} \; is the problematic part, and Webcam is the test folder. The output of this command is the following:

-rw-r--r-- debian/debian 162406 2014-04-12 13:42 2014-04-12-134210.jpg
-rw-r--r-- debian/debian 116247 2014-08-09 16:38 2014-08-09-163849.jpg
-rw-r--r-- debian/debian 96597 2015-03-15 19:39 2015-03-15-193905.jpg
-rw-r--r-- debian/debian 100795 2015-04-29 20:23 2015-04-29-202242.jpg
-rw-r--r-- debian/debian 97120 2015-08-02 13:42 2015-08-02-134230.jpg
-rw-r--r-- debian/debian 123835 2015-08-27 23:03 2015-08-27-230306.jpg
-rw-r--r-- debian/debian 97120 2015-08-02 13:42 Webcam/2015-08-02-134230.jpg
-rw-r--r-- debian/debian 100795 2015-04-29 20:23 Webcam/2015-04-29-202242.jpg
-rw-r--r-- debian/debian 116247 2014-08-09 16:38 Webcam/2014-08-09-163849.jpg
-rw-r--r-- debian/debian 96597 2015-03-15 19:39 Webcam/2015-03-15-193905.jpg
-rw-r--r-- debian/debian 162406 2014-04-12 13:42 Webcam/2014-04-12-134210.jpg
-rw-r--r-- debian/debian 123835 2015-08-27 23:03 Webcam/2015-08-27-230306.jpg

Now the output resembles to the output of tar -ztvf:

-rw-r--r-- debian/debian 162406 2014-04-12 13:42 Webcam/2014-04-12-134210.jpg
-rw-r--r-- debian/debian 116247 2014-08-09 16:38 Webcam/2014-08-09-163849.jpg
-rw-r--r-- debian/debian 96597 2015-03-15 19:39 Webcam/2015-03-15-193905.jpg
-rw-r--r-- debian/debian 100795 2015-04-29 20:23 Webcam/2015-04-29-202242.jpg
-rw-r--r-- debian/debian 97120 2015-08-02 13:42 Webcam/2015-08-02-134230.jpg
-rw-r--r-- debian/debian 123835 2015-08-27 23:03 Webcam/2015-08-27-230306.jpg

with the obvious flaw of ls listing every found item twice, one time with the required path, and one time without it. How can I "fix" ls to list every item twice?

Additional insight about the nature of this error (e.g. what's happening under the hood) is more than welcome, while more practical ways to solve the whole archiving problem are also welcome as side notes. However, now I consider this as a challenge, which I would like to solve, so the main focus should be kept on limiting the output of ls.

Answer 1

The problem is that find finds the Webcam directory, too, and runs ls Webcam which lists all the files there. To only list files, not directories, tell find

-type f

Answer 2

Your problem is that ls -lR will be executed for all files (which will display the files) and every directory (which will display the contents of the directory). If your directory-hierarchy would not be flat, but contain sub-directories, this would display the contents even more often, as -R tells ls to traverse subdirectories again. Instead you should call find as follows:

1. find . -type f -exec ls -l {}, which would only display files.
2. find . -exec ls -dl {}, which would display both files and directories.

Answer 3

The root cause of this problem is amazingly short: . (yes: a dot).

Understand that find (without a dir) is equivalent to find .. From man find:

If no paths are given, the current directory is used.

And, when you execute find . the dot appears in the generated list
( Using only four files with distinct names to make it simple ):

$ find                        ### Works the same with or without the dot.
.
./2014-08-09-163849.jpg
./2014-04-12-134210.jpg
./Webcam
./Webcam/2015-04-29-202242.jpg
./Webcam/2015-08-27-230306.jpg

And then you are giving this list to ls -R (recursively).
Just the dot will reproduce all files:

$ ls -1R .         ### The 1 will make the list 1 column.
.:
Webcam
2014-04-12-134210.jpg
2014-08-09-163849.jpg

./Webcam:
2015-04-29-202242.jpg
2015-08-27-230306.jpg

So, it is no wonder that all files appear twice (once for the dot, once from find).

$ find -exec ls -1R {} \;
.:
2014-04-12-134210.jpg
2014-08-09-163849.jpg
Webcam

./Webcam:
2015-04-29-202242.jpg
2015-08-27-230306.jpg
./2014-08-09-163849.jpg
./2014-04-12-134210.jpg
./Webcam:
2015-04-29-202242.jpg
2015-08-27-230306.jpg
./Webcam/2015-04-29-202242.jpg
./Webcam/2015-08-27-230306.jpg

Solutions

Remove the dot (use \( ! -name . -prune \) -a).

Remove the dot from the list of found files from find:

$ find . \( ! -name . -prune \) -a -print
./2014-08-09-163849.jpg
./2014-04-12-134210.jpg
./Webcam

Then you can use a recursive ls -R:

$  find . \( ! -name . -prune \) -a -exec ls -1R {} \;
./2014-08-09-163849.jpg
./2014-04-12-134210.jpg
./Webcam:
2015-04-29-202242.jpg
2015-08-27-230306.jpg

Avoid recursive ls (simpler but demands ls -d).

Not use a recursive ls -R (with also an -d option to avoid expanding into dirs) and let find do the walking of the tree to find all files:

$ find . ! -path . -exec ls -1d {} \;  
./2014-08-09-163849.jpg
./2014-04-12-134210.jpg
./Webcam
./Webcam/2015-04-29-202242.jpg
./Webcam/2015-08-27-230306.jpg    

Archiving

Of course, for the specific use of archiving:
you do not need the directory names, just full path filenames:

$ find . ! -path ./ -type f -a -exec ls -1R {} \;
./2014-08-09-163849.jpg
./2014-04-12-134210.jpg
./Webcam/2015-04-29-202242.jpg
./Webcam/2015-08-27-230306.jpg

---

About find dir (not dot).

If you use a path, the problem is the same if the depth of directories is bigger than one.

This will not trigger a problem:

$ find Webcam -exec ls -1R {} \;
Webcam:
2015-04-29-202242.jpg
2015-08-27-230306.jpg
Webcam/2015-04-29-202242.jpg
Webcam/2015-08-27-230306.jpg

However, increasing the depth by one (cd ..) will:

$ cd ..
$  find jpg -exec ls -1R {} \;
jpg:
2014-04-12-134210.jpg
2014-08-09-163849.jpg
Webcam

jpg/Webcam:
2015-04-29-202242.jpg
2015-08-27-230306.jpg
jpg/2014-08-09-163849.jpg
jpg/2014-04-12-134210.jpg
jpg/Webcam:
2015-04-29-202242.jpg
2015-08-27-230306.jpg
jpg/Webcam/2015-04-29-202242.jpg
jpg/Webcam/2015-08-27-230306.jpg

which this will not trigger:

$  find jpg \( ! -path ./jpg -prune \) -exec ls -1R {} \;
jpg:
2014-04-12-134210.jpg
2014-08-09-163849.jpg
Webcam

jpg/Webcam:
2015-04-29-202242.jpg
2015-08-27-230306.jpg

I insist that the use of two recursive tools is very ill advised.

Change to a simpler structure, like find jpg -ls or even try a shell provided listing solution similar to echo **:

$ (shopt -s globstar nullglob; printf '%s\n' jpg/**/*)
jpg/2014-04-12-134210.jpg
jpg/2014-08-09-163849.jpg
jpg/Webcam
jpg/Webcam/2015-04-29-202242.jpg
jpg/Webcam/2015-08-27-230306.jpg