0

I'm trying to count the number of non-symbolic linked files passed by the locate command. I've tried a number of options and think this is the most promising:

locate -r "$PWD.*\.c$" | xargs -0 -I{} test -f {} && echo "regular file" | wc -l

problem is it doesn't work.

There are 30 files in total, 1 is a symbolic link so wc -l should be 29.

I've tried skipping xargs altogether:

locate -r "$PWD.*\.c$" | test -f && echo "regular file" | wc -l

I've tried not symbolic link:

locate -r "$PWD.*\.c$" | test ! -h && echo "regular file" | wc -l
locate -r "$PWD.*\.c$" | test ! -L && echo "regular file" | wc -l

What is the most efficient way to pipe locate output and count number of regular files vs. symbolic links?


Reply to comments

Some people love locate command and some people love find command. I hope I'm agnostic but I like to use locate when possible. Comments were posted and I'd like to reply to them here in the question:

  • updatedb takes 30 seconds to run the first time, but only 4 seconds on subsequent times. Having cron run it every 5 minutes was a knee-jerk reaction to only once-a-day default. But laptop only runs 10 to 20% CPU usage and there is never any lag.
  • After clearing caches find takes 1 minute 9 seconds to find a single file.
  • After clearing caches locate takes 1 second to find the same file.

Here are some benchmarks you can duplicate on your system:

$ sudo -i
# sync; echo 1 > /proc/sys/vm/drop_caches; sync; echo 2 > /proc/sys/vm/drop_caches; sync; echo 3 > /proc/sys/vm/drop_caches; exit
logout

$ time locate .hidden.c | wc -l
1

real    0m0.790s
user    0m0.758s
sys     0m0.028s

$ sudo -i
# sync; echo 1 > /proc/sys/vm/drop_caches; sync; echo 2 > /proc/sys/vm/drop_caches; sync; echo 3 > /proc/sys/vm/drop_caches; exit
logout

$ time find / iname '.hidden.c'  2>/dev/null | wc -l
1888926

real    1m9.044s
user    0m5.158s
sys     0m15.004s

$ sudo -i
# sync; echo 1 > /proc/sys/vm/drop_caches; sync; echo 2 > /proc/sys/vm/drop_caches; sync; echo 3 > /proc/sys/vm/drop_caches; exit
logout

$ time sudo updatedb

real    0m29.323s
user    0m1.267s
sys     0m4.784s

$ time sudo updatedb

real    0m3.592s
user    0m0.479s
sys     0m1.211s

find is unquestionably a lot more robust than locate but locate is many times faster and has an easier to remember syntax.

It's true you have to remember to run sudo updatedb or one-time pass the argument -u to the locate command to update the database to include files created today or exclude files deleted today. But on the other hand with find you have to remember to pass the 2>/dev/null parameter.

I chose to have cron run updatedb every five minutes because my laptop is under-worked in the first place and I'm lazy in the second place.

2

This command came close:

locate -r "$PWD.*\.c$" | xargs -0 -I{} test -f {} && echo "regular file" | wc -l

Problems:

  • You're using nul delimited input with xargs, but locate isn't providing nul delimited output.
  • The && echo is run for the locate | xargs pipe as a whole and not for the individual runs of test

Try:

locate -0r "$PWD.*\.c$" | xargs -0 -I{} sh -c 'test -f "$1" && echo "regular file"' _  {} | wc -l
  • enabling nul delimited output for locate with -0
  • combining the test and echo (this could be improved with a loop over arguments in sh -c so that multiple files are processed per invocation of sh)

There's still the problem of regex operators appearing in $PWD.

| improve this answer | |
  • Deja vu @ regex...Why is it "regular expression" seems so irregular to me :( FTR -f counts a symbolic link as a regular file so the total is 30. I have to use ! -L to get 29 for Zanna's answer: askubuntu.com/questions/1023244/… – WinEunuuchs2Unix Apr 12 '18 at 1:53
  • @WinEunuuchs2Unix, -f counts a symbolic link as a regular file only if the link points to regular file. -f tests for regular file after symlink resolution. It would return false for a symlink to non-regular file (device, fifo, directory...). – Stéphane Chazelas Apr 20 '18 at 8:58
  • @StéphaneChazelas Sorry I read your comment 13 hours too late. The project scope is to exclude symbolic links even if they do point to a valid .c regular file. Test data was provided by the Zanna in the link above. – WinEunuuchs2Unix Apr 20 '18 at 23:43
2

With zsh:

set -o extendedglob # best in ~/.zshrc
c_regular_files=(
  ${(0)^"$(locate -0 "${${PWD%/}//(#m)[]\\*?]/\\$MATCH}/*.c")"}(N.)
)
echo there are at least $#c_regular_files regular files whose name ends in .c
  • We need to escape the [, ?, \ and * in $PWD for locate not to interpret them as wildcard operators (it would be even worse with -r for regexps which have more operators including ., common in file names)
  • $PWD == / has to be treated specially; with $PWD instead of ${PWD%/}, we would be running locate -0 "//*.c" which wouldn't return anything.
  • -0 for files to be NUL-delimited (newline wouldn't work as it's allowed in file paths).
  • . is for regular file. Contrary to [ -f, that excludes symlinks to regular file. If you want every non-symlink .c file (allowing any other type of file like directory, fifo, socket...), replace . with ^@.

In any case note that locate returns a list based on the last time the locate database was updated which may not reflect the current reality.

| improve this answer | |
  • Does that also handle a filename with, say \* in it? Maybe you could provide the canonical answer to the question that inspired this one: askubuntu.com/q/1023244/158442 – muru Apr 20 '18 at 9:26
  • 2
    @muru, with a PWD='/foo*bar', that should do a locate '/foo\\\*bar/*.c' which works for me with mlocate 0.26 and should work with any implementation where backslash is used as an escape operator (I expect all would). There could still be potential problems with file names containing sequences of bytes nor forming valid characters or charsets where the encoding of some characters contain the encoding of wildcard characters. Fixing the locale to C (for both locate and zsh) would address those. – Stéphane Chazelas Apr 20 '18 at 9:49
  • Thank you for this bullet proof answer. As you alluded [ -f wasn't working on test data so I had used [ ! -L instead. As @muru mentioned this question was to address an answer in sister-site Ask Ubuntu: askubuntu.com/a/1023474/307523 – WinEunuuchs2Unix Apr 20 '18 at 10:22
  • Unfortunately this doesn't work in bash. The first line generates error: bash: set: extendedglob: invalid option name. The second line c_regular_files=(${(0)^"$(locate -0 "${${PWD%/}//(#m)[]\\*?]/\\$MATCH}/*.c")"}(N.)) has error: bash: syntax error near unexpected token ('`. – WinEunuuchs2Unix Apr 21 '18 at 0:22
  • @WinEunuuchs2Unix Yes. Like I said, it's for zsh. Zsh is a shell with many more features than bash generally a superset of bash but with a few significant differences as well. – Stéphane Chazelas Apr 21 '18 at 2:06
2

Rather than parsing the output of locate (which is fragile and may miss things that have changed since its database was last updated, or that are not available to all users), use find.

The following will find all .c files in the current directory that are regular files (not symbolic links):

find . -type f -name '*.c'

Given the directory structure

.
|-- file-a.c
|-- file-b.c
|-- file-c.c
|-- file-d.c
|-- link-b.c -> file-b.c
`-- link-d.c -> file-d.c

This would return

./file-a.c
./file-b.c
./file-c.c
./file-d.c

To count them:

find . -type f -name '*.c' | wc -l

or, if you have filenames with newlines in their names,

find .//. -name '*.c' -type f | grep -c //

Doing the same for symbolic links would involve changing the -type f to -type l.

| improve this answer | |
  • I have updatedb running in cron every five minutes. find is faster than locate on a single directory with a few subdirectories. For many drives and partitions though, locate is infinitely faster. I was attempting a locate only answer for this Q&A in Ask Ubuntu. – WinEunuuchs2Unix Apr 12 '18 at 10:08
  • 1
    @WinEunuuchs2Unix You are aware that updating the locate database more or less involves running a find across the whole filesystem, and that doing that every five minutes is a bit silly? I forgot to mention that locate will also miss files that the nobody user can not access (for example in directories that you have protected). – Kusalananda Apr 12 '18 at 10:46
  • I appreciate your comment and have read many similar ones in the past. I updated my question with some benchmarks that you and others can repeat on your own systems. I appreciate the value and popularity of the find command but the purpose of this investigation was to see how the locate command could be used instead of the find command like the other answers to the Ask Ubuntu question were using. I hope you aren't offended... – WinEunuuchs2Unix Apr 13 '18 at 0:04
  • @Kusalananda, that very much depends on the locate implementation, several allow users to list their own files. Some may employ tricks to avoid re-scanning the full file system. – Stéphane Chazelas Apr 20 '18 at 9:15
1

With GNU Parallel it looks like this:

locate -r "$PWD.*\.c$" | parallel 'test -f {} && echo "regular file"' | wc -l

As you can see it is extremely close to your initial attempt.

If you have fewer than 100 hits, you can use that GNU Parallel sets $? to the number of failed jobs up to 100 (THIS DOES NOT SCALE):

ls  *txt | parallel \! test -f {}
echo $?

If you need it even faster:

locate -r "$PWD.*\.c$" |
  perl -ne 'chomp; -l $_ or $s+= -f $_; END{print "$s\n"}'

Or the combination:

locate -r "$PWD.*\.c$" |
  parallel --block 10k --pipe -q perl -ne 'chomp; -l $_ or $s+= -f $_; END{print "$s\n"}' |
  awk '{s+=$1} END {print s}'
| improve this answer | |
  • +1 because it is interesting and has promise but I'm afraid it doesn't work. I installed moreutils. If I run locate .bashrc | wc -l I get 9. If I run locate .bashrc | parallel 'test -f {} && echo "regular file"' | wc -l I get 0 but it should be 9 on my system. – WinEunuuchs2Unix Apr 19 '18 at 22:55
  • You need GNU Parallel, not parallel from moreutils (see 2009 of gnu.org/software/parallel/history.html) – Ole Tange Apr 20 '18 at 7:36
  • When I tried parallel command in Ubuntu 16.04 (Debian based) it recommended the moreutils package. I'll remove it tonight and install the GNU version. The multi-core processing abilities of parallel is enticing because some operations like stat can be very taxing for many files on a single CPU core. – WinEunuuchs2Unix Apr 20 '18 at 10:26
  • I read your link and the history of parallel, moreutils version of parallel (duplicate), xargs, xxargs, xxargs+parallel` and GNU parallel but what was missing is the download link??? I found this link: ftp.gnu.org/gnu/parallel though. Is the file parallel-20120122.tar.bz2 the most recent? Also why not call your program pxargs (parallel xargs) to disambiguate from the other parallel and stress the drop in replacement for xargs angle? – WinEunuuchs2Unix Apr 20 '18 at 22:49
  • Latest version is always ftp.gnu.org/gnu/parallel/parallel-latest.tar.bz2 which currently is 20180322. The naming confusion is less and less these days: moreutils' parallel is being split into a separate package in many distributions (e.g. Ubuntu 17.04) and you can then install both moreutils and GNU Parallel. – Ole Tange Apr 21 '18 at 6:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.