0

I've found myself in the horrible position of needing to suck file system data into a database for analysis. One of the ways I was using to extract this data was using the following GNU find command line.

find . -printf '__:__%M__:__%u__:__%g__:__%s__:__%Cs__:__%p\n'

This uses __:__ as the separators which in theory will never appear in an actual file or directory name.

The problem is that I now need to also incorporate cksum because I need a quick checksum of the files.

What I want to do is to do something like:

find . -exec cksum {} \; -printf '__:__%M__:__%u__:__%g__:__%s__:__%Cs__:__%p\n'

Which keeps it all as a one liner and a single pass over the filesystem. BUT this outputs the cksum on a separate line.

Is there some way of combining the -exec cksum so it appears as a value I can use in the printf?

Is find the best tool to do this? Should I be using another tool instead?

Thank you!

0

1 Answer 1

3

First, some notes:

  • There's nothing stopping __:__ or newline from appearing in a file path, __:__ cannot occur in any of the fields you print before the path.

    Try mkdir -p $'__:__/\n\n\n' if you want to check.

    Any byte value other than 0 can occur in a file path. Those bytes don't even have to make up characters, so generally you can't treat file paths as text, let alone single lines of text. Usually we use NUL-delimited records to represent lists of file paths reliably.

  • %u and %g give you a user/group name corresponding to the uid/gid of the file. There can be several user names for one user id and the user name for uid 123 may be something today and something else tomorrow. In other words, what you get is not intrinsic to the file but also embeds information from the system's user database.

  • The order of the files reported by find is non-deterministic. If the aim is to be able to detect when something in the directory hierarchy has changed, you'll want to sort that list.

  • beware %Cs only gives you precision to the second. Use %C@ for full precision.

Here, you could do (with zsh or bash)

find . -printf '%M/%U/%G/%s/%C@/%p\0' | LC_ALL=C sort -z |
  while IFS=/ read -rd '' mode uid gid size ctime file; do
    cksum=$(cksum < "$file") || continue
    # do what to have to do with $mode $uid $gid $size $ctime $cksum $file
  done

You may also want to pick a more reliable checksum algorithm than that of cksum.

Note that with bash, you can't pick any other separator than / above. If you pick : for instance and there's a file called ./dir/file: (with a trailing :), bash's read splits mode:uid:gid:size:ctime:./dir/file: into mode, uid, gid, size, ctime, ./dir/file and the trailing : is lost. That's a POSIX requirement (ignored by zsh). / is guaranteed not to occur at the end of file paths output by find.

Also note that the -printf predicate is specific to the GNU implementation of find and is not portable. -z and the ability to cope with non-text data is also a GNU sort extension.

2
  • Agree, the use of : as a separator is lazy but it works for me in this instance. cksum is quick and dirty, given the number of files I'm dealing with, using something more robust is going to extend the time required to process all the files. Thanks for your answer, it got me to where I needed to be, much appreciated!
    – davx8342
    Feb 25, 2023 at 12:35
  • @davx8342, I've added a note about why / is used as the separator in my code. Feb 25, 2023 at 12:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .