3

I want to find out what files in a certain directory are not managed by Git. This is so I because I use Git for backups, and I want to eventually get all my files personal in there.

What unix tool could accomplish this? Is there a way to use find to do this in a reasonably efficient way?

Example:

I have a folder where I store RFC-related stuff, it looks like this:

.
├── TheFile
└── tests
    ├── 4180
    │   └── data
    │       ├── bad
    │       └── good
    │           └── linebreaks.csv
    ├── get-rfc.sh
    ├── .git <contents omited>
    ├── LICENSE
    ├── README
    └── rfc4180.txt

I'm looking for a command that will output me:

TheFile
8
  • 1
    Is find . | grep -v '\.git' not efficient enough?
    – pfnuesel
    Nov 15, 2016 at 14:42
  • Welcome to stackexchange. Are you asking to walk the tree, pruning away any parts that have a .git subdirectory? This is an easier problem than you asked - git has a --git-dir option so files can be tracked in repos which are not in the working tree.
    – icarus
    Nov 15, 2016 at 14:50
  • Could you clarify what you want to achieve? As I understand it you want directories where there is no .git subdirectory and no parent directory has a .giteither. Meaning the directory is not under version control and also is not ignored or uncommited. Nov 15, 2016 at 14:58
  • @RaphaelAhrens: You're correct, except that I don't want ignored and uncommited files (I think you meant "all files outside git repos and uncommited changes and ignored file in git repos", but I just want "all files outside git repos").
    – Willem
    Nov 15, 2016 at 15:38
  • 1
    @icarus That's a pretty nifty feature and I'll remember it, thanks for mentioning that, but for the sake of simplicity I'm not going to support it in this case.
    – Willem
    Nov 15, 2016 at 15:58

4 Answers 4

4
find the_starting_dir \( -type d -exec test -d '{}'/.git \; -prune \) -o -print

Not the most portable of find invocations, but works with GNU find.

Find walks the directory tree. The term -prune returns true but stops find from further processing the subtree. So the left hand side of the -o says "if this is a directory, and if test says there is a sub-directory called .git and if prune returns true then we are done processing this thing in the tree". The right hand side says "otherwise print it"

If you don't want directories printed then change -print to \( ! -type d -print \) but then you will not get any indication about empty directories.

You can change the -print to -ls to get listings, -printf see manual etc etc.

5
  • That goes a long way! Two little things, one: this command also returns the root directory (.) along with the files not in git (i.e. ./test), would there be a way to have find not output the .? Second: can find print full paths? I can pipe the result through xargs -n 1 realpath -P but then symlinks in the path are removed, which I want to avoid.
    – Willem
    Nov 15, 2016 at 15:57
  • Change the . to the /full/path to get the absolute pathnames out. Change it to /full/path/* to give visible files/directories, or pipe to tail +1 to drop the first line of the output.
    – icarus
    Nov 15, 2016 at 16:20
  • @Willem, please see Why is looping over find's output bad practice?
    – Wildcard
    Nov 16, 2016 at 1:49
  • With find (GNU findutils) 4.4.2 it gives find: paths must precede expression: print error. Changing print to -print fixes problem. Feb 9, 2017 at 10:11
  • @MateuszKonieczny Fixed
    – icarus
    Feb 9, 2017 at 10:22
4

I don't understand why you don't want to use git...

git status --untracked-files

[edit]

As I understand the request, there is a directory tree. Inside this tree there are multiple git repos, but there is not one covering the whole tree. The example shows a git repo in the tests sub directory. Therefore your suggested command would get a git error fatal: Not in a git repository rather than TheFile as requested.

find . -name .git  -exec echo  $(realpath '{}') \; |\
sed 's/\(.*\)\(.git\)/git --git-dir=\1\2 --work-tree=\1 status --untracked-files/g' | bash  
1
  • 1
    As I understand the request, there is a directory tree. Inside this tree there are multiple git repos, but there is not one covering the whole tree. The example shows a git repo in the tests sub directory. Therefore your suggested command would get a git error fatal: Not in a git repository rather than TheFile as requested. Of course an empty git repo could be created at the top level, in which case It would output the requested file and tests/, which could be filtered out.
    – icarus
    Nov 17, 2016 at 7:46
0

There are a lot of specific cases in your request.

  1. Files actually outside a git-managed directory.
    • Your TheFile fits this case.
  2. Files inside a directory managed by Git, with some .git marker. .git is not always a directory. It can be a file as well, with a path to the real GIT_DIR. We can further break these files down as follows:
    1. Known files, those present in the Git index.
    2. Ignored files, those files matching a pattern per gitignore(5):
      • .gitignore
      • $HOME/.config/git/ignore
      • $GIT_DIR/info/exclude
    3. Files under an actual $GIT_DIR directory, but NOT part of the repo.
      • .git/hooks are the most likely
      • Could also be malware

So the most reliable case, is going to be generating TWO lists, relative to your given base directory $D, and comparing them (be sure to sort them and remove duplicates beforehand).

I can't think of a reliable way to generate the sub-list for 2.3 above, so I leave that as an open problem (I'd love to know about it, because I've lost hooks before).

Shell script to list known files per 2.1 above:

for g in $(find $D -name .git) ; do
  echo $g 
  p=${g%/.git} g2=`readlink -f $g` ;
  ( cd $p && GIT_DIR=$g2 \
  git ls-files --exclude-standard --full-name ) \
  | sed "s,^,${p}/,g" ; 
done > list-2.1

Shell script to list ignored files per 2.2 above:

for g in $(find $D -name .git) ; do
  p=${g%/.git} g2=`readlink -f $g` ;
  ( cd $p && GIT_DIR=$g2 \
  git ls-files \
  --others -i --exclude-standard ) \
  | sed "s,^,${p}/,g" ; 
done > list-2.2

Shell script to list files per 2.3 above:

TODO > list-2.3

Shell script to process the lists and find what's not on side B:

comm -23 <(find $D ! -type d |sort) <(sort 2.1 2.2 2.3 | uniq)
-1
find <root_dir> -type d -name ".git"

is this what you are looking for ?

you could also do something like

find <root_dir> -type d -name ".git" -print0 | xargs -0 -r dirname

to output just the directories names without the /.git part ; you can even avoid dirname and just use the output of find plus /../ at the end of each folder name .

find <root_dir> -type d -name ".git" -print0 | xargs -0 -r printf "%s/../"

EDIT

You can negate the -name parameter and play with the -maxdepth value; for example I can easily list all the directories in my kernel tree that do not have a name that starts with a c ( uppercase or lowercase ) .

find linux-4.8.2/ -type d ! -iname "c*" -maxdepth 1

I still don't really get what you mean with :

I'm looking for the exact opposite of this, namely a command which find all files not in a Git repo;

simply because the files that are in the same directory on the same filesystem of the .git root dir might not be even be part of a repository, it just means that that literally there are files in the same dir ( or subdirs ) where there is a dir named .git too . Remember that git has the concept of staging and in general what you see on a disk may not be your git repository .

Take a look at https://libgit2.github.com/ which is the official C lib for git and there are a lot of bindings for it plus a lot of docs, examples and how to .

4
  • I'm looking for the exact opposite of this, namely a command which find all files not in a Git repo; this (a command which prints all GIt repos) I already have.
    – Willem
    Nov 15, 2016 at 15:28
  • As an aside, find . -type d -name .git -prune -print | xargs -n 1 dirname is a lot faster. (Because the contents of the .git directories are not traversed through the -prune.)
    – Willem
    Nov 15, 2016 at 15:44
  • @Willem I have updated my answer
    – user31223
    Nov 15, 2016 at 16:40
  • I have a bunch of git repos, where I follow the convention of storing the .git metadata folder in the root of the folder containing all the repo's data. I assume my repos don't have files outside the repo folders. I have tooling to check the status of my repos, and I assume all files ignored by git were ignored for good reason by me. Now, I want to find out what folders I forgot to put in a repo, and if you assume what I just did, this can be solved with: find \( -type d -exec "test -d '{}/.git'" -prune \) -o -print. That's what my question is about.
    – Willem
    Nov 19, 2016 at 21:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .