3

Noob here - I want to run grep -r asdf however, I only want unique matches in my directories (i.e. disregarding any directory, display unique matches only).

So I ran grep -r asdf | sort --unique. However - this does not work since the directory names are different (dir1/a.txt asdf and dir2/a.txt asdf).

I didn't see an option (I tried e.g. grep -riol) to exclude directories and I guess that barely makes sense for the scope of the function. Can I somehow cut-away the directories and only show the matched filename + match (possibly without a mind/universe-bending regex/sed/...)?

Improve this question
  • Do I understand correctly: The files dir1/a.txt and dir2/a.txt are the same and you want only a.txt as a result? – pLumo Apr 15 '20 at 6:53
  • I see, you want a.txt:asdf, right? – pLumo Apr 15 '20 at 7:06
  • @pLumo precisely, last comment – user2305193 Apr 15 '20 at 20:59
1

I think with the default capabilities of grep there in no way of doing this.

You could go with something like this, which is just a "small" regex:

grep -r asdf | sed '#^.*/##' | sort --unique

Note: This approach will not work if the search-pattern contains a /

Improve this answer
1

Try this,

grep -r abcd | awk -F ':' '{gsub(/.*\//,"",$1) }1' | awk '!seen[$1]++'
  • gsub will remove directory structure. (/.*\//,"",$1) will remove("") all(.*) before the last match of '/' in first field($1).
  • !seen[$1]++ will uniq the filename.

Note: directory name must not have : in it.

Improve this answer
  • can you briefly explain the awk regex? – user2305193 Apr 15 '20 at 21:08
  • 1
    check the description... – Siva Apr 16 '20 at 5:02
  • This doesn't work properly for me. if the string is in the middle or it contains : it gets buggy somehow, can't make out where exactly this fails – user2305193 Apr 18 '20 at 11:34
1

This groups by basename and grep output:

   ]# grep -ro '#include' include/ |sed -E 's|.*/(.*:)|\1|' |uniq -c |sort|tail -n7
         28 kvm_host.h:#include
         28 mm.h:#include
         29 ib_verbs.h:#include
         31 net_namespace.h:#include
         32 sock.h:#include
         44 fs.h:#include
         48 drmP.h:#include

I used grep -o to get some duplicates. Same time it leaves out the slashes...

If the names contain : the sed will not work correctly. The regex first throws away everything until the last /, then stores everything until a : as \1.

I used -E because of the (subexpression) and | because of the slash.

The subexpression (.*:) is a bit simple (will fail if a grepped line contains a colon). If you leave out the colon, it will fail when the line contains a slash.

Looking at this output I say this is impossible in theory (to parse grep's output in that way):

]# grep -r "" d*
d:/ir:/afile...in file "d"
d:/ir:/afile...in file "ir"

This is identical. I needed a dir with a colon at the end and a file with overlapping name and contents.

]# ls d* 
d

'd:':
ir

grep --color makes the difference!

The include directory is the one from the linux kernel source. One full line in one include-file looks like this.

]# grep -rH '#incl' include/linux/aio.h 
include/linux/aio.h:#include <linux/aio_abi.h>
Improve this answer
  • grep: include/: No such file or directory, sorry if I didn't use the syntax right... I also never saw a ]# before – user2305193 Apr 18 '20 at 11:25
  • I have the same problem: ls dir1/ ls: cannot access 'dir1/': No such file or directory – user373503 Apr 18 '20 at 12:27
  • PS1=']# ' is how I set my geometrical prompt: no letters. The dollar sign looks too much like an "S" to me. I have enough things to read on my screen. – user373503 Apr 18 '20 at 12:43
  • I'm running this on raspbian, is this the issue @kernel? – user2305193 Apr 18 '20 at 15:54
1

With perl -pe 's|.*?([^/]*:)|\1|' it seems to work with colons and slashes in the files (content, not name!). Two lines below have both. 

]# grep -r ':' include |head -n3 | perl -pe 's|.*?([^/]*:)|\1|'
cistpl.h:/* SPDX-License-Identifier: GPL-2.0-only */
ss.h:/* SPDX-License-Identifier: GPL-2.0-only */
ss.h: * are three options:

Is this mind-bending? The idea seems simple:

Keep all non-slash chars in front of the first colon.

The regex |.*?([^/]*:)| has these parts:

  • .*? match anything, but as little as possible, then...
  • ([^/]*:) this subexpression which is...
  • [^/]*: as much non-slash as possible, and then colon.

At least it bends in the right direction.

Improve this answer
1

With output from grep --null, the following GNU awk program should work with any filename:

BEGIN {
    # OFS will be printed between
    # each filename and matched line
        OFS = ":"

    # Use null-byte as a record separator
    # to use with output from grep --null
        RS = "\0"

    # Apart from the first record,
    # everything up to the first newline
    # of a record is the matched line from grep
    # Everything after first newline is a filename
        FPAT = "^[^\n]*|\n.*$"

}
NR == 1 {
    # Entire first record
    # is the first filename
    # set `file` to the basename
        file = gensub(".*/","",1)
        next
}
! seen[file, $1]++ {
    # If filename+match
    # not seen, print it
        print file, $1
}
{
    # Get basename of file
    # from next match
        file = gensub(".*/","",1,$2)
}

grep --null -rF asdf . | awk -f see_above.gawk
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.