Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a bunch of files and I want to find which one contains sequential lines starting with a certain string.

For example for the following file :

Aaaaaaaaaaaa
Baaaaaaaaaaa
Cxxxxxxxxx
Cyyyyyyyyy
Czzzzzzzzz
Abbbbbbbbbbb
Bbbbbbbbbbbb
Caaaaaa
Accccccccccc
Bccccccccccc
Cdddddd
Ceeeeee

There is more than one line starting with 'C', so I want this file to be found by command.
For example for the following file :

Aaaaaaaaaaaa
Baaaaaaaaaaa
Cxxxxxxxxx
Abbbbbbbbbbb
Bbbbbbbbbbbb
Caaaaaa
Accccccccccc
Bccccccccccc
Cdddddd

There is always one line starting with 'C', I don't want this file. I thought of using a grep or a sed but I don't know exactly how to do it. Maybe using a regexp ^C.*$^C or something like that. Any idea ?

share|improve this question
    
There're two lines starting with C in your second example. –  Gnouc Mar 25 at 13:40
4  
This question is unclear. Are you looking for files that have more than one consecutive line starting with C? –  Graeme Mar 25 at 13:40
    
Yes this is what I want. Sorry for the misunderstanding. –  Jérémie Mar 25 at 14:03
2  
@terdon, it looks like multi-line searches with -P worked until 2.5.4 and not anymore after that, though I can't find anything in the changelog that would explain why. –  Stéphane Chazelas Mar 25 at 16:33
1  
@Graeme you might want to undelete your answer, see Stephane's comment, apparently it does work for some older grep versions. –  terdon Mar 25 at 16:37

6 Answers 6

up vote 5 down vote accepted

With pcregrep:

pcregrep -rMl '^C.*\nC' .

POSIXly:

find . -type f -exec awk '
  FNR==1 {last=0; printed=0; next}
  printed {next}
  /^C/ {if (last) {print FILENAME; printed=1; nextfile} else last=1; next}
  {last=0}' {} +

(though that means reading all the files fully with those awk implementations that don't support nextfile).


With versions of GNU grep up to 2.5.4:

grep -rlP '^C.*\nC' .

appears to work, but it's by accident and it is not guaranteed to work.

Before it was fixed in 2.6 (by this commit), GNU grep had overlooked that the pcre searching function it was using would match on the whole buffer currently processed by grep, causing all sorts of surprising behavior. For instance:

grep -P 'a\s*b'

would match on a file containing:

bla
bla

This would match:

printf '1\n2\n' | grep -P '1\n2'

But this:

(printf '1\n'; sleep 1; printf '2\n') | grep -P '1\n2'

Or:

(yes | head -c 32766; printf '1\n2\n') > file; grep -P '1\n2' file

would not (as the 1\n2\n is across two buffers processed by grep).

That behaviour ended up being documented though:

15- How can I match across lines?

Standard grep cannot do this, as it is fundamentally line-based. Therefore, merely using the '[:space:]' character class does not match newlines in the way you might expect. However, if your grep is compiled with Perl patterns enabled, the Perl 's' modifier (which makes '.' match newlines) can be used:

     printf 'foo\nbar\n' | grep -P '(?s)foo.*?bar'

After it was fixed in 2.6, the documentation was not amended (I once reported it there).

share|improve this answer
    
Is there any reason not to use exit and -exec \; instead of nextfile? –  terdon Mar 25 at 14:48
    
@terdon, that would mean running one awk per file. You'd want to do that only if your awk doesn't support nextfile and you've got a large proportion of files that are large and have matching lines towards the beginning of the file. –  Stéphane Chazelas Mar 25 at 14:51
    
How about this grep technique ( I guess with more recent versions of GNU grep) that facilitates multiline matches by make the whole file look like a single string by setting line terminator to NUL - would you be aware if there are any limitations to it? –  1_CR Mar 27 at 12:28
1  
@1_CR, That would load the whole file in memory if there's no NUL character in there and that assumes lines don't contain NUL characters. Also note that older versions of GNU grep (which the OP has) can't use -z with -P. There's no \N without -P, you'd need to write it $'[\01-\011\013-\0377]' which would only work in C locales (see thread.gmane.org/gmane.comp.gnu.grep.bugs/5187) –  Stéphane Chazelas Mar 27 at 13:52
    
@StephaneChazelas, very useful detail, thanks –  1_CR Mar 27 at 15:18

With awk:

awk '{if (p ~ /^C/ && $1 ~ /^C/) print; p=$1}' afile.txt

This will print contents of the file if there are consecutive lines starting with a C. The expression (p ~ /^C/ && $1 ~ /^C/) will look into successive lines in the file and will evaluate to true if the first character in both match C. If that is the case, the line will be printed.

In order to find all the files that have such a pattern, you can run the above awk through a find command:

find /your/path -type f -exec awk '{if (p ~ /^C/ && $1 ~ /^C/) {print FILENAME; exit;} p=$1}' {} \;

In this command, the find + exec will go through each of the files and perform similar awk filtering on each file and print its name via FILENAME if the awk expression evaluated to true. In order to avoid printing FILENAME multiple times for a single file with multiple matches the exit statement is used (thanks @terdon).

share|improve this answer
    
My question was not clear enough, I want to know the name of the files with more than one consecutive line starting with C –  Jérémie Mar 25 at 14:09
    
@Jérémie I updated my answer. –  Ketan Mar 25 at 14:31
    
Could you please add an explanation of how this works? Also, there's no need for flag, just exit instead. That way, you don't need to keep processing files after a match has been found. –  terdon Mar 25 at 14:41
    
@terdon Thanks, I updated my answer. –  Ketan Mar 25 at 14:47

Yet another option with GNU sed:

For a single file:

sed -n -- '/^C/{n;/^C/q 1}' "$file" || printf '%s\n' "$file"

(though it will also report the files it cannot read).

For find:

find . -type f ! -exec sed -n '/^C/{n;/^C/q 1}' {} \; -print

The problem with unreadable files being printed can be avoided by writing it:

find . -type f -size +2c -exec sed -n '$q1;/^C/{n;/^C/q}' {} \; -print
share|improve this answer
    
Can you please detail the sed -n '$q1;/^C/{n;/^C/q}' ? –  Jérémie Mar 26 at 15:13
    
Anyone to explain me ? –  Jérémie Mar 28 at 7:47
    
@Jérémie $q1 -- forces sed to quit with an error if pattern is not found. It also will finish with error if something is wrong with file (it's unreadable or broken). So it will quit with 0 exit status only in case pattern is found and it will be passed to print. Part with /^C/{n;/^C/q is pretty simple. If it find string that starts with C it will read the next line and if it also starts with C it will quit with zero exit status. –  rush Mar 28 at 10:28

Assuming your files are small enough to be read into memory:

perl -000ne 'print "$ARGV\n" if /^C[^\n]*\nC/sm' *

Explanation:

  • -000 : set \n\n as the record separator, this turns on paragraph mode which will treat paragraphs (separated by consecutive newlines) as single lines.
  • -ne : apply the script given as an argument to -e to each line of the input file(s).
  • $ARGV : is the file currently being processed
  • /^C[^\n]*\nC/ : match C at the beginning of a line (see the description of the sm modifiers below for why this works here) followed by 0 or more non-newline characters, a newline and then another C. In other words, find consecutive lines starting with C. *//sm : these match modifiers are (as documented [here]):

    • m : Treat string as multiple lines. That is, change "^" and "$" from matching the start or end of line only at the left and right ends of the string to matching them anywhere within the string.

    • s: Treat string as single line. That is, change "." to match any character whatsoever, even a newline, which normally it would not match.

You could also do something ugly like:

for f in *; do perl -pe 's/\n/%%/' "$f" | grep -q 'C[^%]*%%C' && echo "$f"; done

Here, the perl code replaces newlines with %% so, assuming you have no %% in your input file (big if of course), the grep will match consecutive lines starting with C.

share|improve this answer

SOLUTION:

( set -- *files ; for f ; do (
set -- $(printf %c\  `cat <$f`)
while [ $# -ge 1 ] ;do [ -z "${1#"$2"}" ] && {
    echo "$f"; break ; } || shift
done ) ; done )

DEMO:

First, we'll create a test base:

abc="a b c d e f g h i j k l m n o p q r s t u v w x y z" 
for l in $abc ; do { i=$((i+1)) h= c= ;
    [ $((i%3)) -eq 0 ] && c="$l" h="${abc%"$l"*}"
    line="$(printf '%s ' $h $c ${abc#"$h"})"
    printf "%s$(printf %s $line)\n" $line >|/tmp/file${i}
} ; done

The above creates 26 files in /tmp named file1-26. In each file there are 27 or 28 lines beginning with the letters a-z and followed by the rest of the alphabet. Every 3rd file contains two consecutive lines in which the first character is duplicated.

SAMPLE:

cat /tmp/file12
...
aabcdefghijkllmnopqrstuvwxyz
babcdefghijkllmnopqrstuvwxyz
cabcdefghijkllmnopqrstuvwxyz
...
kabcdefghijkllmnopqrstuvwxyz
labcdefghijkllmnopqrstuvwxyz
labcdefghijkllmnopqrstuvwxyz
mabcdefghijkllmnopqrstuvwxyz
...

And when I change:

set -- *files

to:

set -- /tmp/file[0-9]*

I get...

OUTPUT:

/tmp/file12
/tmp/file15
/tmp/file18
/tmp/file21
/tmp/file24
/tmp/file3
/tmp/file6
/tmp/file9

So, in brief, the solution works like this:

sets subshell positionals to all of your files, and for each

sets a nested subshell's positionals to the first letter of each line in each file as it loops.

[ tests ] if $1 negates $2 indicating a match, and if so

echoes the filename then breaks the current loop iteration

else shifts to the next single character positional to try again

share|improve this answer

This script uses grep and cut to obtain line numbers of matching lines, and checks for any two consecutive numbers. The file is assumed a valid filename passed as the first argument to the script:

#!/bin/bash

checkfile () {
 echo checking $1
 grep -n -E "^C.*$" $1 | cut -d: -f1 | while read linenum
     do
        : $[ ++PRV ] 
        if [ $linenum == $PRV ]; then return 1; fi
        PRV=$linenum
     done
     return 0
}

PRV="-1"
checkfile $1
if [ $? == 0 ]; then
   echo Consecutive matching lines found in file $1
fi
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.