1

I have a large-size text file.(about 2GB)

I'll call you A.txt

I want to know if the A.txt file contains the sentence I want(The size of the sentence is 1Byte~500MB)

Also, if the sentence is included in A.txt, I would like to know the location included in byte

5
  • 1
    You seem to know grep, you probably know options -b and -o for GNU grep, so what exactly is your question?
    – Philippos
    Commented Aug 29, 2019 at 5:28
  • @Philippos I know grep can't compare multiple lines. The search keyword I want is not a line. I want to search for a number of lines of 500MB size
    – puregyu
    Commented Aug 29, 2019 at 5:40
  • Ah. Maybe better edit to say "multi-line sentence". Where is that "sentence" stored -- in a file? You probably don't want a 500 MB pattern in your command line (-;
    – Philippos
    Commented Aug 29, 2019 at 5:57
  • @Philippos Actually, that's a problem. I am worried about making a 500 MB size pattern into a file without putting it in the command line. And that would eventually compare files to files.
    – puregyu
    Commented Aug 29, 2019 at 6:06
  • perhaps a better solution would be by using perl?
    – ahron
    Commented Aug 29, 2019 at 9:04

1 Answer 1

1
#!/usr/bin/perl -w

use strict;

my $re=shift;
$re =~ s/[[:blank:]]+/\\s+/g;#/this comment only exists to fix SEs perl highlighting

my $blocksize = 100000;
my $filesize;

foreach my $file (@ARGV) {
  open(F,"<",$file) || die "couldn't open '$file': $!\n";

  $filesize=(stat($file))[7];

  my @matches=();
  for (my $i = 0; $i <= $filesize; $i+=$blocksize-(length($re))) {
    seek(F,$i,0);
    read F,$_,$blocksize;
    @matches=(m/$re/gmo);
    print join("\n",@matches), "\n" if (@matches);
    };
  close(F);
};

Save this as,e.g., puregyu.pl, make it executable with chmod +x puregyu.pl and run it as:

./puregyu.pl 'my sentence' a.txt

This script takes the first argument (the search string or pattern) and changes any blank spaces ([[:blank:]], i.e. spaces and tabs) to \s* (any whitespace, including spaces, tabs, linefeeds and formfeeds) to allow sentences to match even if they cross lines. See man perlrecharclass for details on how whitespace is defined.

Any remaining arguments are treated as filenames to search.

In order to avoid loading the entire input file into memory at once, this script scans a "block" of input at a time. To avoid missing any input, it increments the offset variable ($i) used by the read statement by ($blocksize - length($re) instead of just $blocksize).

I've set the blocksize to 1000000, so it will read 100KiB at a time - I don't have a suitable large text file to test with, so you might have to tweak this for optimal performance.

NOTE: Getting the byte location of the matches is more work than I have time for right now. Perhaps someone else could help with that.


To match an enormous multi-line "sentence", the following would work better. It allows you to specify a filename instead of a string as the first argument. then it slurps in that entire file and turns it into a regular expression.

#!/usr/bin/perl -w

use strict;

use File::Slurp;

my $re = read_file(shift,chomp=>1);
$re=~s/[[:blank:]]+/\\s+/gm;#/this comment only exists to fix SEs perl highlighting

my $blocksize = length($re)*5;
my $filesize;

foreach my $file (@ARGV) {
  open(F,"<",$file) || die "couldn't open '$file': $!\n";

  $filesize=(stat($file))[7];

  my @matches=();
  for (my $i = 0; $i <= $filesize; $i+=$blocksize-(length($re))) {
    seek(F,$i,0);
    read F,$_,$blocksize;
    @matches=(m/$re/gmo);
    print join("\n",@matches), "\n" if (@matches);
    };
  close(F);
};

Save this as,e.g., puregyu2.pl, make it executable with chmod +x puregyu2.pl and run it as:

./puregyu2.pl searchfile.txt a.txt

Finally, neither of these scripts are fully-tested. I don't have either an enormous text file to search or an enormous "sentence" to search for. Accordingly, these are offered as approximate pointers in one possible right direction rather than as complete, working code.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .