1

I am trying to use grep to find the lines in a file that start with a specific variable I am defining. I know that you need to use double quotes with this command when searching with a variable, but it is just printing an empty line for me. Here is what I've tried.

grep -En "^$i" examplefile.txt

I've also tried seeing if using the regular expression was the problem, but grep -En "$i" examplefile.txt also did not work for me. If you can help, I would greatly appreciate it!

EDIT: Problem identified by @steeldriver, the initial problem was that the text file supplying my $i variable had Windows carriage returns. It runs now for just the first iteration, but then generates empty files for the subsequent lines. Any ideas?

for j in {1..48}
     do
       echo $j
       i=$(cat barcodes.txt | sed -n ${j}p)
       echo $i
       grep -E "^*:$i" subset.fq >> GrepBarcode_$i.txt
     done

Was asked to provide a sample of the file I'm searching and the barcodes I'm using to search. Sorry for not providing that initially! Here is what I'm searching for:

6:AAAGAGAAATGTAATTTATACATACAGTACATATATATATGGCAGCTGTCTCCCCAAATCCTGCTCTACTGCGTCATTGTTGTGGGAATTATTCCTGGGAGGGATGCGTGAAAAATGCAAGGATATGTGCCAAGAGTACTGCAGCACTA
10:AAAGACACTGCAGATAAACCCTGTGTAATAAATACATAAAATATGTTCCAACCATTTTTATAAATTTTCTGAGTAATCTGTGTTGGATTTTCAGAGTAAGCAAATGAGAAATTAGAGTATTTGATTCCCTGTTGCTTATCCAGGACTTT
14:AATTCTATTCTATTCTATTCTATTCTATTCTATTCTATTCTATCCTATCCTATCCTATCCTATCCTATTCTATTCTATCCTATCCTATCCTATCCTATTCCTTTTCTATTCTATTCTATTCTATTCTATTCTATTCTATTCATTTTCTA
18:AAATGCAAAAAGGAAACATGGAAGAGCACTGGATCTCTTACCATTAAACTGCTCAAGTTATTGGTTCGTTTATGTAATAACAAATGACAAAAGTATTACAACCCAGCCATTTATTTATCTATTCCAGTCTACTCCATCTTGATAAATTC
22:TATGGTTTCCGTTGCTGCCATCTCAAAAACATTTGGACTGCTCCGCTTCCTCCTGAGACTGAGCTTTCTCGCCAAATGACGACTTCTACCACATCTATTGACATTATGGGTCTGCAAGCTGCTTATGCTAATTTGCATACTGACCAAGA
26:AAAAACTTGGGTTCCCACCACCCGGCAAGCCTTCAGGAAATCAGCTACAGTGGAGGAGGGATTGGCTGCCACGGGCTGCAAGACTTTCTGACAGTCCGCATTAGCATTTTCCCAAGCTAATTTCCGCACCAATTCAAACTGAGCGTCCT
30:CAATCTTTCCAAGCAACAGCAGGTTTCCGAGATTATGCGCCAAATGCTTACTCAAGCTCAAACGGCTGGTCAGTATTTTACCAATGACCAAATCAAAGAAATGACTCGCAAGGTTAGTGCTGAGGTTGACTTAGTTCATCAGCAAACGC
34:TGATTATTTTGAGTTTGAGCGTATTGAGGCTCTTAAACCTGCTATTGATGCTTGTGGCATTTCTACTCGTTCTCAATTTCCAATTCTTGGCATCCATAAGCTGACGGATAAGCGTATTACGCCGGTTGAATAGGTTCTGTCGCTTCGGA
38:AAGCAACCATACAAATATAACAAATACAAAAGCACACCAAGGCACAACCAAGTCAATGAGACAAAGTTTCGGAAACTTTGTGGTATCACTAGGTTTTCATACAGGATTGATATTTCCCATTACGTTTATCTAATAAATTCAGGAATTTG

What I want to do is search for lines that begin (after the number and colon) with a certain 5-letter code contained in a text file that looks like this:

GCAGA
ACTGA
TATCC

I need it to find the lines that begin with each barcode and print the full line into a new file (which I am calling GrepBarcode_$i.txt with $i being the barcode).

7
  • Can you give an example content of the $i variable ? Does grep "$i" filename output something ?
    – ramius
    Commented Jun 10, 2023 at 7:37
  • Does this helps you ? stackoverflow.com/questions/18147884/…
    – ramius
    Commented Jun 10, 2023 at 7:41
  • Hello Marissa. What about grep -n '^$i' examplefile.txt or awk -vOFS=":" '/^\$i/{print NR,$0}' examplefile.txt? Why would je need -E there? It is a simple BRE matching which grep can handle without without whistles and bells. Commented Jun 10, 2023 at 9:03
  • Are you searching for the value in shell variable $i OR the literal string $i? If the latter, then use single-quotes, not double-quotes (and it's worth noting that if the file(s) you are grepping are shell scripts then it's quite unlikely that a line would start with a literal '$i' - that would only occur if the script were using $i to store the name of a program to run or if it was a continuation of the previous line)
    – cas
    Commented Jun 10, 2023 at 9:55
  • Does examplefile.txt have Windows (CRLF) line endings (check with file examplefile.txt for example) - if so "printing an empty line" may be related to see grep --color=auto breaks when ^M is inside colored match Commented Jun 10, 2023 at 12:36

2 Answers 2

2

Your script above will grep your MKD_nsi_lib1_R1_001.fq file 48 times. If that file is a non-trivial size, your script will be extremely slow.

It also runs cat and sed against barcodes.txt 48 times, which isn't fast but isn't going to be anywhere near as "expensive" (in terms of time and disk I/O) as reading the .fq file 48 times.

Instead of running grep multiple times over the same input file, you would be much better off writing an awk or perl script to do what you need in just one pass (and the larger your barcodes.txt and MKD_nsi_lib1_R1_001.fq files, the better off you will be).

Something like this:

#!/usr/bin/perl

use strict;
# %patterns is a hash where the keys are fixed-text
# strings, and the values are file-handles to 
# files opened for append.
my %patterns;

# First open the barcodes.txt file and read it into
# the %patterns hash
my $barcodes;
open($barcodes,'<','barcodes.txt') || 
  die "Couldn't open 'barcodes.txt' for read: $!\n";

while(<$barcodes>) {
  chomp; # strip the newline at the end of each line
  my $outfile = "GrepBarcode_$_.txt";
  open($patterns{$_}, ">>", $outfile) ||
    die "Couldn't open '$outfile' for append: $!\n";};
close($barcodes);

# Now process the .fq file(s) listed on the command line.
# also works with stdin.
while(<>) {
  # this assumes that the keyword is at the start
  # of the line and is followed by whitespace. This
  # is only a guess on my part, since you didn't describe
  # or provide a sample of your file.  If there's a different
  # delimiter in the input file, adjust the regex in the split
  # function.
  my ($p,undef) = split /\s+/, $_, 2;

  if (defined($patterns{$p})) {
    print { $patterns{$p} } $_;
  };
};

To run this, you'd save it to a file (e.g. split-fq.pl), make it executable with chmod +x split-fq.pl, and run it with the name of the file(s) you want to process (the barcodes.txt file is hard-coded into the script), e.g.

./split-fq.pl MKD_nsi_lib1_R1_001.fq

This is written to use fixed strings because it's much faster than applying 48 different regular expression tests against each line of MKD_nsi_lib1_R1_001.fq. It just extracts the first "word" from each input line and checks to see if it is a key in the %patterns hash - if it is, then write the current line to the associated file handle.

However, it is possible (but slower) to use regular expressions, e.g.

#!/usr/bin/perl

use strict;

# %patterns is a hash where the keys are pre-compiled
# regular expressions anchored to the start of line ^,
# and the values are handles to files opened for append.
my %patterns;

my $barcodes;
open($barcodes,'<','barcodes.txt') ||
    die "Couldn't open 'barcodes.txt' for read: $!\n";

while(<$barcodes>) {
  chomp;
  my $outfile = "GrepBarcode_$_.txt";
  open($patterns{qr/^$_/}, ">>", $outfile) ||
    die "Couldn't open '$outfile' for append: $!\n";
};

close($barcodes);

while(<>) {
  MATCH: foreach my $re (keys %patterns) {
    if (m/$re/) {
      print { $patterns{$re} } $_;
      last MATCH; # no need to test any more patterns against current line
    };
  };
};

This will be slower than the fixed-text version above, but still much faster than running grep 48 times in a shell for loop - it only has to read the .fq file ONCE, not 48 times.

NOTE: these are just examples of HOW you might do something like this. I have no idea whether they will work correctly with your data because I don't know what's in your files - you didn't provide samples of either barcodes.txt or the .fq file. You will almost certainly have to modify the scripts to suit your actual data.


Also note that it's entirely possible that much better tools for splitting fastq files already exist. In fact, there's a huge library of scripts and tools written in perl for bioinformatics at https://bioperl.org/

If you prefer python, see https://biopython.org/

And, of course, there's a stack exchange site especially for bioinformatics questions at https://bioinformatics.stackexchange.com/


The following version should work with the sample data you provided.

It works similarly to the first fixed-string version (and should be about as fast), but it splits each input line of the .fq file into two fields (variables $num and $data) using a colon (:) as the field separator.

Then it uses perl's substr() function to extract the first 5 letters of $data into another variable called $start.

If there is a key with the value of $start in the %patterns array, it writes the current line ($_) to the associated output file (the file-handle in $patterns{$start}).

#!/usr/bin/perl

use strict;
my %patterns;
my $barcodes;

open($barcodes,'<','barcodes.txt') || 
  die "couldn't open 'barcodes.txt' for read: $!\n";
while(<$barcodes>) {
  chomp;
  my $outfile = "GrepBarcode_$_.txt";
  open($patterns{$_},">>","$outfile") ||
    die "couldn't open '$outfile' for append: $!\n";
};
close($barcodes);

while(<>) {
  my ($num,$data) = split /:/, $_, 2;
  my $start = substr($data,0,5);

  if (defined($patterns{$start})) {
    print { $patterns{$start} } $_;
  };
};

When I ran it to test it, it produced only empty GrepBarcode_?????.txt output files - that's because none of the lines in your examplefile.txt match any of your 5-letter codes. I added AAAGA to barcodes.txt and it produced the file GrepBarcode_AAAGA.txt with the following contents:

$ cat GrepBarcode_AAAGA.txt 
6:AAAGAGAAATGTAATTTATACATACAGTACATATATATATGGCAGCTGTCTCCCCAAATCCTGCTCTACTGCGTCATTGTTGTGGGAATTATTCCTGGGAGGGATGCGTGAAAAATGCAAGGATATGTGCCAAGAGTACTGCAGCACTA
10:AAAGACACTGCAGATAAACCCTGTGTAATAAATACATAAAATATGTTCCAACCATTTTTATAAATTTTCTGAGTAATCTGTGTTGGATTTTCAGAGTAAGCAAATGAGAAATTAGAGTATTTGATTCCCTGTTGCTTATCCAGGACTTT
0
1

If the barcodes are always 5 characters large, you can do something like:

awk -F: '! x {barcode[$0]; next}
         {key = substr($2, 1, 5)}
         key in barcode {print >> ("GrepBarcode_"key".txt")}
        ' barcodes.txt x=1 examplefile.txt

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .