0

I have a bunch of paired files with unneeded barcode tags within the middle of the file, for example:

LIB008983_TRA00020080_TAAGGCGA-TATCCTCT_L001_R1.fastq.gz
LIB008983_TRA00020080_TAAGGCGA-TATCCTCT_L001_R2.fastq.gz
LIB008983_TRA00020081_TAAGGCGA-AGAGTAGA_L001_R1.fastq.gz
LIB008983_TRA00020081_TAAGGCGA-AGAGTAGA_L001_R2.fastq.gz
LIB008983_TRA00020082_TAAGGCGA-GTAAGGAG_L001_R1.fastq.gz
LIB008983_TRA00020082_TAAGGCGA-GTAAGGAG_L001_R2.fastq.gz
LIB008983_TRA00020083_TAAGGCGA-ACTGCATA_L001_R1.fastq.gz
LIB008983_TRA00020083_TAAGGCGA-ACTGCATA_L001_R2.fastq.gz

I need to get rid of the barcode (which varies throughout the files) without modifying identifiers at the beginning or end of the file.

I have tried writing a script myself from what I've read online, but it appears to be a relatively poor attempt:

for f in LIB008983_TRA000{19916..20167}_*_L001_R*.fastq.gz;
do
  newName=${f/_*_\ _L001_R*.fastq.gz}
  mv -i "$f" "$newName";
done

here's the error message I get:

mv: cannot stat ‘LIB008983_TRA00019917_*_L001_R*.fastq.gz’: No such file or directory

Ideally, my final file name would be, for example:

LIB008983_TRA00020136_L001_R1.fastq.gz
LIB008983_TRA00020136_L001_R2.fastq.gz
LIB008983_TRA00020137_L001_R1.fastq.gz
LIB008983_TRA00020137_L001_R2.fastq.gz
..
..

and so on

  • Hi. Welcome to Unix StackExchange. I hope someone can answer your question. However please note that images of text typically aren't the best way to present information. Please see unix.meta.stackexchange.com/questions/4086/…. If you can update your question with text rather than images then it'll help both you and people after you looking for a similar answer. – Stephen Harris Apr 14 at 0:19
  • @StephenHarris thanks I updated the question with text! – ezhang1 Apr 14 at 0:40
  • Better, but not quite :-) There's still one screenshot of the script. You can cat that file and cut'n'paste the results. Thanks! – Stephen Harris Apr 14 at 0:42
  • I THOUGHT THE COLORS WOULD HELP. replaced with text thanks. – ezhang1 Apr 14 at 0:50
  • The advantage of leaving it as text is that people looking to answer the question can cut'n'paste your code onto their machine for testing. I've modified your question slightly for formatting reasons. Thanks for performing the updates! – Stephen Harris Apr 14 at 0:52
0

So the problem you're seeing here is that the for loop is expanding in ways you're not expecting. The {...} range operator is giving a complete list of all possible filenames, not just the ones that exist.

For example, file 19917 doesn't exist, causing that mv error message.

You can see this by putting an echo in the loop:

for f in LIB008983_TRA000{19916..20167}_*_L001_R*.fastq.gz
do
  echo "$f"
done

This gives output like:

LIB008983_TRA00019916_*_L001_R*.fastq.gz
LIB008983_TRA00019917_*_L001_R*.fastq.gz
LIB008983_TRA00019918_*_L001_R*.fastq.gz
...
LIB008983_TRA00020078_*_L001_R*.fastq.gz
LIB008983_TRA00020079_*_L001_R*.fastq.gz
LIB008983_TRA00020080_TAAGGCGA-TATCCTCT_L001_R1.fastq.gz
LIB008983_TRA00020080_TAAGGCGA-TATCCTCT_L001_R2.fastq.gz
...
LIB008983_TRA00020084_*_L001_R*.fastq.gz
LIB008983_TRA00020085_*_L001_R*.fastq.gz
LIB008983_TRA00020086_*_L001_R*.fastq.gz

All those lines with * in them represent files that don't exist.

There's two ways to solve this. Firstly, if you want to keep the range then put a test around the mv:

  if [ -f "$f" ]
  then
    mv -i "$f" "$newName"
  fi

Now the mv command is only run if the file exists.

The second way is if you don't care about the range, and just let the glob pattern match:

for f in LIB008983_TRA000*_*_L001_R*.fastq.gz
do
  newName=${f/_*_\ _L001_R*.fastq.gz}
  mv -i "$f" "$newName"
done

In both cases you'll no longer try to mv files that don't exist.

As a side note; you don't need some of the ; so I removed them from my answer.

You have a second problem, that your "$newName" isn't what you want. I'm an oldschool ksh coder and there may be better bash expressions, but I'd do something like

  tail=L${f##*_L}
  head=${f%_*_$tail}_
  newName="$head$tail"
  mv -i "$f" "$newName"

So now given your input file list, we have

LIB008983_TRA00020080_L001_R1.fastq.gz
LIB008983_TRA00020080_L001_R2.fastq.gz
LIB008983_TRA00020081_L001_R1.fastq.gz
LIB008983_TRA00020081_L001_R2.fastq.gz
LIB008983_TRA00020082_L001_R1.fastq.gz
LIB008983_TRA00020082_L001_R2.fastq.gz
LIB008983_TRA00020083_L001_R1.fastq.gz
LIB008983_TRA00020083_L001_R2.fastq.gz
  • thank you Stephen, I've been playing around with the script according to your suggestions trying to get it to run, but I keep getting a syntax error. here's what I have: 'for f in LIB008983_TRA000{19916..20167}__L001_R.fastq.gz if [-f "$f"] then #mv -i "$f" "$newName" tail=L${f##*L} head=${f%_$tail} newName="$head$tail" #newName=${f/___L001_R*.fastq.gz} mv -i "$f" "$newName";' error message: ./og_rename.sh: line 5: syntax error near unexpected token if' ./og_rename.sh: line 5: if [-f "$f"] ' – ezhang1 Apr 14 at 3:03
  • for f in LIB008983_TRA000{19916..20167}__L001_R.fastq.gz if [-f "$f"] then #mv -i "$f" "$newName" tail=L${f##*L} head=${f%_$tail} newName="$head$tail" #newName=${f/___L001_R*.fastq.gz} mv -i "$f" "$newName"; error message: ./og_rename.sh: line 5: syntax error near unexpected token if' ./og_rename.sh: line 5: if [-f "$f"] ' I guess it has to do w/ if-then? edit: formatting in comments appears to be difficult for me. – ezhang1 Apr 14 at 3:10
  • It looks like you're missing spaces around the [ and ]; it should be if [ -f "$f" ] - those spaces are important. – Stephen Harris Apr 14 at 10:43
  • just for clarification, here's the final script that worked: for f in LIB008982_TRA000**_L002_R*.fastq.gz do tail=L${f##*_L} head=${f%*$tail} newName="$head$tail" mv -i "$f" "$newName" done – ezhang1 Apr 22 at 21:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.