1

I have a series of files in a common bioinformatic format (FASTA) that consist of a string of: 1 special character ">", text as the sample name (length of the string varies, can include numbers or letters), and extra digits added by the bioinformatic pipeline. I am on Mac OSX using Bash.

I want to find and replace a text string using a special pattern to eliminate all digits after the sample name (where here the sample is "Tsuc2"). The best way I have found to do this so far is manually in TextEdit using a special pattern for "digit". This works, but ideally I would like to come up with a for loop for all files in a folder instead of opening them and doing FIND-REPLACE one by one. I imagine that it would use grep or sed. The problem that I have been having is that the number of extra digits varies, and there are not delimiters between the sample name and the digits that need to be removed.

My best guess would be something like

$ for FILE in *_R2.fastq.gz.fa; do sed -i 's\[b$FILE]\[0-9]\b/$FILE/g'; done

This doesn't work (probably for obvious reasons). Any ideas?

Edit to address comments: Yes, this is a fasta file (basically the same as .txt, but unique in that the format always follows ">" + "sample name" + "nucleotide sequence." It is not compressed. It was converted from a compressed fastq file to contain only the trimmed sequence without the base call scores.

I cannot rephrase the questions as "remove all digits from all lines" because the sample name contains digits ("Tsuc2"). I am able to remove the extra digits from a single file easily, my question is "how can I find and replace a string for all files in a folder when the number of characters and digits to be retained varies between files, and there are no delimiters between digits in the sample name and digits superficially added by the pipeline."

I had provided the file layout as an image because the text editor here interprets ">" as an indent, but I can paste it inside brackets below.

>Tsuc21
TTAAGTTCAGCGGGTATTCCTACCTGATCCGAGGTCAACCATTAAAATTGGGGGGTTTAGCGGCTGAGAGCTGCAACACC
TAACCAAAGCGAGAAAAAAATTACTACGCTAAGAGGATATTACAGATCCGCCGTTGTATTTCAGGAACTACAATACAGTA
>Tsuc22
TTAAGTTCAGCGGGTATTCCTACCTGATCCGAGGTCAACCATTAAAATTGGGGGGTTTAGCGGCTGAGAGCTGCAACACC
TAACCAAAGCGAGAAAAAAATTACTACGCTCAGAGGATATTACAGATCCGCCGTTGTATTTCAGGAACTACAATACAGTA
GATTCCCAACACTAAGCTAGGCTTAAGGGT
>Tsuc23
TTAAGTTCAGCGGGTATTCCTACCTGATCCGAGGTCAACCTTGTAAAAATTACAGTGGTGTTTAACGGCCGAGAACCGCC
GAGTCTCCCAAGCGAGAAAGTAAACTACTACGCTCGGAGCCACGGAGGGCCCTCCACTGATCTTGGGGGCCTGCGAGAGT
TTTTAATTAACGCTCGCAGAGGGCCCCAAC
>Tsuc24
TTAAGTTCAGCGGGTATTCCTACCTGATCCGAGGTCAACCATTAAAATTGGGGGGTTTAGCGGCTGAGAGCTGCAACACC
TAACCAAAGCGAGAAAAAAATTACTACGCTCAGAGGATATTACAGATCCGCCGTTGTATTTCAGGAACTACAATACAGTA
GATTCCCAACACTAAGCTAGGCTTAAGGGT

Image of a FASTA file containing multiple lines of data, where each line add an additional number infinitely (sometimes a single digit, up to tens of thousands of digits)

4
  • 4
  • This will not work because you surround the variables in sed expression with single quotes ' and the variables are not interpreted by shell. And please provide text, not image! Commented Feb 2, 2023 at 17:42
  • 1
    Please edit your question and 1) show us an actual text example and the output you want from that example. We cannot work with an image. 2) Are you sure you have fasta files? fastq is not the same as fasta, which one do you have? 3) are the files compressed and do they need to remain compressed? 4) What operating system are you using? 5) Can your question be simplified to "how can I remove all numbers from all lines starting with >?
    – terdon
    Commented Feb 2, 2023 at 18:09
  • Thanks @terdon, noted. 1) Done. 2) Yes, these are fasta. Fastq files appear very different in that they use special characters for the quality of the base calls. These files are output from the pipline and contain only the sample name and trimmed sequence. 3) Not compressed. Were formerly compressed, hence the extra characters in the file name. 4) Mac OXS using Bash. 5) No, because the sample name also contains digits. I cannot remove all digits. The sample name is in the file name (delimited by _) and I want to rename each line after > in the file with that string. Commented Feb 2, 2023 at 18:33

1 Answer 1

0

I can adapt this if you show us the actual file names, but assuming your sample name is always the longest string before the first _ in the file name, you can do something like this:

for file in *_R2.fastq.gz.fa; do
   sed -i '.bak' "s/\(>${file%%_*}\).*/\1/" "$file"
done

The ${file%%_*} or, more generally, ${var%%pattern} syntax is bash string manipulation for "remove the longest match of the pattern from the end of the string". Here, the pattern is _* so that will remove everything from the first _ to the end of the string, leaving you with the sample name. Note the -i '.bak' which will ensure your sed makes backup copies of every modified file. If you don't want that, you can use -i '', but you cannot use -i alone on BSD sed (which is what macOS has). Finally, the sed expression captures the part we want to keep, making it \1, and removes the rest.


If, as seems to be the case, you just want to remove an incrementing number, so you want to remove 1 from the first sequence ID line, 2 from the second and so on, you can do it more directly like this:

perl -i '.bak' -pe '$k++ if />/; s/(>.+?)$k.*/$1/' *_R2.fastq.gz.fa

The trick is that we increment $k by one each time a > is seen, and then keep anything before the value of $k and remove the rest.

1
  • This worked perfectly, thanks for your explanation of the syntax for keeping text strings. The backup files are a great idea, that is something I have read about but never implemented. Thank you! Your thoroughness is helping me learn, I appreciate it. Commented Feb 3, 2023 at 16:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .