1

I have a lot of files (>100,000) from which I would like to extract a specific string and the filename itself. Files are organized like this:

Filename: sequence.php?ID=gi|90022703|ref|YP_528530.1|

[Random text]

Description: Xylanase/Chitin Deacetylase-Like Protein [Saccharophagus Degradans 2-40]
Source: NCBI-Bacteria
Species: Saccharophagus degradans 2-40

Taxonomy: Bacteria; Proteobacteria; Gammaproteobacteria; Alteromonadales; Alteromonadaceae; Saccharophagus

I would like to extract both the filename (preferentially only everything after the ID=) and everything after 'Taxonomy' in the file.

Something like:

gi|90022703|ref|YP_528530.1| Bacteria; Proteobacteria; Gammaproteobacteria; Alteromonadales; Alteromonadaceae; Saccharophagus

Is this possible? (also for multiple files, i.e. append each new information to a text output file)

2

When dealing with multiple files, grep prints each filename before the matches. Since you don't need to search for a regular expression, you can use the -F option to search for a fixed string (which will speed things up a bit); the -r option tells grep to act recursively. The -r option is from GNU's version of grep, so it won't work on systems with other versions of grep.

grep -r -F 'Taxonomy' /path/to/directory

This will get you lines like:

sequence.php?ID=gi|90022703|ref|YP_528530.1|:Taxonomy: Bacteria; Proteobacteria; Gammaproteobacteria; Alteromonadales; Alteromonadaceae; Saccharophagus

There are a number of ways to massage this into the output you want; one would be to use cut:

grep -r -F 'Taxonomy' /path/to/directory | cut -d = -f 2 | cut -d : -f 1,3

This should turn that line into:

gi|90022703|ref|YP_528530.1|: Bacteria; Proteobacteria; Gammaproteobacteria; Alteromonadales; Alteromonadaceae; Saccharophagus

That's not an exact match to what you described; you can put | tr -d : on the end of the pipe if you really want to get rid of the colon. Finally, to redirect the output to a file:

grep -r -F 'Taxonomy' /path/to/directory | cut -d = -f 2 | cut -d : -f 1,3 > file.txt

If you want to append to a file, rather than overwriting it, use >> instead of >.

  • +1 good solution with a chain of simple tools. In case there are equal signs in the taxonomy text, use cut -d= -f 2- with the trailing dash. Similarly for the other cut command, -f 1,3- – glenn jackman Sep 29 '14 at 12:59
0
awk '/^Taxonomy:/{t=FILENAME; sub(/^.*\?ID=/, x, t); $1=t; print; close(FILENAME)}' /files/location/* > output.txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.