Extracting file name and string from multiple files

Question

I have a lot of files (>100,000) from which I would like to extract a specific string and the filename itself. Files are organized like this:

Filename: sequence.php?ID=gi|90022703|ref|YP_528530.1|

[Random text]

Description: Xylanase/Chitin Deacetylase-Like Protein [Saccharophagus Degradans 2-40]
Source: NCBI-Bacteria
Species: Saccharophagus degradans 2-40

Taxonomy: Bacteria; Proteobacteria; Gammaproteobacteria; Alteromonadales; Alteromonadaceae; Saccharophagus

I would like to extract both the filename (preferentially only everything after the ID=) and everything after 'Taxonomy' in the file.

Something like:

gi|90022703|ref|YP_528530.1| Bacteria; Proteobacteria; Gammaproteobacteria; Alteromonadales; Alteromonadaceae; Saccharophagus

Is this possible? (also for multiple files, i.e. append each new information to a text output file)

Answer 1

When dealing with multiple files, grep prints each filename before the matches. Since you don't need to search for a regular expression, you can use the -F option to search for a fixed string (which will speed things up a bit); the -r option tells grep to act recursively. The -r option is from GNU's version of grep, so it won't work on systems with other versions of grep.

grep -r -F 'Taxonomy' /path/to/directory

This will get you lines like:

sequence.php?ID=gi|90022703|ref|YP_528530.1|:Taxonomy: Bacteria; Proteobacteria; Gammaproteobacteria; Alteromonadales; Alteromonadaceae; Saccharophagus

There are a number of ways to massage this into the output you want; one would be to use cut:

grep -r -F 'Taxonomy' /path/to/directory | cut -d = -f 2 | cut -d : -f 1,3

This should turn that line into:

gi|90022703|ref|YP_528530.1|: Bacteria; Proteobacteria; Gammaproteobacteria; Alteromonadales; Alteromonadaceae; Saccharophagus

That's not an exact match to what you described; you can put | tr -d : on the end of the pipe if you really want to get rid of the colon. Finally, to redirect the output to a file:

grep -r -F 'Taxonomy' /path/to/directory | cut -d = -f 2 | cut -d : -f 1,3 > file.txt

If you want to append to a file, rather than overwriting it, use >> instead of >.

Answer 2

awk '/^Taxonomy:/{t=FILENAME; sub(/^.*\?ID=/, x, t); $1=t; print; close(FILENAME)}' /files/location/* > output.txt