1

I have a file (list_file) that is a list of names and another file (data_file) that contains hundreds of thousands of lines of text, each beginning with a name in list_file. I want to do a search of the data_file for all lines that start with each name in list_file and export the lines for each name to a new file *.txt (where * is the name in the list)

I have this

grep -f "list_file" data_file > out.txt

but this does not separate each line in the list into its own file.

Sample list_file:

100_fullA
100_fullB
105_fullA
105_fullB
112_fullA
112_fullB
121_fullA
121_fullB

Sample data_file:

100_fullA NGATCATCGACAC
100_fullB NGATCATCGACAC
105_fullA NGATCATCGACAC
105_fullB NGATCATCGACAC
112_fullA NGATCATCGACAC
112_fullB NGATCATCGACAC
121_fullA NGATCATCGACAC
0

3 Answers 3

1

You could construct a look-up table (or hash) from list_file, for example using an associative array in awk:

awk 'NR==FNR {list[$1]=1; next} $1 in list {print > $1".txt"}' list_file data_file

The outputs will go in files 100_fullA.txt, 100_fullB.txt and so on.

0

You could do it this way by constructing the commands on the fly:

grep -f list_file data_file | sed -e "s/^\([^ ]*\).*/echo '&' >> \1;/" | sh
2
  • care to explain?
    – Rakesh.N
    Commented Feb 21, 2017 at 4:30
  • grep gets all lines in data_file which match the patterns stored in list_file. The sed then prepares these lines into a form suitable for bash. For example, a line 100_fullA TCAGGNG => echo 100_fullA TCAGGNG >> 100_fullA. These generated on the fly lines are the fed to bash to effect the operation. Commented Feb 22, 2017 at 5:55
0

Here is a solution using Bash:

#!/bin/bash

while read pointer; do
 filename="$(echo $pointer | cut -d ' ' -f 1)" 
 if grep $filename list_file > /dev/null; then
  echo $pointer >> output/"$filename".txt
 fi
done < data_file

Here's a breakdown line-by-line:

  • Line 3 is the beginning of the while loop used to loop through data_file.

  • Line 4 echos the line read in the current iteration. The line is then passed to cut, which cuts out the first part of the line useing a space as the delimiter. The result is then assigned to a variable called "filename".

  • Line 5 uses grep to determine whether or not the previously determined value exists in list_file. If grep succeeds in finding the value (return status 0), the script continues to line 6. If grep does not find anything (return status 1), the script starts the loop over.

  • Line 6 echos the entire line to output/"$filename".txt.

  • Line 7 closes the if statement.

  • Line 8 closes the loop and is where data_file is referenced.

Other important notes:

  • The "output/" directory must be created before the script is run, otherwise you'll get an error like "output: no such file or directory". If this is a problem, it can be easily solved by adding "mkdir output" to the beginning of the script.
  • You mentioned data_file has hundreds of thousands of lines. Because of this, this script will likely take a long time to complete. If you find yourself querying this file for information often, it would be worth translating the information into MariaDB or similar database.
  • If list_file also contains many many entries, the script will take an insanely long time to run, because the grep on line 5 queries the entire list_file for every iteration of the loop. Again, this is a problem that can be solved with the tools available in a SQL database.
2
  • This worked well but looking at the new file it would be great to remove the name string from the beginning of each line.
    – Josh
    Commented Feb 23, 2017 at 20:03
  • This can be easily done with a similar cut command on the $pointer variable, then write the new variable to the file instead of $pointer: pastebin.com/75kvHjn4 Edited to use a PasteBin link instead, since I guess comments can't do code blocks. Commented Feb 25, 2017 at 19:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .