1

I have a file (list_file) that is a list of names and another file (data_file) that contains hundreds of thousands of lines of text, each beginning with a name in list_file. I want to do a search of the data_file for all lines that start with each name in list_file and export the lines for each name to a new file *.txt (where * is the name in the list)

I have this

grep -f "list_file" data_file > out.txt

but this does not separate each line in the list into its own file.

Sample list_file:

100_fullA
100_fullB
105_fullA
105_fullB
112_fullA
112_fullB
121_fullA
121_fullB

Sample data_file:

100_fullA NGATCATCGACAC
100_fullB NGATCATCGACAC
105_fullA NGATCATCGACAC
105_fullB NGATCATCGACAC
112_fullA NGATCATCGACAC
112_fullB NGATCATCGACAC
121_fullA NGATCATCGACAC
1

You could construct a look-up table (or hash) from list_file, for example using an associative array in awk:

awk 'NR==FNR {list[$1]=1; next} $1 in list {print > $1".txt"}' list_file data_file

The outputs will go in files 100_fullA.txt, 100_fullB.txt and so on.

0

You could do it this way by constructing the commands on the fly:

grep -f list_file data_file | sed -e "s/^\([^ ]*\).*/echo '&' >> \1;/" | sh
  • care to explain? – Rakesh.N Feb 21 '17 at 4:30
  • grep gets all lines in data_file which match the patterns stored in list_file. The sed then prepares these lines into a form suitable for bash. For example, a line 100_fullA TCAGGNG => echo 100_fullA TCAGGNG >> 100_fullA. These generated on the fly lines are the fed to bash to effect the operation. – Rakesh Sharma Feb 22 '17 at 5:55
0

Here is a solution using Bash:

#!/bin/bash

while read pointer; do
 filename="$(echo $pointer | cut -d ' ' -f 1)" 
 if grep $filename list_file > /dev/null; then
  echo $pointer >> output/"$filename".txt
 fi
done < data_file

Here's a breakdown line-by-line:

  • Line 3 is the beginning of the while loop used to loop through data_file.

  • Line 4 echos the line read in the current iteration. The line is then passed to cut, which cuts out the first part of the line useing a space as the delimiter. The result is then assigned to a variable called "filename".

  • Line 5 uses grep to determine whether or not the previously determined value exists in list_file. If grep succeeds in finding the value (return status 0), the script continues to line 6. If grep does not find anything (return status 1), the script starts the loop over.

  • Line 6 echos the entire line to output/"$filename".txt.

  • Line 7 closes the if statement.

  • Line 8 closes the loop and is where data_file is referenced.

Other important notes:

  • The "output/" directory must be created before the script is run, otherwise you'll get an error like "output: no such file or directory". If this is a problem, it can be easily solved by adding "mkdir output" to the beginning of the script.
  • You mentioned data_file has hundreds of thousands of lines. Because of this, this script will likely take a long time to complete. If you find yourself querying this file for information often, it would be worth translating the information into MariaDB or similar database.
  • If list_file also contains many many entries, the script will take an insanely long time to run, because the grep on line 5 queries the entire list_file for every iteration of the loop. Again, this is a problem that can be solved with the tools available in a SQL database.
  • This worked well but looking at the new file it would be great to remove the name string from the beginning of each line. – Josh Feb 23 '17 at 20:03
  • This can be easily done with a similar cut command on the $pointer variable, then write the new variable to the file instead of $pointer: pastebin.com/75kvHjn4 Edited to use a PasteBin link instead, since I guess comments can't do code blocks. – Brad Johnson Feb 25 '17 at 19:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.