6

I have two files genelist.txt and data.txt. genelist.txt simply contains a single column of ~500 gene names, whereas data.txt is a tab delimited file that contains ~1000 columns (the samples) and ~30,000 rows (gene names). The general scheme of data.txt is outlined below.

       Sample 1 Sample 2 Sample 3 Sample 4  Gene A      1.04       1.81        1.92        0.45     Gene B      1.11       1.12        1.32        0.92     Gene C      0.72       0.71        0.85        1.12     Gene D      1.19       1.42        0.13        0.32   

I need to extract each row (the entire row, i.e. all samples) from data.txt containing each of the ~500 gene names in genelist.txt and have these rows extracted to a separate file. I've been told to use grep or awk and have looked into how to do this, however as a simple biologist with little/no coding experience I'm having a bit of trouble. Would it be possible for someone to explain how this is done, and hopefully provide some code for me to get underway.

It would also be neat if the extraction returned only terms that match the entire gene name in genelist.txt. For example, if I had ABC123 but not ABC1234 in genelist.txt, I would want only ABC123 to be extracted and not ABC1234.

Furthermore, after this is done how would I then check to see which of my genes from genelist.txt were not included in the extraction? (i.e. some genes may be incorrectly named, so I would have to go back and re-extract them with their alternative and/or correct name).

8

To extract the lines from data.txt with the genes listed in genelist.txt:

grep -w -F -f genelist.txt data.txt > newdata.txt

grep options used:

  • -w tells grep to match whole words only (i.e. so ABC123 won't also match ABC1234).
  • -F search for fixed strings (plain text) rather than regular expressions
  • -f genelist.txt read search patterns from the file

If you want the header (Sample 1, Sample 2, etc) line as well:

grep -w -F -f genelist.txt -e Sample data.txt > newdata.txt
  • -e Sample also search for "Sample"

To find lines in genelist.txt that aren't in newdata.txt:

grep -v -w -F -f <(sed -E -e 's/(\t|  +).*//' newdata.txt) genelist.txt
  • -v invert the search, print non-matching lines.

The rest of the grep options are the same, but instead of using a file with the -f option, it's using something called Process Substitution (See also), which allows you to use a command in place of an actual file. Whatever output the command creates is treated as the "file"'s contents.

In this case, we're using the command sed -E -e 's/(\t| +).*//' newdata.txt, which outputs each line of newdata.txt after first deleting everything from either the first TAB character or the first pair of spaces it sees. In other words, the first field (e.g. "Gene A"). I had to use TAB or double space because a) i wasn't sure if your data was space-separated or TAB separated and b) the first fields in your example contained spaces.

sed options used:

  • -E use extended regular expressions, so we can use plain (, ), and + which are more readable than having to escape them with \ as \(, \), \+.
  • -e 's/(\t| +).*//' specifies the sed script to apply against the input (newdata.txt)

Running that command on your sample data.txt would produce the following output:

$ sed -E -e 's/(\t|  +).*//' data.txt

Gene A
Gene B
Gene C
Gene D

Anyway, the output of that sed command is used as the list of search patterns by the grep command.

  • Thanks! I took a subset of data (about 20 rows) and created a genelist.txt with some of the genes included in the subset of data to play around with (data.txt is 1gb), however using your grep function the new data.txt was just a copy of data.txt. I thought maybe it was something to do with the files (they're both tab delimited, any problem there?), so I tried using csv files instead (data.csv, genelist.csv and newdata.csv) and this worked however it was only showing the last match of the grep (i.e. if genelist.csv contained genes ABC & DEF, it would only show matches for DEF). – DiscoA Jul 5 '16 at 7:47
  • are the genes in genelist.txt listed one per line (they should be)? or did the file come from an MS-DOS/Windows box? (convert with fromdos or dos2unix or tr -d '\r'). BTW, tab-delimited is fine. – cas Jul 5 '16 at 10:26
  • I successfully grep'd the list! I think it might have had something to do with the data.txt - I re-downloaded the file and your function worked fine. The genes in genelist.txt were one per line - I created it in excel and converted it to .txt. I was using the default mac text editor for opening the files, so maybe something became misaligned in it (I have since downloaded textwrangler and use that instead). – DiscoA Jul 6 '16 at 3:32
  • I will try the second part (finding the missing genes) this afternoon and report back! – DiscoA Jul 6 '16 at 3:32
  • good to hear. BTW, if your data files are TAB delimited, you can use cut -f1 newdata.txt instead of the sed command. cut defaults to using TAB as delimiter. – cas Jul 6 '16 at 3:44
3

This is quite an undertaking without any previous Linux experience. However, I think I understand what you need, and it shouldn't be too difficult. PArdon me in advance, this is a very concise crash course in addition to a very basic explanation but I'd be happy to expound in detail if it doesn't make sense, or edit as necessary.

If you simply want to parse the data.txt and move it to the genelist.txt you could simply use cat data.txt >> genelist.txt newfile.txt. (newfile.txt is the other file you mentioned it going to - the name is arbitrary).

If you want to print out the lines for a specific name, you could use cat data.txt | grep ABCD123 >> genelist.txt newfile.txt and change ABCD123 to whatever you want.

This command will ONLY output the lines found using grep (kind of like a "search" function, but it searches only by line.)

The "|" is called piping, and when coupled with "grep" command, acts a little like a filter for whatever you're looking for. (cat zoofile.txt | grep pandas for instance will look for all lines including the word "pandas" is a file names "zoofile." Note Linux IS CASE SENSITIVE and will only find EXACTLY what you put in. If you want ALL instances of either "panda, pandas, panderoons, or pandering, you could use pand*, where * is a wildcard and could be any character from 0 to 255 bits in length. This would pick up pand to pandzzzzzzzzzz and anything in between, including numbers).

You can use awk for more fancy column parsing (it's one of my favorite tools!) but it doesn't seem like it would fit here unless you ONLY want data from one of the columns based on certain parameters.

Finally, here is a good place to learn a bit about the command line. This may help with grep, but it doesn't cover awk.

https://www.codecademy.com/learn/learn-the-command-line

After that, this should cover awk in more detail. There are a lot of VERY expansive courses on awk, but they're easy to get lost in. This is a practical site that demonstrates more what you're looking to do.

https://www.ibm.com/developerworks/library/l-awk1/

EDIT - after re-reading, I may have missed something - are you looking to compare the two files and print out only things that match from one to the other? Please advise and provide an example and I'd be happy to edit my answer accordingly.

  • Thanks for the help! Yes, I'm looks to compare the files and print out only the lines from data.txt that contain the terms in genelist.txt. – DiscoA Jul 5 '16 at 7:37
3

To actually answer your question:

fgrep -w -f genelist.txt data.txt >results.txt
  • fgrep looks for fixed strings, rather than regular expressions (as grep and egrep do)
  • -w tells fgrep to match whole words, so ABC123 won't match ABC1234
  • -f genelist.txt tells fgrep to read search patterns from genelist.txt.

Seeing which genes from genelist.txt were not included in the extraction is a little more complicated. One way to do it:

awk '{ print $1 }' results.txt | fgrep -w -v -f - genelist.txt >outsiders.txt
  • awk '{ print $1 }' prints the first column in a text file; these is the list of matched genes
  • fgrep again matches fixed strings
  • -w tells fgrep to match whole words
  • -v tells it to print lines that don't match
  • -f - tells it to read the list of patterns from stdin, that is the list of matched genes from awk.

You can also make things a little more efficient by eliminating duplicates from the list of matched genes before searching, by interceding sort -u between awk and fgrep:

awk '{ print $1 }' results.txt | sort -u | fgrep -w -v -f - genelist.txt >outsiders.txt
  • Thanks! I tried this and like I said in a comment to @Cas, the results.txt file was just a copy of the data.txt file. I tried using csv formats instead of tab delimited .txt - this worked but was only displaying the last match (i.e. if I had genes ABC, DEF and GHI, only the matches for GHI would be in the results.csv file). Any more help would be appreciated. – DiscoA Jul 5 '16 at 7:50
  • 1
    @DiscoA One possible explanation for what you're describing would be if your files have DOS line endings while your grep, awk, etc. expect UNIX line endings. If that's the case, you need to convert the files with dos2unix, or whatever it's called these days. Something like sed -i 's/\r//' *.txt might work, too. – Satō Katsura Jul 5 '16 at 10:00
  • Thanks for the advice! I re-downloaded the data.txt file and tried the function again and it worked. I was using the default mac text editor to open files (maybe something became accidentily misaligned when I had it open), and I created the genelist.txt file in excel, so yep I think you're right in it being some formatting issue. I have since downloaded textwrangler to make things unix-friendly. I will try the second part this afternoon (finding the missing genes) and report back, thanks again. – DiscoA Jul 6 '16 at 3:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.