1

I have gene.csv file which is about 1.3 GB in size and has got 300 columns and more than a million rows. it looks like following

id1      id2    id3    id4         count1      count2
S1001    450    GAF    ARHGAP18    1.56E-05    1483
S1001    450    GAF    ARHGAP12    5E-05       3698
S1001    450    GAF    ARHGAP15    2.75E-06    93
S1001    450    GAF    ARHGAP17    3E-05       1889
S1001    450    GAF    ARHGAP19    4.291E-06   596
S1002    450    GAF    ARHGAP18    5.955E-05   5353
S1002    450    GAF    ARHGAP12    8.578E-08   14
S1002    450    BAF    ARHGAP15    2.91E-05    5381
S1002    450    BAF    ARHGAP17    1.78E-06    105
S1002    450    BAF    ARHGAP19    3.62E-05    5764
S1003    450    BAF    ARHGAP18    5.2697E-06  330
S1003    450    BAF    ARHGAP12    5.955E-05   2263
S1003    450    BAF    ARHGAP15    8.578E-08   3147
S1003    450    CAF    ARHGAP17    2.91E-05    50
S1003    450    CAF    ARHGAP19    5.955E-05   1595
S1004    450    CAF    ARHGAP18    8.578E-08   970
S1004    450    CAF    ARHGAP12    2.91E-05    816
S1004    450    CAF    ARHGAP15    5.955E-05   4981
S1004    450    CAF    ARHGAP17    8.578E-08   816
S1004    450    CAF    ARHGAP19    2.91E-05    4981

I want only id1, id4 and count2 data fetched in the following format (as a matrix)

id4        S1001   S1002 S1003  S1004
ARHGAP18    1483   5353  330    970
ARHGAP12    3698   14    2263   816
ARHGAP15    93     5381  3147   4981
ARHGAP17    1889   105   50     816
ARHGAP19    596    5764  1595   4981

Please note i need to be repeated only once and as a column name in a new file (Since id1 is repeated in each row for each gene). How can i get this in from simple bash command (matrix of gene names(rows) and sample ids (columns).

  • do you have python pandas installed? – RomanPerekhrest Jan 13 at 15:57
  • Hi Roman, I am not a programmer, I am a biologist and I am not very well with python. – David Jan 13 at 16:14
2


with Miller, using reshape

mlr --tsv cut -o -f id4,id1,count2 then reshape -s id1,count2 input.tsv

you will have

id4     S1001   S1002   S1003   S1004
ARHGAP18        1483    5353    330     970
ARHGAP12        3698    14      2263    816
ARHGAP15        93      5381    3147    4981
ARHGAP17        1889    105     50      816
ARHGAP19        596     5764    1595    4981

As input and output I have set a TSV file. Is you text file tab separated?

Some comment:

  • --tsv is to set input & output format;
  • cut to extract only id4,id1 and count2;
  • reshape to modify the structure.
  • it did not output as desired, in fact after reshape it showed only 3 columns. my command was mlr reshape -s icgc_specimen_id,raw_read_count miller_1.tsv – David Jan 14 at 5:56
  • @David I think you are using another input file. Please could you share some rows here? – aborruso Jan 14 at 9:00
  • I have made an example file with only min rows, but the original file is of 2.7 GBs.Please have a look at the following link, here I need only [drive.google.com/file/d/1joMO2D1RuppJBbepeEu3-lLA7LQZJeJR/… i need icgc_specimen_id as colmns, gene_id as row and raw_read_count column. Thanks a lot :) – David Jan 14 at 13:54
  • 1
    Awesome! It worked! Thanks a lot for your kind help... You made my day! – David Jan 15 at 4:46
  • 1
    First, it crashed my system, Later I had split the file into 4 parts and the file was processed in half an hour. – David Jan 15 at 13:18
0

What you are asking for is a cross-tabulation (or pivot table). You can achieve it using GNU Datamash

datamash --header-in --whitespace crosstab id4,id1 unique count2 < file
    S1001   S1002   S1003   S1004
ARHGAP12    3698    14  2263    816
ARHGAP15    93  5381    3147    4981
ARHGAP17    1889    105 50  816
ARHGAP18    1483    5353    330 970
ARHGAP19    596 5764    1595    4981

(you can skip the --whitespace if your data are tab delimited).

However there may be dedicated bioinformatics tools that are better suited to the task.

  • Thanks a lot, the code works like a charm. Thanks a ton! – David Jan 13 at 16:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.