1

I have 3 file like these:

file1:

ko00980 Metabolism of xenobiotics by cytochrome P450 (5)

ko:K00121 frmA; S-(hydroxymethyl)glutathione dehydrogenase / alcohol dehydrogenase [EC:1.1.1.284 1.1.1.1]
ko:K00699 UGT; glucuronosyltransferase [EC:2.4.1.17]
ko:K00799 GST; glutathione S-transferase [EC:2.5.1.18]
ko:K07408 CYP1A1; cytochrome P450, family 1, subfamily A, polypeptide 1 [EC:1.14.14.1]
ko:K07409 CYP1A2; cytochrome P450, family 1, subfamily A, polypeptide 2 [EC:1.14.14.1]

ko00982 Drug metabolism - cytochrome P450 (5)

ko:K00121 frmA; S-(hydroxymethyl)glutathione dehydrogenase / alcohol dehydrogenase [EC:1.1.1.284 1.1.1.1]
ko:K00485 FMO; dimethylaniline monooxygenase (N-oxide forming) [EC:1.14.13.8]
ko:K00699 UGT; glucuronosyltransferase [EC:2.4.1.17]
ko:K00799 GST; glutathione S-transferase [EC:2.5.1.18]
ko:K07409 CYP1A2; cytochrome P450, family 1, subfamily A, polypeptide 2 [EC:1.14.14.1]

ko00983 Drug metabolism - other enzymes (4)

ko:K00088 guaB; IMP dehydrogenase [EC:1.1.1.205]
ko:K00699 UGT; glucuronosyltransferase [EC:2.4.1.17]
ko:K00857 tdk; thymidine kinase [EC:2.7.1.21]
ko:K00876 udk; uridine kinase [EC:2.7.1.48]

file2:

ko00980 Metabolism of xenobiotics by cytochrome P450 (6)

ko:K00001 E1.1.1.1; alcohol dehydrogenase [EC:1.1.1.1]
ko:K00079 CBR1; carbonyl reductase 1 [EC:1.1.1.184 1.1.1.189 1.1.1.197]
ko:K00121 frmA; S-(hydroxymethyl)glutathione dehydrogenase / alcohol dehydrogenase [EC:1.1.1.284 1.1.1.1]
ko:K00799 GST; glutathione S-transferase [EC:2.5.1.18]
ko:K07408 CYP1A1; cytochrome P450, family 1, subfamily A, polypeptide 1 [EC:1.14.14.1]
ko:K07409 CYP1A2; cytochrome P450, family 1, subfamily A, polypeptide 2 [EC:1.14.14.1]

ko00982 Drug metabolism - cytochrome P450 (4)

ko:K00001 E1.1.1.1; alcohol dehydrogenase [EC:1.1.1.1]
ko:K00121 frmA; S-(hydroxymethyl)glutathione dehydrogenase / alcohol dehydrogenase [EC:1.1.1.284 1.1.1.1]
ko:K00799 GST; glutathione S-transferase [EC:2.5.1.18]
ko:K07409 CYP1A2; cytochrome P450, family 1, subfamily A, polypeptide 2 [EC:1.14.14.1]

ko00983 Drug metabolism - other enzymes (8)

ko:K00088 guaB; IMP dehydrogenase [EC:1.1.1.205]
ko:K00106 XDH; xanthine dehydrogenase/oxidase [EC:1.17.1.4 1.17.3.2]
ko:K00760 hprT; hypoxanthine phosphoribosyltransferase [EC:2.4.2.8]
ko:K00876 udk; uridine kinase [EC:2.7.1.48]
ko:K01431 UPB1; beta-ureidopropionase [EC:3.5.1.6]
ko:K01464 DPYS; dihydropyrimidinase [EC:3.5.2.2]
ko:K01519 ITPA; inosine triphosphate pyrophosphatase [EC:3.6.1.19]
ko:K13421 UMPS; uridine monophosphate synthetase [EC:2.4.2.10 4.1.1.23]

file3: 

ko00980 Metabolism of xenobiotics by cytochrome P450 (7)

ko:K00001 E1.1.1.1; alcohol dehydrogenase [EC:1.1.1.1]
ko:K00079 CBR1; carbonyl reductase 1 [EC:1.1.1.184 1.1.1.189 1.1.1.197]
ko:K00121 frmA; S-(hydroxymethyl)glutathione dehydrogenase / alcohol dehydrogenase [EC:1.1.1.284 1.1.1.1]
ko:K00699 UGT; glucuronosyltransferase [EC:2.4.1.17]
ko:K00799 GST; glutathione S-transferase [EC:2.5.1.18]
ko:K07408 CYP1A1; cytochrome P450, family 1, subfamily A, polypeptide 1 [EC:1.14.14.1]
ko:K07409 CYP1A2; cytochrome P450, family 1, subfamily A, polypeptide 2 [EC:1.14.14.1]

ko00982 Drug metabolism - cytochrome P450 (6)

ko:K00001 E1.1.1.1; alcohol dehydrogenase [EC:1.1.1.1]
ko:K00121 frmA; S-(hydroxymethyl)glutathione dehydrogenase / alcohol dehydrogenase [EC:1.1.1.284 1.1.1.1]
ko:K00485 FMO; dimethylaniline monooxygenase (N-oxide forming) [EC:1.14.13.8]
ko:K00699 UGT; glucuronosyltransferase [EC:2.4.1.17]
ko:K00799 GST; glutathione S-transferase [EC:2.5.1.18]
ko:K07409 CYP1A2; cytochrome P450, family 1, subfamily A, polypeptide 2 [EC:1.14.14.1]

ko00983 Drug metabolism - other enzymes (8)

ko:K00088 guaB; IMP dehydrogenase [EC:1.1.1.205]
ko:K00207 DPYD; dihydropyrimidine dehydrogenase (NADP+) [EC:1.3.1.2]
ko:K00699 UGT; glucuronosyltransferase [EC:2.4.1.17]
ko:K00857 tdk; thymidine kinase [EC:2.7.1.21]
ko:K00876 udk; uridine kinase [EC:2.7.1.48]
ko:K01431 UPB1; beta-ureidopropionase [EC:3.5.1.6]
ko:K01489 cdd; cytidine deaminase [EC:3.5.4.5]
ko:K01951 guaA; GMP synthase (glutamine-hydrolysing) [EC:6.3.5.2]

Each file has header lines which start with ko***** and a name and number of sub-header lines in a parenthesis, for example:

ko00980 Metabolism of xenobiotics by cytochrome P450 (5)

the sub-header lines start with ko:K*****

I want to merge the sub-header lines for each header line in 3 files and do uniq. I want result like these:

ko00980:

ko:K00121 frmA; S-(hydroxymethyl)glutathione dehydrogenase / alcohol dehydrogenase [EC:1.1.1.284 1.1.1.1]
ko:K00699 UGT; glucuronosyltransferase [EC:2.4.1.17]
ko:K00799 GST; glutathione S-transferase [EC:2.5.1.18]
ko:K07408 CYP1A1; cytochrome P450, family 1, subfamily A, polypeptide 1 [EC:1.14.14.1]
ko:K07409 CYP1A2; cytochrome P450, family 1, subfamily A, polypeptide 2 [EC:1.14.14.1]
ko:K00001 E1.1.1.1; alcohol dehydrogenase [EC:1.1.1.1]
ko:K00079 CBR1; carbonyl reductase 1 [EC:1.1.1.184 1.1.1.189 1.1.1.197]

ko00982

ko:K00121 frmA; S-(hydroxymethyl)glutathione dehydrogenase / alcohol dehydrogenase [EC:1.1.1.284 1.1.1.1]
ko:K00485 FMO; dimethylaniline monooxygenase (N-oxide forming) [EC:1.14.13.8]
ko:K00699 UGT; glucuronosyltransferase [EC:2.4.1.17]
ko:K00799 GST; glutathione S-transferase [EC:2.5.1.18]
ko:K07409 CYP1A2; cytochrome P450, family 1, subfamily A, polypeptide 2 [EC:1.14.14.1]
ko:K00001 E1.1.1.1; alcohol dehydrogenase [EC:1.1.1.1]
ko:K00088 guaB; IMP dehydrogenase [EC:1.1.1.205]
ko:K00207 DPYD; dihydropyrimidine dehydrogenase (NADP+) [EC:1.3.1.2]
ko:K00857 tdk; thymidine kinase [EC:2.7.1.21]
ko:K00876 udk; uridine kinase [EC:2.7.1.48]
ko:K01431 UPB1; beta-ureidopropionase [EC:3.5.1.6]
ko:K01489 cdd; cytidine deaminase [EC:3.5.4.5]
ko:K01951 guaA; GMP synthase (glutamine-hydrolysing) [EC:6.3.5.2]

ko00983

ko:K00088 guaB; IMP dehydrogenase [EC:1.1.1.205]
ko:K00699 UGT; glucuronosyltransferase [EC:2.4.1.17]
ko:K00857 tdk; thymidine kinase [EC:2.7.1.21]
ko:K00876 udk; uridine kinase [EC:2.7.1.48]
ko:K00106 XDH; xanthine dehydrogenase/oxidase [EC:1.17.1.4 1.17.3.2]
ko:K00760 hprT; hypoxanthine phosphoribosyltransferase [EC:2.4.2.8]
ko:K01431 UPB1; beta-ureidopropionase [EC:3.5.1.6]
ko:K01464 DPYS; dihydropyrimidinase [EC:3.5.2.2]
ko:K01519 ITPA; inosine triphosphate pyrophosphatase [EC:3.6.1.19]
ko:K13421 UMPS; uridine monophosphate synthetase [EC:2.4.2.10 4.1.1.23]
ko:K00207 DPYD; dihydropyrimidine dehydrogenase (NADP+) [EC:1.3.1.2]
ko:K01489 cdd; cytidine deaminase [EC:3.5.4.5]
ko:K01951 guaA; GMP synthase (glutamine-hydrolysing) [EC:6.3.5.2]
Improve this question
3
  • so, what have you tried so far?
    – Sundeep
    Jul 25, 2016 at 13:53
  • first i tried to use split and csplit but they didnot worek
    – Mehdi
    Jul 25, 2016 at 14:02
  • can you add those commands to question and your reasoning to use those commands
    – Sundeep
    Jul 25, 2016 at 14:02

3 Answers 3

Reset to default
2

With awk you could run:

awk '/^ko[^:]/{fn=$1;next};/./{id=fn$1;if (!(seen[id]++)){print > fn}}' file[123]

On each header line it saves the identifier ko***** as fn, on sub-header lines it saves fn$11 as id using it as index of array seen and if it's the first time that id is seen it writes the line to fn.

1: you may as well use fn$0

Improve this answer
1

There may be some magic super-duper mashup of commands, but sometimes "linear" is easiest to understand and maintain.

So we just need to keep track of the filename, based on the header line, and append data. We can then sort -u the results to get the unique lines:

#!/bin/bash

# Clean out old results from previous runs
/bin/rm -f ko*

for file in $@
do
  filename=UNKNOWN
  echo Processing $file
  while read -r line
  do
    case $line in
      ko:*) printf "%s\n" "$line" >> $filename ;;
       ko*) filename=${line%% *} ; echo Switching to $filename ;;
        "") # Do nothing
            ;;
         *) echo Ignoring unknown line: $line
    esac
  done < $file
done

for file in ko*
do
  echo Making unique: $file
  sort -u -o $file $file
done

We can run this with the three source files:

$ ./pattern_split file1 file2 file3
Processing file1
Switching to ko00980
Switching to ko00982
Switching to ko00983
Processing file2
Switching to ko00980
Switching to ko00982
Switching to ko00983
Processing file3
Switching to ko00980
Switching to ko00982
Switching to ko00983
Making unique: ko00980
Making unique: ko00982
Making unique: ko00983

We can see it created three files that were made unique. Looking at the first:

$ cat ko00980
ko:K00001 E1.1.1.1; alcohol dehydrogenase [EC:1.1.1.1]
ko:K00079 CBR1; carbonyl reductase 1 [EC:1.1.1.184 1.1.1.189 1.1.1.197]
ko:K00121 frmA; S-(hydroxymethyl)glutathione dehydrogenase / alcohol dehydrogenase [EC:1.1.1.284 1.1.1.1]
ko:K00699 UGT; glucuronosyltransferase [EC:2.4.1.17]
ko:K00799 GST; glutathione S-transferase [EC:2.5.1.18]
ko:K07408 CYP1A1; cytochrome P450, family 1, subfamily A, polypeptide 1 [EC:1.14.14.1]
ko:K07409 CYP1A2; cytochrome P450, family 1, subfamily A, polypeptide 2 [EC:1.14.14.1]

Now this solution should be hardened against rogue data in the data files (eg what if there's a ko123/456 file? That'd break. But this is the outline of how the problem can be solved.

Improve this answer
0

So, moving the lines in the file into a separate file based on header?

Something like this will do it I think: 

#!/usr/bin/env perl
use strict;
use warnings 'all'; 

#hash of output filehandles. 
my %output_files; 

#detect dupes
my %seen; 

my $ko_num = 'NULL'; 

#<> is the 'magic' filehandle. You can either use it to iterate STDIN
#or take a list of file names on the command line (just like sed/grep etc.)
while ( my $line = <> ) { 
   #see if the line starts with 'ko':
   if ( $line =~ m/(^ko\d+)/) {  
       $ko_num = $1;
       #open a new file - for overwriting (so we only do this once)
       open ( $output_files{$ko_num}, '>', $ko_num ) or die $! unless $output_files{$ko_num}; 
       #skip printing - could write a header here instead. 
       next;
   }
   #look for a 'K' number. 
   if ( my ($K_id) = $line =~ m/ko:(K\d+)/ ) {
       #skip it if we've already seen this combination of 'ko' number 
       #and k number.    
       next if $seen{$ko_num}{$K_id}++; 
       #print the output to this particular output file. 
       print {$output_files{$ko_num}} $line; 
   }
}
#close the filehandles. 
close ( $_ ) for values %output_files;

So in this way - you can run 'myscript.pl file1.txt file2.txt file3.txt' and it should do the right thing, in a scalable way. It doesn't even care if they're separate files, or a single stream.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .