5

I would like to split a file to different files based on the information in the first line. For example, I have:

Input:

1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 3 3 3 3 3 3 4 4 4 4 4 4 4 30 30 30 30
0 2 2 0 2 0 2 0 2 0 2 2 0 0 2 2 2 0 1 1 1 2 0 2 0 0 0 2 0 0 2 0 2
0 2 1 0 1 0 1 1 1 0 2 2 0 0 2 2 2 0 0 0 0 2 0 2 0 0 1 2 0 0 2 0 2
0 2 1 0 1 0 1 1 1 0 2 2 0 0 2 2 2 0 0 0 0 2 0 2 0 0 1 2 0 0 2 0 2

Desired output:

output1.txt

02202020
02101011
02101011

output2.txt

2022002
1022002
1022002

output3.txt

220111
220000
220000

output4.txt

202000200202
202001200202
202001200202

output30.txt

0202
0202
0202
  • Every line has the same amount of characters? – tachomi Sep 30 '15 at 18:56
  • @ don_crissti, smart! :) ok, this is a genome data for 31 chromosomes. so I made a header (11.. 222....31..) and I attached it to the genome data and now I want to slpit the whole genome data to separated ones based on chromosome numbers :) – zara Sep 30 '15 at 19:34
  • @ don_crissti, no the other file was separated one which I merged it with the headers using:cat header.txt calls.txt > callss.txt – zara Sep 30 '15 at 19:39
  • Would you accept a couple of Perl lines? (Perl is available on most dists) – Ring Ø Sep 30 '15 at 20:13
  • @ ring0 , yes I appreciate that! – zara Sep 30 '15 at 20:14
3

Perl script.

Set the file name in $in in place of genome.txt or give the name as argument.

Name the script counter.pl and give it executable rights, and run it as ./counter.pl

chmod 755 counter.pl
./counter.pl

or alternatively

chmod 755 counter.pl
./counter.pl genome.txt

counter.pl:

#!/usr/bin/perl

use strict;
use warnings;

my $in = $ARGV[0] || 'genome.txt'; # input file name

open (my $F, '<', $in) or die "Cannot open input file $!";
my $n = 0;
my %fd = ();
my @fd = ();

while (<$F>) {
        # trim
        s/^\s+//;
        s/\s+$//;
        next if (!$_); # Skip empty lines
        my @x = split(/\s+/, $_);
        # 1st line, open files
        if ( ! $n++)  {
           my $fd = 0;
           for (@x) {
              open ($fd{$_}, '>', "output$_.txt") 
                or die ("Cannot open file $!")
                  if (!exists($fd{$_}));
              $fd[$fd++] = $_;
           }
        }
        else { # Write data
           die ("Should have " . ($#fd+1) . " entries on line $n")
             if ($#x != $#fd);
           for (0 .. $#x) {
              print {$fd{$fd[$_]}} ($x[$_]);
           }
           print {$fd{$_}} ("\n") for (keys %fd);
        }
}

close $fd{$_} for (keys %fd);
close $F;
# the end

Fixed the numbers of words per line (was sometimes 32, sometimes 33 in the example).

This version can accommodate any variation of columns, but all lines have to have the same numbers of words. An error will occur (the die lines) if the number of words is different, or if it cannot open files.

Just adjust the file name ($in).

Input file: (removed the extra 0 near the end)

1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 3 3 3 3 3 3 4 4 4 4 4 4 4 30 30 30 30
0 2 2 0 2 0 2 0 2 0 2 2 0 0 2 2 2 0 1 1 1 2 0 2 0 0 0 2 0 2 0 2
0 2 1 0 1 0 1 1 1 0 2 2 0 0 2 2 2 0 0 0 0 2 0 2 0 0 1 2 0 2 0 2
0 2 1 0 1 0 1 1 1 0 2 2 0 0 2 2 2 0 0 0 0 2 0 2 0 0 1 2 0 2 0 2

output1.txt

02202020
02101011
02101011

output2.txt

2022002
1022002
1022002

output30.txt

0202
0202
0202

output3.txt

220111
220000
220000

output4.txt

2020002
2020012
2020012
  • how should I run this? where I should put my input file name? should I save this in a .pl format? – zara Sep 30 '15 at 21:20
  • all lines have the same amount of charactors except the header line(first line) has a lot more character. is that a problem? – zara Sep 30 '15 at 21:29
  • Number of characters doesn't matter, number of words does. E.g. "1 2 3 / 456 67 99" fits (3 words 1st line then 3 words below), but "1 2 3 / 4 5 6 7" doesn't (3 words then 4 words). If this is not what you want the script can easily be changed. – Ring Ø Sep 30 '15 at 21:31
  • my file name is genome.txt should I write genome.txt instead of in or instead of x.in ? – zara Sep 30 '15 at 21:38
  • ok I did but I got this error:./counter.pl Missing right curly or square bracket at ./counter.pl line 36, at end of line syntax error at ./counter.pl line 36, at EOF Execution of ./counter.pl aborted due to compilation errors. do you know whay? :( – zara Sep 30 '15 at 21:42
3
$ awk '
    NR == 1 {
        for (i=1; i<=NF; i++) {
            output[i] = "output" $i ".txt"
            files[output[i]] = 1
        }
        next
    }
    {
        for (i=1; i<=NF; i++)  printf "%s", $i > output[i]
        for (file in files)    print ""        > file
    }
' input.filename

$ for f in output*.txt; do echo $f; cat $f; done
output1.txt
02202020
02101011
02101011
output2.txt
2022002
1022002
1022002
output3.txt
220111
220000
220000
output30.txt
00202
00202
00202
output4.txt
2020002
2020012
2020012

Note that your header line has 32 fields and the other lines have 33. That needed to be fixed first.

  • this gives me error all the time :( where should I put y input file name? how should I run this? – zara Sep 30 '15 at 21:25
  • i get this error:awk: cmd. line:8: (FILENAME=file FNR=2) fatal: expression for `>' redirection has null string value – zara Sep 30 '15 at 21:55
  • I mentioned that in my answer: you have to ensure that the first line has the same number of fields as all the other lines. You get to the 33rd field on line 2, but there's no value for output[33] – glenn jackman Oct 1 '15 at 13:18
2

OK, also for fun - a pure Bash version (as requested) that leans heavily on the builtin read to send words to arrays and save them to file. The files are nicely formatted as output001.txt .... output030.txt. Used a data file as modified by @ringO for testing purposes. Not tested but on very large files it could be more time and resource efficient than others.

Data:

1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 3 3 3 3 3 3 4 4 4 4 4 4 4 30 30 30 30
0 2 2 0 2 0 2 0 2 0 2 2 0 0 2 2 2 0 1 1 1 2 0 2 0 0 0 2 0 2 0 2
0 2 1 0 1 0 1 1 1 0 2 2 0 0 2 2 2 0 0 0 0 2 0 2 0 0 1 2 0 2 0 2
0 2 1 0 1 0 1 1 1 0 2 2 0 0 2 2 2 0 0 0 0 2 0 2 0 0 1 2 0 2 0 2

Source:

#!/usr/bin/env bash

# genome : to sort genome data sets according to patterns of the first (header)
# line of the file.  Data must be space delimited.  No dependencies.
#
# Usage:
#
#                      ./genome "data.txt" 

# global arrays
sc=(  )             # array of set element counts
sn=(  )             # array of set id numbers

# output_file "set id"

# change the output pattern and digit output width as required - default
# pattern is output.txt and digit width of three : output000.txt
output_file(){
    # format concept: pattern000.txt
    local op='output.txt'     # output pattern
    local ow=3                # output width: 3 => 000
    printf "%s%0${ow}d.%s" "${op%%.*}" "$1" "${op##*.}"
}

# define_sets "input.txt"

# identify sets - get elements count and sets id numbers from file
# header.
define_sets(){
    # declare and initialize
    local a an b c n
    read -r c < "$1"
    read -r a b <<< "$c"
    n=0; sn=( $a )

    # recurse header, identify sets
    until [[ -z $b ]]
    do
        n=$((n+1))
        an=$a
        read -r a b <<< "$b"
        [[ $an == $a ]] || { sn+=( $a ); sc+=( $n ); n=0; }
    done
    n=$((n+1))
    sc+=( $n )
}

# reset_files

# optional function, clears file data, otherwise data is appended to existing
# output files.
reset_files(){
    for s in ${sn[@]}
    do
        > "$(output_file "$s")"
    done
}

# extract_data "input.txt"

# use defined sets to extract data from the input file and send it to required
# output files. Uses nested 'while read' to bypass file header as data is saved.
extract_data(){
    local a c n s fn da=( )
    while read -a da
    do
        while read -a da
        do
            a=0 n=0
            for s in ${sc[@]}
            do
                c="$(echo "${da[@]:$a:$s}")" # words => string
                echo "${c// /}" >> "$(output_file "${sn[$n]}")"  # save
                n=$((n+1))
                a=$((a+s))
            done
        done
    done < "$1"
}

define_sets "$1"    # get data set structure from header
reset_files         # optional, clears and resets files
extract_data "$1"   # get data from input file and save

# end file

Data Output:

$ cat output001.txt 
02202020
02101011
02101011

$ cat output002.txt 
2022002
1022002
1022002

$ cat output003.txt 
220111
220000
220000

$ cat output004.txt 
2020002
2020012
2020012

$ cat output030.txt 
0202
0202
0202
  • @zara - this uses a couple of tricks for better efficiency - it recurses without a function call and the nested while read avoids unnecessary truth tests. You may find that it is faster than Perl and Awk (although this is speculation of course). But at any rate you will not need any dependencies. – AsymLabs Oct 5 '15 at 19:49
  • can I ask another question from you? – zara Oct 5 '15 at 20:24
  • @zara you can, can't guarantee it can be answered... – AsymLabs Oct 5 '15 at 20:58
  • if I want to do vice versa , having several files and combine them to look like the original on, then how may I do it? – zara Oct 5 '15 at 21:02
  • @zara it should be possible to use nested while read -r line; do ... done; and then echo the lines to a single file in the order required - there may be a more efficient way but not having seen the problem it is difficult to say more... – AsymLabs Oct 5 '15 at 21:41
1

Just for fun, some other solution:

awk '{ for (i=1; i<=NF;i++){
          if (NR==1) { file[i]=$i }
          if (NR!=1) { f="output" file[i]   ".txt";
                       g="output" file[i+1] ".txt";
                       printf("%s%s",$i,f==g?OFS:ORS)>>f;
                       close(f);
                      }
          }
      }' file

If you need fields that are not separated, change ?OFS: to ?"":.

The default file that receive un-paired values is output.txt. That file will receive values if the number of columns on the first line does not match the next lines processed. If all is correct, it should be empty. If it exists after the script has run, there is a problem somewhere.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.