5

I would like to split a file to different files based on the information in the first line. For example, I have:

Input:

1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 3 3 3 3 3 3 4 4 4 4 4 4 4 30 30 30 30
0 2 2 0 2 0 2 0 2 0 2 2 0 0 2 2 2 0 1 1 1 2 0 2 0 0 0 2 0 0 2 0 2
0 2 1 0 1 0 1 1 1 0 2 2 0 0 2 2 2 0 0 0 0 2 0 2 0 0 1 2 0 0 2 0 2
0 2 1 0 1 0 1 1 1 0 2 2 0 0 2 2 2 0 0 0 0 2 0 2 0 0 1 2 0 0 2 0 2

Desired output:

output1.txt

02202020
02101011
02101011

output2.txt

2022002
1022002
1022002

output3.txt

220111
220000
220000

output4.txt

202000200202
202001200202
202001200202

output30.txt

0202
0202
0202
5
  • Every line has the same amount of characters?
    – tachomi
    Commented Sep 30, 2015 at 18:56
  • @ don_crissti, smart! :) ok, this is a genome data for 31 chromosomes. so I made a header (11.. 222....31..) and I attached it to the genome data and now I want to slpit the whole genome data to separated ones based on chromosome numbers :)
    – zara
    Commented Sep 30, 2015 at 19:34
  • @ don_crissti, no the other file was separated one which I merged it with the headers using:cat header.txt calls.txt > callss.txt
    – zara
    Commented Sep 30, 2015 at 19:39
  • Would you accept a couple of Perl lines? (Perl is available on most dists)
    – Déjà vu
    Commented Sep 30, 2015 at 20:13
  • @ ring0 , yes I appreciate that!
    – zara
    Commented Sep 30, 2015 at 20:14

4 Answers 4

4
$ awk '
    NR == 1 {
        for (i=1; i<=NF; i++) {
            output[i] = "output" $i ".txt"
            files[output[i]] = 1
        }
        next
    }
    {
        for (i=1; i<=NF; i++)  printf "%s", $i > output[i]
        for (file in files)    print ""        > file
    }
' input.filename

$ for f in output*.txt; do echo $f; cat $f; done
output1.txt
02202020
02101011
02101011
output2.txt
2022002
1022002
1022002
output3.txt
220111
220000
220000
output30.txt
00202
00202
00202
output4.txt
2020002
2020012
2020012

Note that your header line has 32 fields and the other lines have 33. That needed to be fixed first.

3
  • this gives me error all the time :( where should I put y input file name? how should I run this?
    – zara
    Commented Sep 30, 2015 at 21:25
  • i get this error:awk: cmd. line:8: (FILENAME=file FNR=2) fatal: expression for `>' redirection has null string value
    – zara
    Commented Sep 30, 2015 at 21:55
  • I mentioned that in my answer: you have to ensure that the first line has the same number of fields as all the other lines. You get to the 33rd field on line 2, but there's no value for output[33] Commented Oct 1, 2015 at 13:18
3

Perl script.

Set the file name in $in in place of genome.txt or give the name as argument.

Name the script counter.pl and give it executable rights, and run it as ./counter.pl

chmod 755 counter.pl
./counter.pl

or alternatively

chmod 755 counter.pl
./counter.pl genome.txt

counter.pl:

#!/usr/bin/perl

use strict;
use warnings;

my $in = $ARGV[0] || 'genome.txt'; # input file name

open (my $F, '<', $in) or die "Cannot open input file $!";
my $n = 0;
my %fd = ();
my @fd = ();

while (<$F>) {
        # trim
        s/^\s+//;
        s/\s+$//;
        next if (!$_); # Skip empty lines
        my @x = split(/\s+/, $_);
        # 1st line, open files
        if ( ! $n++)  {
           my $fd = 0;
           for (@x) {
              open ($fd{$_}, '>', "output$_.txt") 
                or die ("Cannot open file $!")
                  if (!exists($fd{$_}));
              $fd[$fd++] = $_;
           }
        }
        else { # Write data
           die ("Should have " . ($#fd+1) . " entries on line $n")
             if ($#x != $#fd);
           for (0 .. $#x) {
              print {$fd{$fd[$_]}} ($x[$_]);
           }
           print {$fd{$_}} ("\n") for (keys %fd);
        }
}

close $fd{$_} for (keys %fd);
close $F;
# the end

Fixed the numbers of words per line (was sometimes 32, sometimes 33 in the example).

This version can accommodate any variation of columns, but all lines have to have the same numbers of words. An error will occur (the die lines) if the number of words is different, or if it cannot open files.

Just adjust the file name ($in).

Input file: (removed the extra 0 near the end)

1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 3 3 3 3 3 3 4 4 4 4 4 4 4 30 30 30 30
0 2 2 0 2 0 2 0 2 0 2 2 0 0 2 2 2 0 1 1 1 2 0 2 0 0 0 2 0 2 0 2
0 2 1 0 1 0 1 1 1 0 2 2 0 0 2 2 2 0 0 0 0 2 0 2 0 0 1 2 0 2 0 2
0 2 1 0 1 0 1 1 1 0 2 2 0 0 2 2 2 0 0 0 0 2 0 2 0 0 1 2 0 2 0 2

output1.txt

02202020
02101011
02101011

output2.txt

2022002
1022002
1022002

output30.txt

0202
0202
0202

output3.txt

220111
220000
220000

output4.txt

2020002
2020012
2020012
6
  • how should I run this? where I should put my input file name? should I save this in a .pl format?
    – zara
    Commented Sep 30, 2015 at 21:20
  • all lines have the same amount of charactors except the header line(first line) has a lot more character. is that a problem?
    – zara
    Commented Sep 30, 2015 at 21:29
  • Number of characters doesn't matter, number of words does. E.g. "1 2 3 / 456 67 99" fits (3 words 1st line then 3 words below), but "1 2 3 / 4 5 6 7" doesn't (3 words then 4 words). If this is not what you want the script can easily be changed.
    – Déjà vu
    Commented Sep 30, 2015 at 21:31
  • my file name is genome.txt should I write genome.txt instead of in or instead of x.in ?
    – zara
    Commented Sep 30, 2015 at 21:38
  • ok I did but I got this error:./counter.pl Missing right curly or square bracket at ./counter.pl line 36, at end of line syntax error at ./counter.pl line 36, at EOF Execution of ./counter.pl aborted due to compilation errors. do you know whay? :(
    – zara
    Commented Sep 30, 2015 at 21:42
2

OK, also for fun - a pure Bash version (as requested) that leans heavily on the builtin read to send words to arrays and save them to file. The files are nicely formatted as output001.txt .... output030.txt. Used a data file as modified by @ringO for testing purposes. Not tested but on very large files it could be more time and resource efficient than others.

Data:

1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 3 3 3 3 3 3 4 4 4 4 4 4 4 30 30 30 30
0 2 2 0 2 0 2 0 2 0 2 2 0 0 2 2 2 0 1 1 1 2 0 2 0 0 0 2 0 2 0 2
0 2 1 0 1 0 1 1 1 0 2 2 0 0 2 2 2 0 0 0 0 2 0 2 0 0 1 2 0 2 0 2
0 2 1 0 1 0 1 1 1 0 2 2 0 0 2 2 2 0 0 0 0 2 0 2 0 0 1 2 0 2 0 2

Source:

#!/usr/bin/env bash

# genome : to sort genome data sets according to patterns of the first (header)
# line of the file.  Data must be space delimited.  No dependencies.
#
# Usage:
#
#                      ./genome "data.txt" 

# global arrays
sc=(  )             # array of set element counts
sn=(  )             # array of set id numbers

# output_file "set id"

# change the output pattern and digit output width as required - default
# pattern is output.txt and digit width of three : output000.txt
output_file(){
    # format concept: pattern000.txt
    local op='output.txt'     # output pattern
    local ow=3                # output width: 3 => 000
    printf "%s%0${ow}d.%s" "${op%%.*}" "$1" "${op##*.}"
}

# define_sets "input.txt"

# identify sets - get elements count and sets id numbers from file
# header.
define_sets(){
    # declare and initialize
    local a an b c n
    read -r c < "$1"
    read -r a b <<< "$c"
    n=0; sn=( $a )

    # recurse header, identify sets
    until [[ -z $b ]]
    do
        n=$((n+1))
        an=$a
        read -r a b <<< "$b"
        [[ $an == $a ]] || { sn+=( $a ); sc+=( $n ); n=0; }
    done
    n=$((n+1))
    sc+=( $n )
}

# reset_files

# optional function, clears file data, otherwise data is appended to existing
# output files.
reset_files(){
    for s in ${sn[@]}
    do
        > "$(output_file "$s")"
    done
}

# extract_data "input.txt"

# use defined sets to extract data from the input file and send it to required
# output files. Uses nested 'while read' to bypass file header as data is saved.
extract_data(){
    local a c n s fn da=( )
    while read -a da
    do
        while read -a da
        do
            a=0 n=0
            for s in ${sc[@]}
            do
                c="$(echo "${da[@]:$a:$s}")" # words => string
                echo "${c// /}" >> "$(output_file "${sn[$n]}")"  # save
                n=$((n+1))
                a=$((a+s))
            done
        done
    done < "$1"
}

define_sets "$1"    # get data set structure from header
reset_files         # optional, clears and resets files
extract_data "$1"   # get data from input file and save

# end file

Data Output:

$ cat output001.txt 
02202020
02101011
02101011

$ cat output002.txt 
2022002
1022002
1022002

$ cat output003.txt 
220111
220000
220000

$ cat output004.txt 
2020002
2020012
2020012

$ cat output030.txt 
0202
0202
0202
5
  • @zara - this uses a couple of tricks for better efficiency - it recurses without a function call and the nested while read avoids unnecessary truth tests. You may find that it is faster than Perl and Awk (although this is speculation of course). But at any rate you will not need any dependencies.
    – AsymLabs
    Commented Oct 5, 2015 at 19:49
  • can I ask another question from you?
    – zara
    Commented Oct 5, 2015 at 20:24
  • @zara you can, can't guarantee it can be answered...
    – AsymLabs
    Commented Oct 5, 2015 at 20:58
  • if I want to do vice versa , having several files and combine them to look like the original on, then how may I do it?
    – zara
    Commented Oct 5, 2015 at 21:02
  • @zara it should be possible to use nested while read -r line; do ... done; and then echo the lines to a single file in the order required - there may be a more efficient way but not having seen the problem it is difficult to say more...
    – AsymLabs
    Commented Oct 5, 2015 at 21:41
1

Just for fun, some other solution:

awk '{ for (i=1; i<=NF;i++){
          if (NR==1) { file[i]=$i }
          if (NR!=1) { f="output" file[i]   ".txt";
                       g="output" file[i+1] ".txt";
                       printf("%s%s",$i,f==g?OFS:ORS)>>f;
                       close(f);
                      }
          }
      }' file

If you need fields that are not separated, change ?OFS: to ?"":.

The default file that receive un-paired values is output.txt. That file will receive values if the number of columns on the first line does not match the next lines processed. If all is correct, it should be empty. If it exists after the script has run, there is a problem somewhere.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .