2

Consider the below input file. Input file:

C1|C2|C3|C4|C5|C6
0|1|2|3|0-1-2-3|4
0|2|2|4|0-1-2-3|5
0|1|2|3|1-3-2-4|4
0|1|2|3|1-1-3-4|4
0|1|2|3|1-1-3-4|5
0|1|2|3|4-5-2-6|4
0|1|2|3|4-5-2-6|6
0|4|5|3|7-4-2-4|4
0|1|2|3|7-4-2-5|4
0|1|2|3|7-4-2-5|5
0|1|2|3|7-4-2-5|6
0|1|2|3|7-5-2-6|5

Based on field 5, for example 0-1-2-3 in the first record, output split files are expected as below Split file1:

C1|C2|C3|C4|C5|C6
0|1|2|3|0-1-2-3|4
0|2|2|4|0-1-2-3|5
0|1|2|3|1-3-2-4|4
0|1|2|3|1-1-3-4|4
0|1|2|3|1-1-3-4|5

Split file 2:

C1|C2|C3|C4|C5|C6
0|1|2|3|4-5-2-6|4
0|1|2|3|4-5-2-6|6
0|4|5|3|7-4-2-4|4
0|1|2|3|7-4-2-5|4
0|1|2|3|7-4-2-5|5
0|1|2|3|7-4-2-5|6

Split file 3:

C1|C2|C3|C4|C5|C6
0|1|2|3|7-5-2-6|5

Based on column 5, for every 3 unique values in the column the file should be split and should even have all the lines where the value of column 5 is repeated. Can someone help me with this?

4

A job for awk. Something like:

awk -F'|' -v fileformat="/abc/output/file_%04d.txt" -v max=3 -v field=5 '
  NR == 1 {header = $0; next}
  ! ($field in seen) {
    seen[$field]
    if (++n % max == 1) {
      close(out)
      out = sprintf(fileformat, ++f)
      print header > out
    }
  }
  {print > out}' < /abc/input/a.txt
  • Hi @StéphaneChazelas.. Thanks for answering. I tried using the logic, i am not able to view the output files. Can you please explain the logic once and also help me with a way to place the split files in a specific path? – unixlearner Aug 22 '17 at 13:56
  • @unixlearner. The output files are file_<x>.txt. See fileformat (the one you want to change if you want the output in different files). That should be pretty straightforward as long as you know the basics of awk. Explaining would just be paraphrasing the code. – Stéphane Chazelas Aug 22 '17 at 13:59
  • The script is working fine. But i am not able to find the input files generated from the script. Can you help me with this: If the your file is the path as "/abc/input/a.txt", i want the output split files generated to be placed in "/abc/output" folder. – unixlearner Aug 22 '17 at 14:19
  • @unixlearner see edit – Stéphane Chazelas Aug 22 '17 at 14:23
  • 1
    @EdMorton. Good point about close() (though here with that fileformat, there's no chance of the file name to be mistakenly taken as 0). About !seen[$field]++, I'll leave it as it is as it's more legible (IMO) and is generally more efficient and has a smaller memory footprint with at least gawk. – Stéphane Chazelas Aug 22 '17 at 16:29
1

awk solution:

awk -F'|' 'NR==1{ h=$0; f=0; c=1 }NR>1{ 
              a[$5]; if(length(a)>3) { f=0;c++; delete a }; 
              fn="file"c".txt"; if(!f) print h > fn; print > fn; f++ 
           }' file
  • h=$0 - header line

  • f=0 - flag pointing to the moment of printing header line into the next new file

  • c=1 - filename suffix (incremented for each new file)

  • a[$5] - indexing array a with unique values of the 5th field

  • if(length(a)>3) { f=0;c++; delete a } - initiating next new filename (c++) when succeeding 3 unique values. (delete a - delete ll items from array a)

  • fn="file"c".txt" - current filename


Viewing results:

for f in file[0-9]*.txt; do echo "$f"; cat "$f"; echo; done

The output:

file1.txt
C1|C2|C3|C4|C5|C6
0|1|2|3|0-1-2-3|4
0|2|2|4|0-1-2-3|5
0|1|2|3|1-3-2-4|4
0|1|2|3|1-1-3-4|4
0|1|2|3|1-1-3-4|5

file2.txt
C1|C2|C3|C4|C5|C6
0|1|2|3|4-5-2-6|4
0|1|2|3|4-5-2-6|6
0|4|5|3|7-4-2-4|4
0|1|2|3|7-4-2-5|4
0|1|2|3|7-4-2-5|5
0|1|2|3|7-4-2-5|6

file3.txt
C1|C2|C3|C4|C5|C6
0|1|2|3|7-5-2-6|5
  • Note that few implementations of awk support length(array). gawk and busybox awk do. bwk's awk does since 2002 (inspired by GNU awk), so it should work as well on most BSDs nowadays. It works with the mawk maintained by @ThomasDickey, but not the one found on Debian and derivatives. It typically doesn't work on traditional Unices like Solaris. It's not POSIX – Stéphane Chazelas Aug 22 '17 at 12:55
  • @StéphaneChazelas, but not the one found on Debian and derivatives - if Ubuntu is a Debian derivative why it's working on my Ubuntu 16.04 ? – RomanPerekhrest Aug 22 '17 at 13:38
  • mawk 'BEGIN{a[0]=123; print length(a)}' gives me mawk: line 1: illegal reference to array a in Ubuntu 16.04 (mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan). Do you get anything different? – Stéphane Chazelas Aug 22 '17 at 13:57
  • Thanks @RomanPerekhrest for answering. I am having the input file in folder path say "/abc/input", but iam not able to view the output files. Can you please help me with that and also with redirecting the split output files to a particular path say "/abc/output"? – unixlearner Aug 22 '17 at 14:01
  • @StéphaneChazelas, *Do you get anything different? * - of course, I'm calling awk ... not mawk, works fine – RomanPerekhrest Aug 22 '17 at 14:46
1
#!/bin/bash

awk -F '|' '
    function print_to_file(str) {
        print str > "file_"file_num;    
    }
    NR == 1 {header = $0;}
    NR > 1 {
        if(!buf) {
            file_num = 1;
            print_to_file(header);
        }
        if(buf != $5) {
            buf = $5;
            cnt++;
        }
        if(cnt > 3) {
            cnt = 1;
            file_num++;
            print_to_file(header);
        }
        print_to_file($0);
    }
' input.txt

Output:

$ tail -n +1 -- file_* # display content of all files with their filenames

==> file_1 <==
C1|C2|C3|C4|C5|C6
0|1|2|3|0-1-2-3|4
0|2|2|4|0-1-2-3|5
0|1|2|3|1-3-2-4|4
0|1|2|3|1-1-3-4|4
0|1|2|3|1-1-3-4|5

==> file_2 <==
C1|C2|C3|C4|C5|C6
0|1|2|3|4-5-2-6|4
0|1|2|3|4-5-2-6|6
0|4|5|3|7-4-2-4|4
0|1|2|3|7-4-2-5|4
0|1|2|3|7-4-2-5|5
0|1|2|3|7-4-2-5|6

==> file_3 <==
C1|C2|C3|C4|C5|C6
0|1|2|3|7-5-2-6|5
  • 1
    Note that if you never close the files, you run the risk of running out of file descriptors with some awk implementations. – Stéphane Chazelas Aug 24 '17 at 10:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.