2

Consider the below input file. Input file:

C1|C2|C3|C4|C5|C6
0|1|2|3|0-1-2-3|4
0|2|2|4|0-1-2-3|5
0|1|2|3|1-3-2-4|4
0|1|2|3|1-1-3-4|4
0|1|2|3|1-1-3-4|5
0|1|2|3|4-5-2-6|4
0|1|2|3|4-5-2-6|6
0|4|5|3|7-4-2-4|4
0|1|2|3|7-4-2-5|4
0|1|2|3|7-4-2-5|5
0|1|2|3|7-4-2-5|6
0|1|2|3|7-5-2-6|5

Based on field 5, for example 0-1-2-3 in the first record, output split files are expected as below Split file1:

C1|C2|C3|C4|C5|C6
0|1|2|3|0-1-2-3|4
0|2|2|4|0-1-2-3|5
0|1|2|3|1-3-2-4|4
0|1|2|3|1-1-3-4|4
0|1|2|3|1-1-3-4|5

Split file 2:

C1|C2|C3|C4|C5|C6
0|1|2|3|4-5-2-6|4
0|1|2|3|4-5-2-6|6
0|4|5|3|7-4-2-4|4
0|1|2|3|7-4-2-5|4
0|1|2|3|7-4-2-5|5
0|1|2|3|7-4-2-5|6

Split file 3:

C1|C2|C3|C4|C5|C6
0|1|2|3|7-5-2-6|5

Based on column 5, for every 3 unique values in the column the file should be split and should even have all the lines where the value of column 5 is repeated. Can someone help me with this?

3 Answers 3

4

A job for awk. Something like:

awk -F'|' -v fileformat="/abc/output/file_%04d.txt" -v max=3 -v field=5 '
  NR == 1 {header = $0; next}
  ! ($field in seen) {
    seen[$field]
    if (++n % max == 1) {
      close(out)
      out = sprintf(fileformat, ++f)
      print header > out
    }
  }
  {print > out}' < /abc/input/a.txt
7
  • Hi @StéphaneChazelas.. Thanks for answering. I tried using the logic, i am not able to view the output files. Can you please explain the logic once and also help me with a way to place the split files in a specific path? Aug 22, 2017 at 13:56
  • @unixlearner. The output files are file_<x>.txt. See fileformat (the one you want to change if you want the output in different files). That should be pretty straightforward as long as you know the basics of awk. Explaining would just be paraphrasing the code. Aug 22, 2017 at 13:59
  • The script is working fine. But i am not able to find the input files generated from the script. Can you help me with this: If the your file is the path as "/abc/input/a.txt", i want the output split files generated to be placed in "/abc/output" folder. Aug 22, 2017 at 14:19
  • @unixlearner see edit Aug 22, 2017 at 14:23
  • 1
    @EdMorton. Good point about close() (though here with that fileformat, there's no chance of the file name to be mistakenly taken as 0). About !seen[$field]++, I'll leave it as it is as it's more legible (IMO) and is generally more efficient and has a smaller memory footprint with at least gawk. Aug 22, 2017 at 16:29
1

awk solution:

awk -F'|' 'NR==1{ h=$0; f=0; c=1 }NR>1{ 
              a[$5]; if(length(a)>3) { f=0;c++; delete a }; 
              fn="file"c".txt"; if(!f) print h > fn; print > fn; f++ 
           }' file
  • h=$0 - header line

  • f=0 - flag pointing to the moment of printing header line into the next new file

  • c=1 - filename suffix (incremented for each new file)

  • a[$5] - indexing array a with unique values of the 5th field

  • if(length(a)>3) { f=0;c++; delete a } - initiating next new filename (c++) when succeeding 3 unique values. (delete a - delete ll items from array a)

  • fn="file"c".txt" - current filename


Viewing results:

for f in file[0-9]*.txt; do echo "$f"; cat "$f"; echo; done

The output:

file1.txt
C1|C2|C3|C4|C5|C6
0|1|2|3|0-1-2-3|4
0|2|2|4|0-1-2-3|5
0|1|2|3|1-3-2-4|4
0|1|2|3|1-1-3-4|4
0|1|2|3|1-1-3-4|5

file2.txt
C1|C2|C3|C4|C5|C6
0|1|2|3|4-5-2-6|4
0|1|2|3|4-5-2-6|6
0|4|5|3|7-4-2-4|4
0|1|2|3|7-4-2-5|4
0|1|2|3|7-4-2-5|5
0|1|2|3|7-4-2-5|6

file3.txt
C1|C2|C3|C4|C5|C6
0|1|2|3|7-5-2-6|5
6
  • Note that few implementations of awk support length(array). gawk and busybox awk do. bwk's awk does since 2002 (inspired by GNU awk), so it should work as well on most BSDs nowadays. It works with the mawk maintained by @ThomasDickey, but not the one found on Debian and derivatives. It typically doesn't work on traditional Unices like Solaris. It's not POSIX Aug 22, 2017 at 12:55
  • @StéphaneChazelas, but not the one found on Debian and derivatives - if Ubuntu is a Debian derivative why it's working on my Ubuntu 16.04 ? Aug 22, 2017 at 13:38
  • mawk 'BEGIN{a[0]=123; print length(a)}' gives me mawk: line 1: illegal reference to array a in Ubuntu 16.04 (mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan). Do you get anything different? Aug 22, 2017 at 13:57
  • Thanks @RomanPerekhrest for answering. I am having the input file in folder path say "/abc/input", but iam not able to view the output files. Can you please help me with that and also with redirecting the split output files to a particular path say "/abc/output"? Aug 22, 2017 at 14:01
  • @StéphaneChazelas, *Do you get anything different? * - of course, I'm calling awk ... not mawk, works fine Aug 22, 2017 at 14:46
1
#!/bin/bash

awk -F '|' '
    function print_to_file(str) {
        print str > "file_"file_num;    
    }
    NR == 1 {header = $0;}
    NR > 1 {
        if(!buf) {
            file_num = 1;
            print_to_file(header);
        }
        if(buf != $5) {
            buf = $5;
            cnt++;
        }
        if(cnt > 3) {
            cnt = 1;
            file_num++;
            print_to_file(header);
        }
        print_to_file($0);
    }
' input.txt

Output:

$ tail -n +1 -- file_* # display content of all files with their filenames

==> file_1 <==
C1|C2|C3|C4|C5|C6
0|1|2|3|0-1-2-3|4
0|2|2|4|0-1-2-3|5
0|1|2|3|1-3-2-4|4
0|1|2|3|1-1-3-4|4
0|1|2|3|1-1-3-4|5

==> file_2 <==
C1|C2|C3|C4|C5|C6
0|1|2|3|4-5-2-6|4
0|1|2|3|4-5-2-6|6
0|4|5|3|7-4-2-4|4
0|1|2|3|7-4-2-5|4
0|1|2|3|7-4-2-5|5
0|1|2|3|7-4-2-5|6

==> file_3 <==
C1|C2|C3|C4|C5|C6
0|1|2|3|7-5-2-6|5
1
  • 1
    Note that if you never close the files, you run the risk of running out of file descriptors with some awk implementations. Aug 24, 2017 at 10:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.