I have large file. It is made of parts of 40 lines each. There are two types of parts and they alternate. The two types of parts should be numbered independently. So the first part should be X_0001, the second part should be Y_0001, then X_0002, Y_0002, etc.

I used this command but it can only split into pieces with the same prefix:

 split -d -l 40 -a 4  inputfile X_ 
  • So each chunk of 80 lines has to be sorted and then the first 40 lines go to X_??? and the last 40 lines go to Y_???, right ? – don_crissti Jul 28 '16 at 23:01
  • exactly this what I want – Mohsen El-Tahawy Jul 28 '16 at 23:03

With gnu split you could use the --filter option:

split --numeric-suffixes=0001 -l 80 -a 4 \
--filter='sed -n -e "1,40w $FILE" -e "41,80w ${FILE/X/Y}"' infile X_

This will split the file into 80-lines pieces, piping the content of each piece to sed which writes the first 40 lines to $FILE (the piece name, in this case split replaces it with X_???? - see man split) and the rest to ${FILE/X/Y} which is the same name but with X replaced by Y.

Since the requirement has changed and you only need to split into pieces with alternating names you could also use awk:

awk 'BEGIN{c=1;p="X"}
{close(fn);fn=sprintf("%s_%04d", p, c);print >> fn} 
NR%40==0{p="Y"}NR%80==0{p="X";c++}' file1

This sets the piece name based on two variables, prefix and counter. Each 40 lines, the prefix changes to Y, each 80 lines the prefix changes to X and the counter is incremented.

  • it works when I replaced --numeric-suffixes=000' by _ -d_; but this sort the lines before splitting and this is unwanted – Mohsen El-Tahawy Jul 29 '16 at 0:09
  • sorry; maybe I didn't read your comment well – Mohsen El-Tahawy Jul 29 '16 at 0:14

One way is to use split and rename the files afterwards.

But the simplest is probably to call awk. You can use the > redirection operator to write to a file instead of standard output. The variable NR contains the current line number.

Awk's redirection automatically takes care of opening files. You should close files explicitly if you use a lot of different ones, otherwise you might run into a limit on open files.

awk '
  (NR-1) % 40 == 0 { close(out); out = sprintf("%s_%04d", (NR % 80 == 1 ? "X" : "Y"), NR/80+1); }
  { print >out }
' inputfile
  • this split the file only to two parts, but the file has 4000 lines so the results should be 50 files of X_{00..50} and 50 files of Y_{00..50} – Mohsen El-Tahawy Jul 28 '16 at 23:31
  • @MohsenEl-Tahawy Oh, I see. Your question was worded strangely but I hadn't understood that. I've updated my answer to match this. – Gilles Jul 29 '16 at 6:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.