41

I have a big file and need to split into two files. Suppose in the first file the 1000 lines should be selected and put into another file and delete those lines in the first file.

I tried using split but it is creating multiple chunks.

Improve this question
2
  • Did you check split --help?
    – Braiam
    Oct 21, 2014 at 16:01
  • Yes i have checked it, but is creating multiple files which doesn't need to me.
    – Aravind
    Oct 21, 2014 at 16:02

4 Answers 4

Reset to default
59

The easiest way is probably to use head and tail:

$ head -n 1000 input-file > output1
$ tail -n +1001 input-file > output2

That will put the first 1000 lines from input-file into output1, and all lines from 1001 till the end in output2

Improve this answer
1
  • 1
    Funny thing that it works pretty well with 2GB+ files too!
    – Gergely Lukacsy
    Oct 21, 2020 at 18:12
29

I think that split is you best approach.

Try using the -l xxxx option, where xxxx is the number of lines you want in each file (default is 1000).

You can use the -n yy option if you are more concerned about the amount of files created. Use -n 2 will split your file in only 2 parts, no matter the amount of lines in each file.

You can count the amount of lines in your file with wc -l filename. This is the 'wordcount' command with the lines option.

References

  • man split
  • man wc
Improve this answer
5
  • 1
    This is how to split into a bunch of files with a fixed number of lines, or how to split evenly into a fixed number of files. Is there a way to split into one 1000-line file and one file with everything else? That's what he was asking for; I couldn't find it in the man page
    – Michael Mrozek
    Oct 21, 2014 at 17:05
  • You´re correct Michael. I think I took a simplistic view on the question. You solution is the best one in this case. Another way would be to use the 'sed' command: sed -n 1,1000 originalfile > first_1000_lines. sed '1,1000d' originalfile > remaining_lines.
    – Lucien Raven
    Oct 21, 2014 at 17:17
  • Of course you could do split -l 1000 bigfile && mv xaa piece1 && cat x?? > piece2 && rm x??.
    – G-Man Says 'Reinstate Monica'
    Oct 21, 2014 at 23:40
  • split is what I was looking for
    – Daniel
    Apr 8, 2020 at 20:53
  • split with both -l and -n options doesn't run ('split: cannot split in more than one way'). Question wanted file into 2 parts, but at a specific line: split is the wrong tool for this job. csplit is the correct tool
    – RGD2
    Jun 28, 2021 at 23:40
17

This is a job for csplit:

csplit -s infile 1001

will silently split infile, the first piece xx00 - up to but not including line 1001 and the second piece xx01 - the remaining lines.
You can play with the options if you need different output file names e.g. using -f and specifying a prefix: 

csplit -sf piece. infile 1001

produces two files named piece.00 and piece.01

With a smart head you could also do something like:

{ head -n 1000 > 1st.out; cat > 2nd.out; } < infile
Improve this answer
1
  • 2
    Wow, it really is a job for csplit. Very nice. (I'm just reading through the list of POSIX commands and had enormous trouble wrapping my head around the csplit command's purpose at first. Turns out it's really really simple.) :)
    – Wildcard
    Nov 2, 2016 at 5:38
5

A simple way to do what the question asks for, in one command:

awk '{ if (NR <= 1000) print > "piece1"; else print > "piece2"; }' bigfile

or, for those of you who really hate to type long, intuitively comprehensible commands,

awk '{ print > ((NR <= 1000) ? "piece1" : "piece2"); }' bigfile
Improve this answer
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.