32

I am writing a PHP script to parse a large text file to do database inserts from it. However on my host, the file is too large, and I hit the memory limit for PHP.

The file has about 16,000 lines; I want to split it up into four separate files (at first) to see if I can load those.

The first part I can get with head -4000 file.txt. The middle sections are slightly trickier -- I was thinking about piping tail output into head ( tail -4001 file.txt | head -4000 > section2.txt ), but is there another/better way?

Actually my logic is messed up -- for section two, I would need to so something like tail -12001 file.txt | head - 4000, and then lower the tail argument for the next sections. I'm getting mixed up already! :P

3 Answers 3

43

If you want not to get messed up but still do it using tail and head, there is a useful way of invoking tail using a line-count from the beginning, not the end:

tail -n +4001 yourfile | head -4000

... But a better, automatic tool made just for splitting files is called... split! It's also a part of GNU coreutils, so any normal Linux system should have it. Here's how you can use it:

split -l 4000 yourInputFile thePrefixForOutputFiles

(See man split if in doubt.)

1
  • Given that split will split the file into three chunks, only one of which one cares about, surely the sed answer is preferable.
    – cbmanica
    Oct 1, 2021 at 16:17
30

Combining head and tail as you did will work, but for this I would use sed

sed -n '1,4000p' input_file # print lines 1-4000 of input_file

This lets you solve your problem with a quick shell function

chunk_it(){
    step=4
    start=1
    end=$step
    for n in {1..4} ; do
        sed -n "${start},${end}p" "$1" > "$1".$start-$end
        let start+=$step
        let end+=$step
    done
}

chunk_it your_file

Now you have your_file.1-4000 and yuor_file.4001-8000 and so on.

Note: requires bash

5
  • 3
    I like the sed way.
    – fanchyna
    Feb 20, 2016 at 15:38
  • This doesn't work for me because sed doesn't exit. It prints out the lines I want to stdout, but I have to ctrl-c out, and as a result, I can't redirect it to a file. Any suggestion to make it usable?
    – Brent212
    Jun 30, 2017 at 18:41
  • Figured it out! "sed -n '<start_line>,<end_line>w <output_file>' <input_file>" works for me.
    – Brent212
    Jun 30, 2017 at 18:54
  • @Brent212 Another option to note is that you can also pipe it into less or redirect the output to a file.
    – Kyle s
    Dec 19, 2018 at 19:54
  • On GNU sed, redirecting can work. The problem is that sed buffers its output by default. Use -u to disable buffering and redirecting should work. Note that the -u option is not available on all versions of sed.
    – rinogo
    Jan 18, 2022 at 16:15
0

You can also use bat like

bat -r 4001:8000 input-file.txt >output-file-1.txt

The benefit is that you can omit the output redirection to get an idea what’s being written:

project specification for Bat

Note: It’s probably overkill to install bat just for this feature; this answer is useful if you already have it installed for some reason.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .