Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am writing a php script to parse a large text file do to database inserts from it. However on my host, the file is too large, and I hit the memory limit for php.

The file has about 16,000 lines; I want to split it up into four separate files (at first) to see if I can load those.

The first part I can get with head -4000 file.txt. The middle sections are slightly trickier -- I was thinking about piping tail output into head ( tail -4001 file.txt|head -4000 > section2.txt ), but is there another/better way?

Edit actually my logic is messed up -- for section two, I would need to so something like tail -12001 file.txt|head - 4000, and then lower the tail argument for the next sections. I'm getting mixed up already! :P

share|improve this question
add comment

2 Answers

up vote 7 down vote accepted

If you want not to get messed up but still do it using tail and head, there is a useful way of invoking tail using a line-count from the beginning, not the end:

tail -n +4001 yourfile | head -4000

... But a better, automatic tool made just for splitting files is called... split! It's also a part of GNU coreutils, so any normal Linux system should have it. Here's how you can use it:

split -l 4000 yourInputFile thePrefixForOutputFiles

(See man split if in doubt.)

share|improve this answer
    
I would never doubt you :) –  user394 Oct 14 '11 at 17:27
add comment

Combining head and tail as you did will work, but for this I would use sed

sed -n '1,4000p' input_file # print lines 1-4000 of input_file

This lets you solve your problem with a quick shell function

chunk_it(){
    step=4
    start=1
    end=$step
    for n in {1..4} ; do
        sed -n "${start},${end}p" "$1" > "$1".$start-$end
        let start+=$step
        let end+=$step
    done
}

chunk_it your_file

Now you have your_file.1-4000 and yuor_file.4001-8000 and so on.

Note: requires bash

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.