I have a large CSV file and want to split it into small chunks. I know that I can split the CSV file using

split -l 1000000 file.csv new

which results in a part having lines of 1000000.But the problem is it splits but also the original one exists. Since I do not have much space in my disk is there any way to split the CSV without keeping the original one? Any help is appreciated. Thanks in advance.

3 Answers 3


One way to do it, assuming GNU stat(1) and GNU truncate(1):

#! /bin/sh


size=$( wc -l "$1" | awk '{print $1}' )
tail=$(( size % lines ))
count=$(( size / lines ))

if [ $tail -ne 0 ]; then
    let count++

while [ $count -gt 0 ]; do
    start=$(( (count - 1) * lines + 1 ))
    fn=$( printf '%s_%05d' "$1" $count )
    sed -n $start,\$p <"$1" >"$fn"
    size_last=$( stat -c %s "$fn" )
    truncate -s -$size_last "$1"
    let count--

Without GNU coreutils, the same thing could be done in Perl.

The original file gets lost in the process, so it's probably wise to test the above against a few smaller files first, with lines set to, say, 100.


If the memory on your system is large enough to hold the whole csv file, you could try this very dangerous way of putting the file into a temporary file system (i.e. a virtual hard drive in your RAM) and then start splitting it from there onto the hard drive.

Note that when the PC is powered down while the file is in this tmpfs the data will be lost! This is very, very prone to making you unhappy.

Usually /dev/shm should be present already, double check if mount | grep shm lists a tmpfs as mounted on /dev/shm, then:

mv file /dev/shm
split -l 1000000 /dev/shm/file /path/to/split/directory/

I don't know how much excess memory you'll need apart from the file size, as I am not familiar with split's memory usage, but it would be at least the one million lines you are splitting, I assume.

Again be prepared for data loss on a simple power outage or anything unexpected.

PS: Maybe you have a USB flash drive at hand to use as your temporary file system - less dangerous, but slower.


If it's OK to reverse the order of the CSV fields, you could try something along the following lines:

SIZ=$(stat -c %s input)
tac input |\
  while read -r LINE
    SIZ=$(( (SIZ-ADJ-1) ))
    truncate -s $SIZ input
    echo "$LINE"
) |\
split -l 10 - output

It will also take longer to run than a plain split command, but maybe not very much longer.

It does conserve disk space, allowing it to be run even if the starting file takes up nearly 100% of available disk space.

You will want to change the file names and increase the -l 10 argument to split. I posted it the way I tested it, on a file with far fewer than a million lines.

  • That's great, until you realize that tac, depending on implementation, might also create a temp file of a size comparable to the initial one. Why do you bother with tac to begin with, all you need is the start of each 1 mil. lines anyway. Nov 29, 2016 at 5:30
  • @SatoKatsura There is no conceivable reason for tac to create a temp file of a size comparable to the initial one. The intermediate storage needed to implement tac is the length of the longest line only. Nov 29, 2016 at 5:38
  • How about input not being seek-able? :) Nov 29, 2016 at 5:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .