0

I have a large CSV file and want to split it into small chunks. I know that I can split the CSV file using

split -l 1000000 file.csv new

which results in a part having lines of 1000000.But the problem is it splits but also the original one exists. Since I do not have much space in my disk is there any way to split the CSV without keeping the original one? Any help is appreciated. Thanks in advance.

1

One way to do it, assuming GNU stat(1) and GNU truncate(1):

#! /bin/sh

lines=1000000

size=$( wc -l "$1" | awk '{print $1}' )
tail=$(( size % lines ))
count=$(( size / lines ))

if [ $tail -ne 0 ]; then
    let count++
fi           

while [ $count -gt 0 ]; do
    start=$(( (count - 1) * lines + 1 ))
    fn=$( printf '%s_%05d' "$1" $count )
    sed -n $start,\$p <"$1" >"$fn"
    size_last=$( stat -c %s "$fn" )
    truncate -s -$size_last "$1"
    let count--
done

Without GNU coreutils, the same thing could be done in Perl.

The original file gets lost in the process, so it's probably wise to test the above against a few smaller files first, with lines set to, say, 100.

1

If the memory on your system is large enough to hold the whole csv file, you could try this very dangerous way of putting the file into a temporary file system (i.e. a virtual hard drive in your RAM) and then start splitting it from there onto the hard drive.

Note that when the PC is powered down while the file is in this tmpfs the data will be lost! This is very, very prone to making you unhappy.

Usually /dev/shm should be present already, double check if mount | grep shm lists a tmpfs as mounted on /dev/shm, then:

mv file /dev/shm
split -l 1000000 /dev/shm/file /path/to/split/directory/

I don't know how much excess memory you'll need apart from the file size, as I am not familiar with split's memory usage, but it would be at least the one million lines you are splitting, I assume.

Again be prepared for data loss on a simple power outage or anything unexpected.

PS: Maybe you have a USB flash drive at hand to use as your temporary file system - less dangerous, but slower.

-1

If it's OK to reverse the order of the CSV fields, you could try something along the following lines:

SIZ=$(stat -c %s input)
tac input |\
(
  IFS=""
  while read -r LINE
  do
    ADJ=${#LINE}
    SIZ=$(( (SIZ-ADJ-1) ))
    truncate -s $SIZ input
    echo "$LINE"
  done
) |\
split -l 10 - output

It will also take longer to run than a plain split command, but maybe not very much longer.

It does conserve disk space, allowing it to be run even if the starting file takes up nearly 100% of available disk space.

You will want to change the file names and increase the -l 10 argument to split. I posted it the way I tested it, on a file with far fewer than a million lines.

  • That's great, until you realize that tac, depending on implementation, might also create a temp file of a size comparable to the initial one. Why do you bother with tac to begin with, all you need is the start of each 1 mil. lines anyway. – Satō Katsura Nov 29 '16 at 5:30
  • @SatoKatsura There is no conceivable reason for tac to create a temp file of a size comparable to the initial one. The intermediate storage needed to implement tac is the length of the longest line only. – DepressedDaniel Nov 29 '16 at 5:38
  • How about input not being seek-able? :) – Satō Katsura Nov 29 '16 at 5:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.