10

I am able to use the split command successfully to split large file into multiple smaller files. This is being achieved by following command

split -b 1G $temp_path $final_filepath

But only caveat is that these files many times contain last line which is split across 2 files. Is there any way to avoid that using split or any other command ?

1 Answer 1

9

Yes, don't use the -b parameter. From the split(1) man page:

-b, --bytes=SIZE put SIZE bytes per output file

-C, --line-bytes=SIZE put at most SIZE bytes of lines per output file

-l, --lines=NUMBER put NUMBER lines per output file

By using -b you are telling split to deliniate files at a specific size in bytes (or Kb or MB). If that is the middle of a line, too bad.

Split supports 'number of lines' and a 'max output file size comprised of whole lines'.

Instead, try this:

split -C 1G $temp_path $final_filepath

The -C flag is not available on all versions of split (notably OS X / Darwin). In that case you can use gsplit which is available in the GNU coreutils package on Homebrew and MacPorts.

2
  • 2
    did not read the man page carefully enough.
    – Albatross
    Apr 3, 2017 at 21:48
  • 1
    No worries, happens to all of us. I'm just glad I was working with split earlier today and happened to know the answer off the cuff. :)
    – 0xSheepdog
    Apr 3, 2017 at 21:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.