3

I am trying to split a dozen 100MB+ csv files into managable smaller files for a curl post.

I have managed to do it but with a lot of temporary files and IO. It's taking an eternity.

I am hoping someone can show me a way to do this much more effectively; preferably with little to no disk IO

#!/bin/sh

for csv in $(ls *.csv); do
    tail -n +2 $csv | split -a 5 -l - $RANDOM.split.
done

# chose a file randomly to fetch the header from   

header=$(ls *.csv |sort -R |tail -1 | cut -d',' -f1)

mkdir split

for x in $(/usr/bin/find . -maxdepth 1 -type f -name '*.split.*'); do
    echo Processing $x
    cat header $x >> split/$x
    rm -f $x
done

The above script may not entirely work. I basically got it working through a combination of these commands.

I decided to make the curl POST another step entirely in the case of upload failure; I didn't want to lose the data if it were all posted. But, if, say, on error from curl the data could be put into a redo folder then that can work.

#!/bin/sh

# working on a progress indicator as a percentage. Never finished.
count=$(ls -1 | wc -l 2> /dev/null | cut -d' ' -f1)

for file in $(/usr/bin/find . -maxdepth 1 -type f); do
    echo Processing $file
    curl -XPOST --data-binary @$file -H "Content-Type: text/cms+csv" $1
done

closed as off-topic by Anthon, Michael Homer, jimmij, jasonwryan, bsd Feb 11 '15 at 10:04

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question has been posted on multiple sites. Cross-posting is strongly discouraged; see the help center and community FAQ for more information." – Anthon, Michael Homer, jimmij, jasonwryan, bsd

  • This question has been cross-posted at SO: stackoverflow.com/questions/26708081/… – John1024 Nov 3 '14 at 5:32
  • this really looks like an XYProblem: what do you need to do? are you sure you need to do it via multiple http post? if you have access to the machine, you can scp, rsync, etc. – Olivier Dulac Nov 3 '14 at 16:44
  • 1
    $(ls *.csv) is redundant; just use for csv in *.csv. – chepner Nov 3 '14 at 19:33
  • @OlivierDulac -- I actually have to POST it because it has to be processed by a webservice. It's not just a straight upload. – Christian Bongiorno Nov 4 '14 at 22:31
4

An entirely different approach would be to use GNU parallel, and use its --header and --pipe options:

cat input.csv | parallel --header : --pipe -N 10 'cat > output{#}.csv'

This will get you 11 lines in each of the files (the header line plus the ten from -n 10, except in the last file written that way).

  • This looks interesting. I will have to give this a try. – Christian Bongiorno Nov 4 '14 at 22:29

Not the answer you're looking for? Browse other questions tagged or ask your own question.