Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm loading a pretty gigantic file to a postgresql database. To do this I first use split in the file to get smaller files (30Gb each) and then I load each smaller file to the database using GNU Parallel and psql copy.

The problem is that it takes about 7 hours to split the file, and then it starts to load a file per core. What I need is a way to tell split to print the file name to std output each time it finishes writing a file so I can pipe it to Parallel and it starts loading the files at the time split finish writing it. Something like this:

split -l 50000000 2011.psv carga/2011_ | parallel ./carga_postgres.sh {}

I have read the split man pages and I can't find anything. Is there a way to do this with split or any other tool?

share|improve this question

3 Answers 3

Use --pipe:

cat 2011.psv | parallel --pipe -l 50000000 ./carga_postgres.sh

It requires ./carga_postgres.sh to read from stdin and not from a file, and is slow for GNU Parallel version < 20130222.

If you do not need exactly 50000000 lines the --block is faster:

cat 2011.psv | parallel --pipe --block 500M ./carga_postgres.sh

This will pass chunks of around 500MB split on \n.

I do not know what ./carga_postgres.sh contains, but my guess is it contains psql with username password. In that case you might want to use GNU SQL (which is part of GNU Parallel):

cat 2011.psv | parallel --pipe --block 500M sql pg://user:pass@host/db

The major benefit is that you do not need to save temporary files, but can keep all in memory/pipes.

If ./carga_postgres.sh cannot read from stdin, but must read from a file, you can save it to a file:

cat 2011.psv | parallel --pipe --block 500M "cat > {#}; ./carga_postgres.sh {#}"

Large jobs often fail half way through. GNU Parallel can help you by re-running the failed jobs:

cat 2011.psv | parallel --pipe --block 500M --joblog my_log --resume-failed "cat > {#}; ./carga_postgres.sh {#}"

If this fails then you can re-run the above. It will skip the blocks that are already processed successfully.

share|improve this answer

An alternative to making split print the file names is to detect when the files are ready. On Linux, you can use the inotify facility, and specifically the inotifywait utility.

inotifywait -m -q -e close_write --format %f carga | parallel ./carga_postgres.sh &
split -l 50000000 2011.psv carga/2011_

You'll need to kill inotifywait manually. Killing it automatically is a little hard because there's a potential race condition: if you kill it as soon as split finishes, it may have received events that it hasn't reported yet. To make sure that all events are reported, count the matching files.

{
  sh -c 'echo $PPID' >inotifywait.pid
  exec inotifywait -m -q -e close_write --format %f carga
} | tee last.file \
  | parallel ./carga_postgres.sh &
split -l 50000000 2011.psv carga/2011_
(
  set carga/2011_??; eval "last_file=\${$#}"
  while ! grep -qxF "$last_file" last.file; do sleep 1; done
)
kill $(cat inotifywait.pid)
share|improve this answer

I found the answers posted here to be way to complex so I asked on Stack Overflow and I got this answer:

If you use GNU split, you can do this with the --filter option

‘--filter=command’
With this option, rather than simply writing to each output file, write through a pipe to the specified shell command for each output file. command should use the $FILE environment variable, which is set to a different output file name for each invocation of the command.

You can create a shell script, which creates a file and start carga_postgres.sh at the end in the background

#! /bin/sh

cat >$FILE
./carga_postgres.sh $FILE &

and use that script as the filter

split -l 50000000 --filter=./filter.sh 2011.psv
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.