2

I have the below call to db2 database command line tool inside a loop, which runs 100k iterations.

(the output from db2 is 5 rows of 20 chars approx per each call, total of approx 100k calls. The input to SP is prefixed to the output line from the SP and dumped to a log file.)

while read line
do
    db2 -x "call stored_procedure_XYZ($line)" |sed "s/^/$line/" >>log_file.txt
done<$infile

I am trying to make the whole thing run faster by taking out the sed and disk write from inside the loop. Is it advisable to store all the output in a variable, and later modify and write it to the disk?

What would be the length the shell variable can hold?

(bash, aix 6.1)

  • If all you are doing is appending output to a file what is the point of putting it all in a variable first? What else are you doing with the data? – jw013 Dec 10 '14 at 20:11
  • I am thinking writing to disk only once after the loop gets over might improve performance, instead of writing multiple times inside the loop – dbza Dec 10 '14 at 20:21
  • 1
    @mikeserv db2 appears to be some sort of IBM database command line utility. It is highly unlikely that nl does anything close to what it does. – jw013 Dec 11 '14 at 0:53
  • 1
    @dbza Since you have edited your question to show the entire loop, I would recommend moving >>log_file.txt outside the loop, so you have done<infile >>log_file.txt as the last line instead. That way you keep the file open and avoid reopening and reclosing it each iteration. If you want to try anything more complicated though, you should first take measurements to see if disk I/O is really taking enough time to be worth optimizing. You can probably do this by replacing >>log_file.txt with >/dev/null to get rid of disk I/O altogether and see how much of a speed-up you get. – jw013 Dec 11 '14 at 1:22
  • 1
    @jw013 i guess you meant done<infile >log_file.txt and not >> to append – dbza Dec 12 '14 at 1:54
5

Bash variable size is not fixed.It is very likely hold arbitrary amounts of data as long as malloc can find sufficient memory and contiguous address space.Let's assume you stored large large amount of data in your variable.When you try to write data to your file,possibly you will get error something like that

/bin/echo ${LARGE_DATA} >> ${YourFile}    
/bin/echo: Argument list too long

This error related to max length of your command argument. Please check Limits on size of arguments and environment section which stated in execve man page http://man7.org/linux/man-pages/man2/execve.2.html

"... the memory used to store the environment and argument strings was limited to 32 pages (defined by the kernel constant MAX_ARG_PAGES). On architectures with a 4-kB page size, this yields a maximum size of 128 kB ... "

EDIT:

Please also note that the above error for /bin/echo is just an example, it is possible to get a similar error, when you try other ways while writing a file.It is about argument size.

SUGGESTION:

If we think writing to file operations atomically, each time pipe is generated for writing, file descriptors are opened and closed.It takes a some time.Instead of using /bin/echo or others, you can write your own "WriteFile" program with higher level language like C/C++.What you need to is I/O redirection.

  1. Open file descriptor
  2. Write data
  3. Close file descriptor
  4. Optimize your code
  5. Done

    Please check System Calls like ssize_t write(int fd, const void *buf, size_t count);

    http://linux.die.net/man/2/write

  • 1
    that's an excellent point - I've run into that before in tests. it gets to the point where only shell builtins run because the kernel cant load another executable - you end up dead in the water. – mikeserv Dec 10 '14 at 21:24
  • The argument list too long problem can easily be bypassed. It's not really worth mentioning here. Who would use /bin/echo when nearly every shell has a built-in version? – jw013 Dec 10 '14 at 21:34
  • 1
    @mikeserv Well obviously everything stops working if you exhaust the memory in the machine. That's why I was making the assumption that wouldn't happen. It is only about 10 MB (100k iterations * 100 chars) of data after all and everyone has 10 MB nowadays. The problem isn't running out of memory. And as I explained, you don't need to put this 10 MB shell variable in any argument lists or environments. So what exactly do you think you are running out of? We aren't passing this 10 MB shell variable to db2 if that is what you are thinking because the question doesn't say that at all. – jw013 Dec 11 '14 at 0:50
  • 1
    @mikeserv I'm sorry, that is not the case. You can see this yourself: foo='not in the environment'; envfoo='in the environment'; export envfoo; env | grep foo= and see what you can get. Only the exported variable is shown. I've looked at your links and nothing in them contradicts this. – jw013 Dec 11 '14 at 0:59
  • 1
    @jw013 could you check unix.stackexchange.com/a/120842/12586 – Cuneyit Kiris Dec 11 '14 at 1:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.