1

I use a bash script to download a large file from a client device into a web server.

But since it will take a large amount of time, nginx will return gateway timeout. What i plan to do is pass the stdout of bash script to a python subprocess and send it as response. Since scp will not produce any output in stdout, I need to send some response every minute until scp command finishes.

Is there any way to achieve this using bash script?

Also I cannot increase Nginx timeout since it will usually take more than 20 minutes

  • Welcome to Unix.stackexchange! To get the most out of the site it is important to ask good questions. A guide to asking questions is at: unix.stackexchange.com/help/how-to-ask. I am having trouble understanding where exactly in the stack you need help. You have mentioned both the client and the server, where and how do you need help? – Stephen Rauch Mar 8 '17 at 5:46
  • nginx when used as a proxy have a timeout if no response is received from the server within a time and it gives a 504 gateway timeout error. so to avoid it i have to send something in every predefined time interval when i have to make the client wait until a big file is downloaded in the server – Haseeb Eqx Mar 8 '17 at 10:38
1

You can spawn a process that does some output in the background and kill it once scp is done:

(while sleep 60; do echo downloading; done) &
PID=$!
scp "$1" "$2"
kill $PID
  • this one looks more efficient for my need – Haseeb Eqx Mar 9 '17 at 5:32
2

Here is more general function to handle download and wait:

#! /bin/bash

# Wait for process end and show seconds count down
# Arg1: PID
# Arg2: Expected seconds
# Arg3: (optional): dump file
# Arg4: (optional): expected file size in bytes
function waitpid() {
[ -n "$3" ] && touch "$3"
local COUNT=$(( $2*10 ))
while [ -e /proc/$1 ]; do 
    if [ $(( COUNT%10 )) -eq 0 ]; then
        echo -en "\r$(( COUNT/10 )) sec"
        if [ -n "$3" ]; then
            bytes=$( stat --format=%s $3 )
            echo -n " $bytes bytes "
            if [ -n "$4" ]; then
                echo -n "$(( $bytes*100/$4 ))% "
            fi
        fi
    fi
    echo -en ".\e[K"
    sleep 0.1
    [ $COUNT -gt 0 ] && COUNT=$(( COUNT-1 ))
done
echo
}

echo "Example 1: Wait for process to be finished"
sleep 10 &
waitpid $! 10

echo "Example 2: Wait and show file size"
for N in {1..5} ; do echo "BAR $N" >> /tmp/foo ; sleep 2 ; done &
waitpid $! 10 /tmp/foo

Example output:

Example 1: Wait for process to be finished
3 sec........
Example 2: Wait and show file size
7 sec 160 bytes .......

Edit: If too complex use the short version:

sleep 10 &
while [ -e /proc/$! ]; do echo -n . ; sleep 1 ; done

Screen shot:

$ sleep 10 &
[1] 5432
$ while [ -e /proc/$! ]; do echo -n . ; sleep 1 ; done
..........[1]+  Done sleep 10
  • nice one, recommended for general use. but i just have to echo something. no need for much complexity – Haseeb Eqx Mar 9 '17 at 5:29
0

OK, I finally did it, I pass arguments $1 and and $2 for scp, here I get the process id of scp command and until it finishes I echo the response downloading in every one minute. For this I use SECONDS in bash

t=$SECONDS
scp $1 $2 &
PROC_ID=$!
while kill -0 "$PROC_ID" >/dev/null 2>&1; do
    n=$(( SECONDS-t))
    if (( n > 60 )); then
        echo downloading
        t=$SECONDS
    fi
done
  • 1
    This is very busy polling. You should add a sleep in at the end of the loop to avoid hogging the CPU. – kmkaplan Mar 8 '17 at 8:23
  • @kmkaplan how much time, if i add sleep even if download finished it will again sleep until that time finishes right? – Haseeb Eqx Mar 8 '17 at 10:33
  • It depends, maybe 1 second, but anything between 100ms and 10s depending on personal preference. PS. On GNU, you can sleep just a fraction of second, eg. sleep 0.5. But other implementations of sleep(1) only accept integer arguments. – Oskar Skog Mar 8 '17 at 10:55
  • @HaseebEqx yes. A good start would be a one second sleep. But see my answer (unix.stackexchange.com/a/350027/26432) to do it the other way round. – kmkaplan Mar 8 '17 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.