Not consuming STDIN causes curl error

Question

I have set up a post-recieve hook for Github. It will issue a POST-Request to an Apache cgi-script, which then is supposed to download the changes to the repo into the locally cloned bare repository. The script is running fine when I add a VARIABLE=$(cat -), yet I get a curl error when I try to issue a post request and delete the line having VARIABLE=

curl: (18) transfer closed with outstanding read data remaining

My POST-request is generated by curl (the post-data is just for testing, it doesn't get used in the actual script):

curl -D - -H "Connection: close" -H "Content-Type: application/json"     -d '{  "repository": {    "name": "webhook-test",    "git_url": "git://github.com/bng44270/webhook-test.git",    "ssh_url": "git@github.com:bng44270/webhook-test.git",    "clone_url": "https://github.com/bng44270/webhook-test.git"  }}' http://10.0.0.2/cgi-bin/clone.cgi --verbose

My vhost config file:

  ScriptAlias "/cgi-bin" "/opt/hooks/cgi-bin"
  DocumentRoot /opt/hooks/html
  <Directory /opt/hooks>
    Options Indexes FollowSymLinks
    AllowOverride All
    Require all granted
  </Directory>
</VirtualHost>

My script running:

#!/bin/bash

/sbin/runuser -l gogsjail -c '/usr/bin/git --git-dir /home/gogsjail/gogs-repositories/admin/upstream.git fetch --prune >/dev/null 2>&1' >/dev/null 2>&1
# VARIABLE=$(cat -)
echo "Content-type: text/json"
echo ""
echo '{"result":"success"}'

and finally, the output from curl:

* Expire in 0 ms for 6 (transfer 0x563f98984f90)
*   Trying 10.0.0.2...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x563f98984f90)
* Connected to 10.0.0.2 (10.0.0.2) port 80 (#0)
> POST /cgi-bin/clone.cgi HTTP/1.1
> Host: 10.0.0.2
> User-Agent: curl/7.64.0
> Accept: */*
> Connection: close
> Content-Type: application/json
> Content-Length: 230
>
* upload completely sent off: 230 out of 230 bytes
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Mon, 08 Feb 2021 12:55:42 GMT
Date: Mon, 08 Feb 2021 12:55:42 GMT
< Server: Apache/2.4.38 (Debian)
Server: Apache/2.4.38 (Debian)
< Connection: close
Connection: close
< Transfer-Encoding: chunked
Transfer-Encoding: chunked
< Content-Type: text/json
Content-Type: text/json

<
{"result":"success"}
* transfer closed with outstanding read data remaining
* Closing connection 0
curl: (18) transfer closed with outstanding read data remaining

If I run with VARIABLE=$(cat -) before echo-calls I get

* Expire in 0 ms for 6 (transfer 0x55ea56355f90)
*   Trying 10.0.0.2...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x55ea56355f90)
* Connected to 10.0.0.2 (10.0.0.2) port 80 (#0)
> POST /cgi-bin/clone.cgi HTTP/1.1
> Host: 10.0.0.2
> User-Agent: curl/7.64.0
> Accept: */*
> Connection: close
> Content-Type: application/json
> Content-Length: 244
>
* upload completely sent off: 244 out of 244 bytes
< HTTP/1.1 200 OK
< Date: Mon, 08 Feb 2021 13:35:28 GMT
< Server: Apache/2.4.38 (Debian)
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: text/json
<
{"result":"success"}
* Closing connection 0

Can someone nudge me in the correct direction?

Answer 1

I imagine that apache is doing something equivalent to: reading all the request from the client, including the POST data, and writing it via a pipe (say) to the cgi process. It also reads the reply from the cgi and sends it to the client. It does a loop with select() continuing these reads and writes simultaneously.

When the cgi exits and closes the pipe, apache's select() sees an exception on read, due to the eof, and an exception on write. If the cgi does not read the POST data from apache, apache sees the pending write getting an ioerror, and so might assume the cgi is badly behaved, and just close the client connection without further cleanup.

Since the cgi does not put out a content-length header, apache cannot determine the length of the reply, so puts out a Transfer-Encoding: chunked header. This protocol surrounds each write in the reply by a mini-heading consisting of the length of the following write. The protocol needs to be cleanly terminated by sending a write length of 0 (0\r\n). Presumably, this last part of the protocol is what is missing, and is the cause of curl's complaint.

The chunked protocol is not exposed by curl, but you might see it with --raw, or by using strace().