3

I have set up a post-recieve hook for Github. It will issue a POST-Request to an Apache cgi-script, which then is supposed to download the changes to the repo into the locally cloned bare repository. The script is running fine when I add a VARIABLE=$(cat -), yet I get a curl error when I try to issue a post request and delete the line having VARIABLE=

curl: (18) transfer closed with outstanding read data remaining

My POST-request is generated by curl (the post-data is just for testing, it doesn't get used in the actual script):

curl -D - -H "Connection: close" -H "Content-Type: application/json"     -d '{  "repository": {    "name": "webhook-test",    "git_url": "git://github.com/bng44270/webhook-test.git",    "ssh_url": "git@github.com:bng44270/webhook-test.git",    "clone_url": "https://github.com/bng44270/webhook-test.git"  }}' http://10.0.0.2/cgi-bin/clone.cgi --verbose

My vhost config file:

  ScriptAlias "/cgi-bin" "/opt/hooks/cgi-bin"
  DocumentRoot /opt/hooks/html
  <Directory /opt/hooks>
    Options Indexes FollowSymLinks
    AllowOverride All
    Require all granted
  </Directory>
</VirtualHost>

My script running:

#!/bin/bash

/sbin/runuser -l gogsjail -c '/usr/bin/git --git-dir /home/gogsjail/gogs-repositories/admin/upstream.git fetch --prune >/dev/null 2>&1' >/dev/null 2>&1
# VARIABLE=$(cat -)
echo "Content-type: text/json"
echo ""
echo '{"result":"success"}'

and finally, the output from curl:

* Expire in 0 ms for 6 (transfer 0x563f98984f90)
*   Trying 10.0.0.2...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x563f98984f90)
* Connected to 10.0.0.2 (10.0.0.2) port 80 (#0)
> POST /cgi-bin/clone.cgi HTTP/1.1
> Host: 10.0.0.2
> User-Agent: curl/7.64.0
> Accept: */*
> Connection: close
> Content-Type: application/json
> Content-Length: 230
>
* upload completely sent off: 230 out of 230 bytes
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Mon, 08 Feb 2021 12:55:42 GMT
Date: Mon, 08 Feb 2021 12:55:42 GMT
< Server: Apache/2.4.38 (Debian)
Server: Apache/2.4.38 (Debian)
< Connection: close
Connection: close
< Transfer-Encoding: chunked
Transfer-Encoding: chunked
< Content-Type: text/json
Content-Type: text/json

<
{"result":"success"}
* transfer closed with outstanding read data remaining
* Closing connection 0
curl: (18) transfer closed with outstanding read data remaining

If I run with VARIABLE=$(cat -) before echo-calls I get

* Expire in 0 ms for 6 (transfer 0x55ea56355f90)
*   Trying 10.0.0.2...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x55ea56355f90)
* Connected to 10.0.0.2 (10.0.0.2) port 80 (#0)
> POST /cgi-bin/clone.cgi HTTP/1.1
> Host: 10.0.0.2
> User-Agent: curl/7.64.0
> Accept: */*
> Connection: close
> Content-Type: application/json
> Content-Length: 244
>
* upload completely sent off: 244 out of 244 bytes
< HTTP/1.1 200 OK
< Date: Mon, 08 Feb 2021 13:35:28 GMT
< Server: Apache/2.4.38 (Debian)
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: text/json
<
{"result":"success"}
* Closing connection 0

Can someone nudge me in the correct direction?

5
  • I'm trying to work out what you're doing. Are you saying that in your real script the commented out VARIABLE=... line is actually VARIABLE=$(curl -D - -H "Connection: close" -H "Content-Type: application/json" -d '{ "repository": { "name": "webhook-test", "git_url": "git://github.com/bng44270/webhook-test.git", "ssh_url": "git@github.com:bng44270/webhook-test.git", "clone_url": "https://github.com/bng44270/webhook-test.git" }}' http://10.0.0.2/cgi-bin/clone.cgi --verbose)? If not, I don't follow you – roaima Feb 8 at 15:47
  • i use curl to make the specified post-request to my apache webserver. it runs the cgi-script which gets the content of the post-request as std-in. therefore VARIABLE=$(cat -) will have the valid json inside, which you could also access with something like jq – siryx Feb 8 at 15:56
  • So the code at issue is curl … | yourScript? That's not clear in your question, either – roaima Feb 8 at 16:11
  • no, the issue is the curl output not being terminated. if i make the post request while commenting out VARIABLE=$(cat -) it gives me the error: curl: (18) transfer closed with outstanding read data remaining – siryx Feb 8 at 16:16
  • If you comment out that line, where can the curl output go? – roaima Feb 8 at 17:05
1

I imagine that apache is doing something equivalent to: reading all the request from the client, including the POST data, and writing it via a pipe (say) to the cgi process. It also reads the reply from the cgi and sends it to the client. It does a loop with select() continuing these reads and writes simultaneously.

When the cgi exits and closes the pipe, apache's select() sees an exception on read, due to the eof, and an exception on write. If the cgi does not read the POST data from apache, apache sees the pending write getting an ioerror, and so might assume the cgi is badly behaved, and just close the client connection without further cleanup.

Since the cgi does not put out a content-length header, apache cannot determine the length of the reply, so puts out a Transfer-Encoding: chunked header. This protocol surrounds each write in the reply by a mini-heading consisting of the length of the following write. The protocol needs to be cleanly terminated by sending a write length of 0 (0\r\n). Presumably, this last part of the protocol is what is missing, and is the cause of curl's complaint.

The chunked protocol is not exposed by curl, but you might see it with --raw, or by using strace().

3
  • 1
    using the --raw switch i could validate that apache indeed was not sending the final 0 neccesary for the protocol, which made curl complain. I will keep consuming the stdin for now, as I have plans for accessing it anyway, but this was enough to satisfy my curiosity. – siryx Feb 8 at 18:59
  • i tried adding printf '0\r\n', yet sadly it did not run without error and I didnt go on from there. I will try inspecting apache with strace like you mentioned and maybe find out more about the inner workings. Thanks for the answer! – siryx Feb 8 at 19:13
  • Looking at mozilla it seems that after the final chunk 0\r\n there is then a final \r\n. But this probably wont work, as apache will encapsulate it within a chunk, so I don't think you can spoof it. – meuh Feb 8 at 21:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.