1

We recently moved our development infrastructure from our own old machines running Ubuntu 12.04 to Google Cloud instances running Ubuntu 18.04.

Developers usually start some screens and run django servers within those screens.

For example, one may create a screen screen -S webserver_5552 and run its django development application within the screen python manage.py runserver 0.0.0.0:5552

On our previous machines, we could detach the screen (ctrl+a d) and come back later (screen -r xxxx.webserver_5552): the django server process would still be here, up and running, and owned by the bash process of the screen.

On the Google Cloud machine, this is, however, working differently and has been driving us crazy. We can still detach the screen, but if we come back later after a while, the django process is no longer owned by the bash process! Instead, it is owned by the init process (ppid set to 1 from ps).

Usually, we got a backtrace from the django process before it received some signals and got his ownership changed, but that's all we got and we can't figure out what's the root cause and how to prevent it:

Traceback (most recent call last):
  File "manage.py", line 65, in <module>
    execute_from_command_line(sys.argv)
  File "/home/testing/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 367, in execute_from_command_line
    utility.execute()
  File "/home/testing/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 359, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/testing/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 294, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/testing/env/local/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 58, in execute
    super(Command, self).execute(*args, **options)
  File "/home/testing/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 345, in execute
    output = self.handle(*args, **options)
  File "/home/testing/env/local/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 97, in handle
    self.run(**options)
  File "/home/testing/env/local/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 106, in run
    autoreload.main(self.inner_run, None, options)
  File "/home/testing/env/local/lib/python2.7/site-packages/django/utils/autoreload.py", line 333, in main
    reloader(wrapped_main_func, args, kwargs)
  File "/home/testing/env/local/lib/python2.7/site-packages/django/utils/autoreload.py", line 304, in python_reloader
    exit_code = restart_with_reloader()
  File "/home/testing/env/local/lib/python2.7/site-packages/django/utils/autoreload.py", line 290, in restart_with_reloader
    exit_code = os.spawnve(os.P_WAIT, sys.executable, args, new_environ)
  File "/home/testing/odesk_android/../env/lib/python2.7/os.py", line 575, in spawnve
    return _spawnvef(mode, file, args, env, execve)
  File "/home/testing/odesk_android/../env/lib/python2.7/os.py", line 548, in _spawnvef
    wpid, sts = waitpid(pid, 0)
OSError: [Errno 4] Interrupted system call

This has been a particularly annoying behavior and we could not figure out the root cause (does it come from ubuntu, GCP, some misconfiguration, ...?).

EDIT:

I made a test by starting the screen and launching screen:

testing@whova-qa-01:/home/simon_ninon_whova_com$ ps -ejf | grep 55530
testing  20764 19638 20764 19638  4 00:12 pts/18   00:00:01 ../env/bin/python manage.py runserver 0.0.0.0:55530
testing  20769 20764 20764 19638 12 00:12 pts/18   00:00:04 /home/testing/appium_android/../env/bin/python manage.py runserver 0.0.0.0:55530

As you can see, I started a django process ../env/bin/python manage.py runserver 0.0.0.0:55530 with PID=20764 and PPID=19638 (bash process)

This django process created a child /home/testing/appium_android/../env/bin/python manage.py runserver 0.0.0.0:55530 with PID=20769 and PPID=20764 (the original process I spawned)

Now, this morning, when I logged back on the machine, before I reattached the screen, everything was still the same:

simon_ninon_whova_com@whova-qa-01:~$ ps -ejf | grep 55530
simon_n+  9026  9011  9025  9011  0 09:09 pts/9    00:00:00 grep --color=auto 55530
testing  20764 19638 20764 19638  0 00:12 pts/18   00:00:01 ../env/bin/python manage.py runserver 0.0.0.0:55530
testing  20769 20764 20764 19638  2 00:12 pts/18   00:13:56 /home/testing/appium_android/../env/bin/python manage.py runserver 0.0.0.0:55530

So when I re-attached the screen, I expected the issue to not be there. However, when I re-attached the screen: boom, the process was killed!

testing@whova-qa-01:~$ ps -ejf | grep 55530
testing   9085  9031  9084  9011  0 09:10 pts/9    00:00:00 grep --color=auto 55530
testing  20769     1 20764 19638  2 00:12 pts/18   00:13:59 /home/testing/appium_android/../env/bin/python manage.py runserver 0.0.0.0:55530

As you can see, the parent process got killed, and the child is still there, with the init process owning it.

Interestingly, checking if the original bash process is still alive proves so:

simon_ninon_whova_com@whova-qa-01:~$ ps aux | grep 19638
simon_n+  9315  0.0  0.0  14664  1016 pts/9    S+   09:16   0:00 grep --color=auto 19638
testing  19638  0.0  0.0  25360  7692 pts/18   Ss+  Feb27   0:00 /bin/bash

So it seems that re-attaching the screen keeps the bash process but leads to the parent django process to be killed for some reason.

Not sure what can happen at that step? Note that if I start the server, detach and reattach the screen in a short time, the problem is not triggered, it only happens after a while.

  • When you attach to the screen session, does it still run the same shell process with the same PID? – Bodo Feb 28 at 7:32
  • Hmm. this would mean the Shell-process died, and the Screen process is handed back to init then. Would that be a possibility? – gerhard d. Feb 28 at 7:38
  • It is expected behaviour orphaned processes whose parents die being owned by 1 – Rui F Ribeiro Feb 28 at 7:38
  • That's a valid point. I would need a bit time to double check that (usually takes a couple of hours for the problem to be triggered: I just recorded the current shell pid and will check back later). If that's the case, how could that happen? We don't have anything that could kill the shell processes and it happens consistently on 5 different GCP instances. Could the OS do some cleanup? Could there be a maximum number of opened shell sessions? – Simon Ninon Feb 28 at 7:43
  • I edited the question with additional details from my observations – Simon Ninon Feb 28 at 17:26
0

I found the root cause.

When re-attaching the screen, SIGWINCH signal got sent to the parent django process. The process don't handle it and just crashes, leaving the child orphan.

This can then be easily retriggered by resizing the term or using kill -28 PID.

I am not sure why it only happens on the GCP instances though, there may be something different in the environment (python version?), anyway, that gives me more clues about where to find a solution.

Edit:

After searching for a while, the root cause was coming from one of our django dependencies importing readline in its source code.

The python readline is a binding to the gnureadline library, and it appears that the signal handling on gnureadline interferes with the one done in python/django.

Considering that it only happens on the GCP machine, but not our previous machines, I suspect that the gnureadline installed on our GCP machine is different (either in terms of version, or in terms of compilation options used), leading to a different signal handling behavior.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.