Docker container hangs in status Created

Question

In an ubuntu terminal, I run:

sudo strace -f -p `pidof containerd` -o strafce_log

In another temrinal I run

docker run ubuntu

The result, both terminals hang :( If I stop the docker service, then I cannot start it anymore

The container stays in status created, and any command I issue against it hangs (docker logs, docker rm, system prune)

Docker commands against other containers or images works fine. Docker ps also works

I also found online that i should look at cat /var/log/syslog | grep docker. Unfortunately nothing over there neither.

Can you guide me on where i could get more logs or some info on what is docker or that container doing?

Answer 1

Analysis of logs produced by sudo strace -ff -p $(pidof containerd) -o strace_log shows that it stalls when execve() is called by a process named runc init. This led me to this bug report on runc GH page that explains that it's a kernel bug that can be reproduced with every program that uses pthreads and execve() and presents a short Go program that can be used to reproduce the problem:

package main

import (
    "os"
    "syscall"
)

func main() {
    syscall.Exec("/bin/echo", []string{"/bin/echo", "Hello"}, os.Environ())
}

Knowing it's a kernel bug I used Raspberry Pi for testing so that I could build and replace Linux kernel quickly without risking making my main machine unbootable or playing around with virtual machines and as Go is not installed by default in Raspbian I used the following C program for testing:

#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <unistd.h>
#include <stdint.h>

void *foo(void *p)
{
  (void) p;
  while(1)
    {
      puts("in thread");
      sleep(2);
    }
}

int main(void)
{
  printf("pid: %jd\n", (intmax_t) getpid());
  pthread_t tid;
  pthread_create(&tid, NULL, foo, NULL);

  char *envp[] = {"var"};
  char *ls_args[] = { "/bin/ls" , "-l", NULL};
  if (execve(ls_args[0], ls_args, envp) < 0) {    
    perror("execve error");
    return EXIT_FAILURE;
  }

  return EXIT_SUCCESS;
}

Build with gcc strace-test.c -o strace-test -pthread and see that strace -f ./strace-test gets stuck as well.

Patch that was supposed to fix strace was sent to Linux kernel mailing list in 2016 but it has not been accepted, you can read the entire discussion to get the idea why. Nonetheless, I applied this patch to my local Linux kernel tree for Raspberry Pi against rpi-5.6.y branch and after rebuilding and replacing the kernel on Raspberry Pi strace -f ./strace-test doesn't get stuck any more. Linux source code has changed a bit since 2016 and the patch doesn't apply cleanly. FWIW, the full patch I applied is:

diff --git a/kernel/fork.c b/kernel/fork.c
index 60a1295..c26aaa1 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -1224,7 +1224,7 @@ struct mm_struct *mm_access(struct task_struct *task, unsigned int mode)
        struct mm_struct *mm;
        int err;

-       err =  mutex_lock_killable(&task->signal->cred_guard_mutex);
+       err =  mutex_lock_interruptible(&task->signal->cred_guard_mutex);
        if (err)
                return ERR_PTR(err);

diff --git a/kernel/signal.c b/kernel/signal.c
index 9ad8dea..ea7c7b5 100644
--- a/kernel/signal.c
+++ b/kernel/signal.c
@@ -108,6 +108,10 @@ static bool sig_ignored(struct task_struct *t, int sig, bool force)
        if (sigismember(&t->blocked, sig) || sigismember(&t->real_blocked, sig))
                return false;

+       /* Do not ignore signals sent from child to the parent */
+       if (current->ptrace && current->parent == t)
+               return 0;
+
        /*
         * Tracers may want to know about even ignored signal unless it
         * is SIGKILL which can't be reported anyway but can be ignored