0

My kubernetes is running on top of AWS, I am using helm for templating, the problem is my tiller pod keeps on dying every few hours when there is even not much load on the cluster. I am getting no clue what so ever from logs

ubuntu@kops:~$ kubectl get pods -owide -n kube-system | grep tiller tiller-deploy-6b985bb7b-88ssk 0/1 CrashLoopBackOff 71 19h 100.96.4.3 ip-172-20-46-194.us-west-2.compute.internal ubuntu@kops:~$ ubuntu@kops:~$ ubuntu@kops:~$ kubectl describe pod tiller-deploy-6b985bb7b-88ssk Error from server (NotFound): pods "tiller-deploy-6b985bb7b-88ssk" not found ubuntu@kops:~$ ubuntu@kops:~$ ubuntu@kops:~$ ubuntu@kops:~$ ubuntu@kops:~$ kubectl logs tiller-deploy-6b985bb7b-88ssk Error from server (NotFound): pods "tiller-deploy-6b985bb7b-88ssk" not found ubuntu@kops:~$

the EC2 on which its running has a lot of free memory and CPU is 8 core load... admin@ip-172-20-46-194:~$ free -h total used free shared buffers cached Mem: 31G 5.1G 26G 1.1M 1.5G 2.4G -/+ buffers/cache: 1.2G 30G Swap: 0B 0B 0B

top - 03:08:29 up 19:20,  1 user,  load average: 79.51, 78.59, 77.98
Tasks: 176 total,   4 running, 172 sleeping,   0 stopped,   0 zombie
%Cpu(s): 28.2 us,  0.5 sy,  0.1 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si, 71.3 st
KiB Mem:  32950672 total,  5371456 used, 27579216 free,  1524240 buffers
KiB Swap:        0 total,        0 used,        0 free.  2561276 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
12203 root      20   0  906520  16756   1492 S 235.0  0.1   1427:46 docker
23974 root      20   0  906520  16796   1492 S 221.1  0.1   1054:34 docker
24072 root      20   0  906520  16796   1492 S 132.0  0.1 669:17.68 docker
12318 root      20   0  906520  16792   1492 S 130.7  0.1 900:54.95 docker
17543 nobody    20   0  906520  16796   1492 S  29.1  0.1  84:01.18 docker
23865 nobody    20   0  906520  16796   1492 R  15.9  0.1  69:02.31 docker
12112 nobody    20   0  906520  16792   1492 S  14.6  0.1  91:27.51 docker
 3013 root      20   0 6753392 124600  50028 S   7.6  0.4  73:59.01 kubelet
 6378 nobody    20   0  683644 432120  29772 R   6.0  1.3  21:09.54 prometheus
0

To get you a bit further, you need to add --namespace kube-system to your describe command:

kubectl --namespace kube-system describe pod tiller-deploy-6b985bb7b-88ssk 

Then, you can root cause more effectively. Same if you want to try to delete the pod to get past it temporarily.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.