0

I am troubleshooting some outage issues on a server, with the suspicion that the server is restarting intermittently. When I run the last command, I see the following output (I added line numbers for clarity): 

01. root@nina-xms-dev:/home/nina# last
02. nina     pts/0        192.168.2.1      Wed Jan 30 12:09   still logged in
03. reboot   system boot  4.15.0-44-generi Wed Jan 30 12:09   still running
04. nina     pts/0        192.168.2.1      Wed Jan 30 11:59 - crash  (00:09)
05. reboot   system boot  4.15.0-44-generi Wed Jan 30 11:55   still running
06. nina     pts/0        192.168.2.1      Wed Jan 30 11:48 - 11:54  (00:06)
07. reboot   system boot  4.15.0-44-generi Wed Jan 30 11:47 - 11:54  (00:07)
08. nina     pts/0        192.168.2.1      Wed Jan 30 11:45 - crash  (00:01)
09. reboot   system boot  4.15.0-44-generi Wed Jan 30 11:34 - 11:54  (00:20)
10. reboot   system boot  4.15.0-44-generi Wed Jan 30 11:23 - 11:54  (00:31)
11. reboot   system boot  4.15.0-44-generi Wed Jan 30 11:08 - 11:54  (00:45)
12. reboot   system boot  4.15.0-44-generi Wed Jan 30 10:58 - 11:54  (00:55)
13. reboot   system boot  4.15.0-44-generi Wed Jan 30 10:50 - 11:54  (01:04)
14. reboot   system boot  4.15.0-44-generi Wed Jan 30 10:31 - 11:54  (01:23)
15. reboot   system boot  4.15.0-44-generi Wed Jan 30 10:18 - 11:54  (01:36)
16. reboot   system boot  4.15.0-44-generi Wed Jan 30 10:04 - 11:54  (01:50)
17. reboot   system boot  4.15.0-44-generi Wed Jan 30 09:49 - 11:54  (02:05)
18. reboot   system boot  4.15.0-44-generi Wed Jan 30 09:35 - 11:54  (02:19)
19. reboot   system boot  4.15.0-44-generi Wed Jan 30 09:25 - 11:54  (02:29)
20. reboot   system boot  4.15.0-44-generi Wed Jan 30 09:18 - 11:54  (02:36)
21. reboot   system boot  4.15.0-44-generi Wed Jan 30 09:06 - 11:54  (02:48)
22. reboot   system boot  4.15.0-44-generi Wed Jan 30 08:53 - 11:54  (03:01)
23. reboot   system boot  4.15.0-44-generi Wed Jan 30 08:44 - 11:54  (03:10)
24. reboot   system boot  4.15.0-44-generi Wed Jan 30 08:32 - 11:54  (03:22)
25. reboot   system boot  4.15.0-44-generi Wed Jan 30 08:19 - 11:54  (03:35)
26. reboot   system boot  4.15.0-44-generi Wed Jan 30 07:59 - 11:54  (03:55)
27. reboot   system boot  4.15.0-44-generi Wed Jan 30 07:54 - 11:54  (04:00)
28. reboot   system boot  4.15.0-44-generi Wed Jan 30 07:45 - 11:54  (04:09)
29. reboot   system boot  4.15.0-44-generi Wed Jan 30 07:33 - 11:54  (04:21)
30. reboot   system boot  4.15.0-44-generi Wed Jan 30 07:25 - 11:54  (04:29)
31. reboot   system boot  4.15.0-44-generi Wed Jan 30 07:08 - 11:54  (04:46)
32. reboot   system boot  4.15.0-44-generi Wed Jan 30 07:00 - 11:54  (04:54)
33. reboot   system boot  4.15.0-44-generi Wed Jan 30 06:41 - 11:54  (05:13)
34. reboot   system boot  4.15.0-44-generi Wed Jan 30 06:20 - 11:54  (05:34)
35. reboot   system boot  4.15.0-44-generi Wed Jan 30 06:02 - 11:54  (05:51)
36. reboot   system boot  4.15.0-44-generi Wed Jan 30 05:52 - 11:54  (06:02)
37. reboot   system boot  4.15.0-44-generi Wed Jan 30 05:43 - 11:54  (06:11)
38. reboot   system boot  4.15.0-44-generi Wed Jan 30 05:28 - 11:54  (06:26)
39. reboot   system boot  4.15.0-44-generi Wed Jan 30 05:16 - 11:54  (06:38)
40. reboot   system boot  4.15.0-44-generi Wed Jan 30 05:00 - 11:54  (06:54)
41. reboot   system boot  4.15.0-44-generi Wed Jan 30 04:48 - 11:54  (07:05)
42. reboot   system boot  4.15.0-44-generi Wed Jan 30 04:32 - 11:54  (07:22)
43. reboot   system boot  4.15.0-44-generi Wed Jan 30 04:17 - 11:54  (07:37)
44. reboot   system boot  4.15.0-44-generi Wed Jan 30 04:09 - 11:54  (07:45)
45. reboot   system boot  4.15.0-44-generi Wed Jan 30 03:40 - 11:54  (08:14)
46. reboot   system boot  4.15.0-44-generi Wed Jan 30 03:29 - 11:54  (08:25)
47. reboot   system boot  4.15.0-44-generi Wed Jan 30 03:17 - 11:54  (08:36)
48. reboot   system boot  4.15.0-44-generi Wed Jan 30 03:14 - 11:54  (08:40)
49. reboot   system boot  4.15.0-44-generi Wed Jan 30 03:02 - 11:54  (08:52)
50. reboot   system boot  4.15.0-44-generi Wed Jan 30 02:51 - 11:54  (09:03)
51. reboot   system boot  4.15.0-44-generi Wed Jan 30 02:33 - 11:54  (09:21)
52. reboot   system boot  4.15.0-44-generi Wed Jan 30 02:23 - 11:54  (09:31)
53. reboot   system boot  4.15.0-44-generi Wed Jan 30 02:15 - 11:54  (09:39)
54. reboot   system boot  4.15.0-44-generi Wed Jan 30 02:02 - 11:54  (09:52)
55. reboot   system boot  4.15.0-44-generi Wed Jan 30 01:54 - 11:54  (10:00)
56. reboot   system boot  4.15.0-44-generi Wed Jan 30 01:40 - 11:54  (10:14)
57. reboot   system boot  4.15.0-44-generi Wed Jan 30 01:30 - 11:54  (10:24)
58. reboot   system boot  4.15.0-44-generi Wed Jan 30 01:21 - 11:54  (10:33)
59. reboot   system boot  4.15.0-44-generi Wed Jan 30 01:05 - 11:54  (10:49)
60. reboot   system boot  4.15.0-44-generi Wed Jan 30 00:59 - 11:54  (10:55)
61. reboot   system boot  4.15.0-44-generi Wed Jan 30 00:46 - 11:54  (11:07)
62. reboot   system boot  4.15.0-44-generi Wed Jan 30 00:32 - 11:54  (11:22)
63. reboot   system boot  4.15.0-44-generi Wed Jan 30 00:17 - 11:54  (11:37)
64. reboot   system boot  4.15.0-44-generi Wed Jan 30 00:03 - 11:54  (11:51)
65. reboot   system boot  4.15.0-44-generi Tue Jan 29 23:46 - 11:54  (12:08)
66. reboot   system boot  4.15.0-44-generi Tue Jan 29 23:39 - 11:54  (12:15)
67. reboot   system boot  4.15.0-44-generi Tue Jan 29 23:34 - 11:54  (12:20)
68. reboot   system boot  4.15.0-44-generi Tue Jan 29 23:18 - 11:54  (12:36)
69. reboot   system boot  4.15.0-44-generi Tue Jan 29 22:48 - 11:54  (13:05)
70. reboot   system boot  4.15.0-44-generi Tue Jan 29 22:44 - 11:54  (13:10)
71. reboot   system boot  4.15.0-44-generi Tue Jan 29 22:39 - 11:54  (13:15)
72. reboot   system boot  4.15.0-44-generi Tue Jan 29 22:25 - 11:54  (13:29)
73. reboot   system boot  4.15.0-44-generi Tue Jan 29 22:19 - 11:54  (13:35)
74. reboot   system boot  4.15.0-44-generi Tue Jan 29 22:08 - 11:54  (13:46)
75. reboot   system boot  4.15.0-44-generi Tue Jan 29 21:45 - 11:54  (14:09)
76. reboot   system boot  4.15.0-44-generi Tue Jan 29 21:24 - 11:54  (14:30)
77. reboot   system boot  4.15.0-44-generi Tue Jan 29 21:09 - 11:54  (14:45)
78. reboot   system boot  4.15.0-44-generi Tue Jan 29 20:58 - 11:54  (14:56)
79. reboot   system boot  4.15.0-44-generi Tue Jan 29 20:43 - 11:54  (15:11)
80. reboot   system boot  4.15.0-44-generi Tue Jan 29 20:40 - 11:54  (15:14)
81. reboot   system boot  4.15.0-44-generi Tue Jan 29 20:30 - 11:54  (15:24)
82. reboot   system boot  4.15.0-44-generi Tue Jan 29 20:20 - 11:54  (15:34)
83. reboot   system boot  4.15.0-44-generi Tue Jan 29 20:13 - 11:54  (15:41)
84. reboot   system boot  4.15.0-44-generi Tue Jan 29 20:07 - 11:54  (15:46)
85. reboot   system boot  4.15.0-44-generi Tue Jan 29 19:49 - 11:54  (16:05)
86. reboot   system boot  4.15.0-44-generi Tue Jan 29 19:24 - 11:54  (16:30)
87. reboot   system boot  4.15.0-44-generi Tue Jan 29 19:09 - 11:54  (16:45)
88. reboot   system boot  4.15.0-44-generi Tue Jan 29 18:51 - 11:54  (17:03)
89. reboot   system boot  4.15.0-44-generi Tue Jan 29 18:36 - 11:54  (17:18)
90. reboot   system boot  4.15.0-44-generi Tue Jan 29 18:18 - 11:54  (17:36)
91. reboot   system boot  4.15.0-44-generi Tue Jan 29 18:08 - 11:54  (17:46)
92. reboot   system boot  4.15.0-44-generi Tue Jan 29 17:57 - 11:54  (17:57)
93. reboot   system boot  4.15.0-44-generi Tue Jan 29 17:31 - 11:54  (18:23)
94. reboot   system boot  4.15.0-44-generi Tue Jan 29 17:18 - 11:54  (18:36)
95. reboot   system boot  4.15.0-44-generi Tue Jan 29 17:02 - 11:54  (18:52)
96. nina     pts/0        192.168.1.222    Tue Jan 29 17:00 - crash  (00:01)
97. reboot   system boot  4.15.0-44-generi Tue Jan 29 16:59 - 11:54  (18:55)
98. nina     pts/0        192.168.1.222    Tue Jan 29 16:54 - crash  (00:04)
99. reboot   system boot  4.15.0-44-generi Tue Jan 29 16:51 - 11:54  (19:03)
100. nina     pts/0        192.168.1.222    Tue Jan 29 16:49 - 16:50  (00:00)
101. reboot   system boot  4.15.0-44-generi Tue Jan 29 16:39 - 16:50  (00:10)
102. nina     pts/0        192.168.1.222    Tue Jan 29 16:07 - crash  (00:31)
103. reboot   system boot  4.15.0-44-generi Tue Jan 29 16:07 - 16:50  (00:43)
104. nina     pts/0        192.168.1.222    Tue Jan 29 15:59 - crash  (00:07)
105. reboot   system boot  4.15.0-44-generi Tue Jan 29 15:51 - 16:50  (00:59)
106. reboot   system boot  4.15.0-44-generi Tue Jan 29 15:46 - 16:50  (01:04)
107. nina     pts/0        192.168.1.222    Tue Jan 29 15:38 - 15:45  (00:07)
108. reboot   system boot  4.15.0-44-generi Tue Jan 29 15:35 - 15:45  (00:09)
109. nina     pts/0        192.168.1.222    Tue Jan 29 15:26 - crash  (00:08)
110. reboot   system boot  4.15.0-44-generi Tue Jan 29 15:12 - 15:45  (00:32)
111. reboot   system boot  4.15.0-44-generi Tue Jan 29 15:02 - 15:45  (00:43)
112. reboot   system boot  4.15.0-44-generi Tue Jan 29 14:40 - 15:45  (01:05)
113. reboot   system boot  4.15.0-44-generi Tue Jan 29 14:18 - 15:45  (01:27)
114. reboot   system boot  4.15.0-44-generi Tue Jan 29 14:08 - 15:45  (01:36)
115. reboot   system boot  4.15.0-44-generi Tue Jan 29 13:41 - 15:45  (02:04)
116. reboot   system boot  4.15.0-44-generi Tue Jan 29 13:36 - 15:45  (02:08)
117. reboot   system boot  4.15.0-44-generi Tue Jan 29 13:27 - 15:45  (02:17)
118. reboot   system boot  4.15.0-44-generi Tue Jan 29 13:14 - 15:45  (02:30)
119. reboot   system boot  4.15.0-44-generi Tue Jan 29 13:01 - 15:45  (02:44)
120. reboot   system boot  4.15.0-44-generi Tue Jan 29 12:41 - 15:45  (03:03)
121. reboot   system boot  4.15.0-44-generi Tue Jan 29 12:34 - 15:45  (03:10)
122. reboot   system boot  4.15.0-44-generi Tue Jan 29 12:27 - 15:45  (03:17)
123. reboot   system boot  4.15.0-44-generi Tue Jan 29 12:13 - 15:45  (03:31)
124. reboot   system boot  4.15.0-44-generi Tue Jan 29 11:57 - 15:45  (03:48)
125. reboot   system boot  4.15.0-44-generi Tue Jan 29 11:48 - 15:45  (03:57)
126. reboot   system boot  4.15.0-44-generi Tue Jan 29 11:16 - 15:45  (04:28)
127. reboot   system boot  4.15.0-44-generi Tue Jan 29 10:16 - 15:45  (05:28)
128. reboot   system boot  4.15.0-44-generi Tue Jan 29 09:08 - 15:45  (06:36)
129. reboot   system boot  4.15.0-44-generi Tue Jan 29 07:38 - 15:45  (08:06)
130. reboot   system boot  4.15.0-44-generi Tue Jan 29 06:45 - 15:45  (08:59)
131. reboot   system boot  4.15.0-44-generi Tue Jan 29 05:10 - 15:45  (10:34)
132. reboot   system boot  4.15.0-44-generi Tue Jan 29 04:36 - 15:45  (11:09)
133. reboot   system boot  4.15.0-44-generi Tue Jan 29 03:48 - 15:45  (11:56)
134. reboot   system boot  4.15.0-44-generi Tue Jan 29 03:08 - 15:45  (12:37)
135. reboot   system boot  4.15.0-44-generi Tue Jan 29 02:19 - 15:45  (13:26)
136. reboot   system boot  4.15.0-44-generi Tue Jan 29 00:18 - 15:45  (15:27)
137. reboot   system boot  4.15.0-44-generi Mon Jan 28 23:45 - 15:45  (15:59)
138. reboot   system boot  4.15.0-43-generi Mon Jan 28 20:49 - 15:45  (18:56)
139. reboot   system boot  4.15.0-43-generi Mon Jan 28 19:54 - 15:45  (19:51)
140. reboot   system boot  4.15.0-43-generi Mon Jan 28 19:21 - 15:45  (20:23)
141. reboot   system boot  4.15.0-43-generi Mon Jan 28 17:14 - 15:45  (22:31)
142. reboot   system boot  4.15.0-43-generi Mon Jan 28 14:03 - 15:45 (1+01:42)
143. reboot   system boot  4.15.0-43-generi Mon Jan 28 13:33 - 15:45 (1+02:11)
144. nina     pts/0        192.168.1.222    Mon Jan 28 10:54 - 11:04  (00:10)
145. reboot   system boot  4.15.0-43-generi Mon Jan 28 10:53 - 15:45 (1+04:51)
146. nina     pts/0        192.168.1.222    Mon Jan 28 10:42 - 10:53  (00:10)
147. reboot   system boot  4.15.0-43-generi Mon Jan 28 10:39 - 10:53  (00:13)
148. nina     pts/0        192.168.1.222    Mon Jan 28 10:35 - crash  (00:03)
149. reboot   system boot  4.15.0-43-generi Mon Jan 28 05:40 - 10:53  (05:12)
150. reboot   system boot  4.15.0-43-generi Mon Jan 28 03:25 - 10:53  (07:27)
151. ... etc....

Given this line:

09. reboot   system boot  4.15.0-44-generi Wed Jan 30 11:34 - 11:54  (00:20)

My understanding is that the line above is saying that the system started up at 11:34, was up for 20 minutes, and then was shut down at 11:54.

What I don't understand is that given multiple lines that repeat with the same shut down time:

09. reboot   system boot  4.15.0-44-generi Wed Jan 30 11:34 - 11:54  (00:20)
10. reboot   system boot  4.15.0-44-generi Wed Jan 30 11:23 - 11:54  (00:31)
11. reboot   system boot  4.15.0-44-generi Wed Jan 30 11:08 - 11:54  (00:45)
12. reboot   system boot  4.15.0-44-generi Wed Jan 30 10:58 - 11:54  (00:55)
13. reboot   system boot  4.15.0-44-generi Wed Jan 30 10:50 - 11:54  (01:04)

Line 10 says the computer started up at 11:23, line 11 says the computer was started at 11:08, line 12 says 10:58, and line 13 says 10:50... but all of them say that the shut down time was 11:54.

Instead, I would expect each shut down time to have ended before the proceeding start up. Something more like this:

09. reboot   system boot  4.15.0-44-generi Wed Jan 30 11:34 - 11:54  (00:20)
10. reboot   system boot  4.15.0-44-generi Wed Jan 30 11:23 - 11:33  (00:10)
11. reboot   system boot  4.15.0-44-generi Wed Jan 30 11:08 - 11:22  (00:14)
12. reboot   system boot  4.15.0-44-generi Wed Jan 30 10:58 - 11:07  (00:09)
13. reboot   system boot  4.15.0-44-generi Wed Jan 30 10:50 - 11:57  (00:07)

Should I assume that there were un-recorded shutdowns between each of those periods (not just the 11:54 one?), OR should I assume that there were not actual separate start ups at each of those times, and the server was on for the whole time?

Are there other log files to correlate the log of start up and shut down events?

Improve this question

1 Answer 1

Reset to default
0

I found a little more information that helps indicate what's going on. It turns out that the "last" command doesn't output shutdowns or run level changes by default. (A run level of 0 is a shut down, and others indicate the system is running. Level 5 appears to be the normal run level.)

When I add the '-x' flag, now I'm seeing the following output:

01. nina   @nina   -xms-dev:~$ last -x
02. nina     pts/0        192.168.2.1      Wed Jan 30 13:12   still logged in
03. runlevel (to lvl 5)   4.15.0-44-generi Wed Jan 30 13:09   still running
04. reboot   system boot  4.15.0-44-generi Wed Jan 30 13:09   still running
05. runlevel (to lvl 5)   4.15.0-44-generi Wed Jan 30 12:59 - 13:09  (00:10)
06. reboot   system boot  4.15.0-44-generi Wed Jan 30 12:59   still running
07. runlevel (to lvl 5)   4.15.0-44-generi Wed Jan 30 12:50 - 12:59  (00:08)
08. reboot   system boot  4.15.0-44-generi Wed Jan 30 12:50   still running
09. runlevel (to lvl 5)   4.15.0-44-generi Wed Jan 30 12:33 - 12:50  (00:17)
10. reboot   system boot  4.15.0-44-generi Wed Jan 30 12:32   still running
11. nina     pts/0        192.168.2.1      Wed Jan 30 12:09 - crash  (00:23)
12. runlevel (to lvl 5)   4.15.0-44-generi Wed Jan 30 12:09 - 12:33  (00:23)
13. reboot   system boot  4.15.0-44-generi Wed Jan 30 12:09   still running
14. nina     pts/0        192.168.2.1      Wed Jan 30 11:59 - crash  (00:09)
15. runlevel (to lvl 5)   4.15.0-44-generi Wed Jan 30 11:55 - 12:09  (00:13)
16. reboot   system boot  4.15.0-44-generi Wed Jan 30 11:55   still running
17. shutdown system down  4.15.0-44-generi Wed Jan 30 11:54 - 11:55  (00:00)
18. nina     pts/0        192.168.2.1      Wed Jan 30 11:48 - 11:54  (00:06)
19. runlevel (to lvl 5)   4.15.0-44-generi Wed Jan 30 11:47 - 11:54  (00:06)
20. reboot   system boot  4.15.0-44-generi Wed Jan 30 11:47 - 11:54  (00:07)
21. nina     pts/0        192.168.2.1      Wed Jan 30 11:45 - crash  (00:01)
22. runlevel (to lvl 5)   4.15.0-44-generi Wed Jan 30 11:34 - 11:47  (00:13)
23. reboot   system boot  4.15.0-44-generi Wed Jan 30 11:34 - 11:54  (00:20)
24. runlevel (to lvl 5)   4.15.0-44-generi Wed Jan 30 11:23 - 11:34  (00:10)
25. reboot   system boot  4.15.0-44-generi Wed Jan 30 11:23 - 11:54  (00:31)
26. runlevel (to lvl 5)   4.15.0-44-generi Wed Jan 30 11:09 - 11:23  (00:14)
27. reboot   system boot  4.15.0-44-generi Wed Jan 30 11:08 - 11:54  (00:45)
28. runlevel (to lvl 5)   4.15.0-44-generi Wed Jan 30 10:59 - 11:09  (00:10)
29. reboot   system boot  4.15.0-44-generi Wed Jan 30 10:58 - 11:54  (00:55)
30. runlevel (to lvl 5)   4.15.0-44-generi Wed Jan 30 10:50 - 10:59  (00:08)
31. reboot   system boot  4.15.0-44-generi Wed Jan 30 10:50 - 11:54  (01:04)
32. runlevel (to lvl 5)   4.15.0-44-generi Wed Jan 30 10:32 - 10:50  (00:18)
33. reboot   system boot  4.15.0-44-generi Wed Jan 30 10:31 - 11:54  (01:23)
34. runlevel (to lvl 5)   4.15.0-44-generi Wed Jan 30 10:18 - 10:32  (00:13)

The reboot commands are the same, and have the odd data, but the runlevel indicators are closer to the output I would expect: 

  19. runlevel (to lvl 5)   4.15.0-44-generi Wed Jan 30 11:47 - 11:54  (00:06)
  20. reboot   system boot  4.15.0-44-generi Wed Jan 30 11:47 - 11:54  (00:07)
  21. nina     pts/0        192.168.2.1      Wed Jan 30 11:45 - crash  (00:01)
> 22. runlevel (to lvl 5)   4.15.0-44-generi Wed Jan 30 11:34 - 11:47  (00:13)
  23. reboot   system boot  4.15.0-44-generi Wed Jan 30 11:34 - 11:54  (00:20)
> 24. runlevel (to lvl 5)   4.15.0-44-generi Wed Jan 30 11:23 - 11:34  (00:10)
  25. reboot   system boot  4.15.0-44-generi Wed Jan 30 11:23 - 11:54  (00:31)
> 26. runlevel (to lvl 5)   4.15.0-44-generi Wed Jan 30 11:09 - 11:23  (00:14)
  27. reboot   system boot  4.15.0-44-generi Wed Jan 30 11:08 - 11:54  (00:45)
> 28. runlevel (to lvl 5)   4.15.0-44-generi Wed Jan 30 10:59 - 11:09  (00:10)
  29. reboot   system boot  4.15.0-44-generi Wed Jan 30 10:58 - 11:54  (00:55)
> 30. runlevel (to lvl 5)   4.15.0-44-generi Wed Jan 30 10:50 - 10:59  (00:08)
  31. reboot   system boot  4.15.0-44-generi Wed Jan 30 10:50 - 11:54  (01:04)

It looks like if the server is shutting down for some unknown reason, the runlevel is more reliable than the reboot information alone.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .