0

Recently I was using my notebook in a normal session when, suddenly, the network interface stopped working. I manually shut down the system (couldn't do it via LXDE neither via terminal) expecting that things would be fixed in the next session.
However, during the startup, several errors appeared:

  • Failed to start Network Time Synchronization
  • Failed to start Raise network interface
  • Timed out waiting for device dev-di
  • Failed to start Light Display Manager
  • Failed to start OpenSSD Secure Shell server
  • ... etc ...

When reaching the login screen (via terminal since the GUI stopped working), it kept asking for username in a continuous loop, which prevents me from accessing the system and trying to fix the problem:

I found several tutorials that might help. However, all of them requires to be logged in order to access configuration files. Therefore, I'm somewhat stuck.

Any ideas?


System Specification

Notebook Gateway NE56R

  • Intel(R) Celeron(R) CPU B820 @ 1.70GHz (2 Core) 2MB L3 Cache 64-bit Processing
  • Intel(R) HD Graphics
  • 8Gb DDR3 SDRAM Memory
  • 128MB shared video memory

 

Debian Strech

  • Software selection
    (  ) Debian desktop environment
    (  ) ... GNOME
    (  ) ... Xfce
    (  ) ... KDE
    (  ) ... Cinnamon
    (  ) ... MATE
    (X) ... LXDE
    (X) web server
    (X) print server
    (  ) SSH server
    (X) standard system utilities

Follow up

Live CD
I've just boot via the official debian live cd lxde.
It seems that all the hardware is working fine, considering that I can browse the web and no errors was found on the HDD:

root@debian:~# fsck /dev/sda1
fsck from util-linux 2.29.2
e2fsck 1.43.4 (31-Jan-2017)
/dev/sda1: clean, 208700/19021824 files, 36045002/76087040 blocks   

It is also possible to mount the HDD via Disks App:

Assessment    Disk is OK (30° C / 86° F)
Device        /dev/sda1
Contents      Mounted at /hddtest

 

Debug: log files

root@debian:~# ls -lart /hddtest/var/log
total 10980
drwx------  2 speech-dispatcher root    4096 Aug  7  2017 speech-dispatcher
drwxr-xr-x  3 root              root    4096 Jul  7  2018 hp
drwxr-xr-x  3 root              root    4096 Jul  7  2018 installer
-rw-r--r--  1 root              root    4847 Jul 26  2018 alternatives.log.7.gz
-rw-r--r--  1 root              root   91559 Jul 26  2018 dpkg.log.7.gz
-rw-r--r--  1 root              root     361 Sep  1 02:05 alternatives.log.6.gz
-rw-r--r--  1 root              root    8492 Sep  1 02:05 dpkg.log.6.gz
drwxr-xr-x 12 root              root    4096 Sep 14 15:44 ..
-rw-r--r--  1 root              root     501 Sep 29 05:42 alternatives.log.5.gz
-rw-r--r--  1 root              root    8915 Sep 29 05:42 dpkg.log.5.gz
drwxr-xr-x  2 root              root    4096 Oct  1 03:06 dbconfig-common
-rw-r--r--  1 root              root     556 Oct 15 02:00 alternatives.log.4.gz
-rw-r--r--  1 root              root    1295 Oct 18 18:19 dpkg.log.4.gz
-rw-r--r--  1 root              root     285 Nov 30 13:57 alternatives.log.3.gz
-rw-r--r--  1 root              root   32032 Nov 30 14:35 faillog
-rw-r--r--  1 root              root    8989 Nov 30 14:36 dpkg.log.3.gz
-rw-r--r--  1 root              root     261 Dec 30 02:33 alternatives.log.2.gz
-rw-r--r--  1 root              root    2381 Dec 30 02:33 dpkg.log.2.gz
-rw-rw-r--  1 root              utmp  292292 Jan 12 08:18 lastlog
-rw-r--r--  1 root              root   15572 Jan 16 04:41 dpkg.log.1
-rw-------  1 root              utmp     768 Jan 18 08:19 btmp.1
-rw-rw-r--  1 root              utmp   26880 Jan 22 05:47 wtmp.1
-rw-r--r--  1 root              root     464 Feb  1 09:18 alternatives.log.1
-rw-r-----  1 root              adm     2724 Feb  2 12:13 debug.4.gz
-rw-r-----  1 root              adm    72135 Feb  3 01:02 kern.log.4.gz
-rw-r-----  1 root              adm     5899 Feb  3 01:39 auth.log.4.gz
-rw-r-----  1 root              adm     6734 Feb  3 02:07 user.log.4.gz
-rw-r-----  1 root              adm    79151 Feb  3 02:07 messages.4.gz
-rw-r-----  1 root              adm    29536 Feb  3 02:07 daemon.log.4.gz
-rw-r--r--  1 root              root    3140 Feb  4 19:03 fontconfig.log
-rw-r-----  1 root              adm     5968 Feb  8 17:25 debug.3.gz
-rw-r-----  1 root              adm   120159 Feb 10 00:19 kern.log.3.gz
-rw-r-----  1 root              adm     7759 Feb 10 01:39 auth.log.3.gz
-rw-r-----  1 root              adm     9313 Feb 10 02:08 user.log.3.gz
-rw-r-----  1 root              adm   123322 Feb 10 02:08 messages.3.gz
-rw-r-----  1 root              adm    63724 Feb 10 02:08 daemon.log.3.gz
-rw-r-----  1 root              adm     7065 Feb 16 17:13 debug.2.gz
-rw-r-----  1 root              adm     8061 Feb 17 02:39 auth.log.2.gz
-rw-r-----  1 root              adm   120942 Feb 17 03:03 kern.log.2.gz
-rw-r-----  1 root              adm     8048 Feb 17 03:05 user.log.2.gz
-rw-r-----  1 root              adm    51792 Feb 17 03:05 daemon.log.2.gz
-rw-r-----  1 root              adm   120647 Feb 17 03:05 messages.2.gz
-rw-r--r--  1 root              root     879 Feb 19 14:06 alternatives.log
drwxr-xr-x  2 root              root    4096 Feb 19 14:07 wicd
-rw-r-----  1 root              adm    53456 Feb 20 03:06 syslog.7.gz
-rw-r-----  1 root              adm    40401 Feb 21 03:06 syslog.6.gz
-rw-r-----  1 root              adm    15238 Feb 22 03:06 syslog.5.gz
drwxr-xr-x  2 root              root    4096 Feb 22 15:01 apt
-rw-r--r--  1 root              root  117117 Feb 22 15:02 dpkg.log
-rw-r-----  1 root              adm   186472 Feb 23 03:05 syslog.4.gz
-rw-r-----  1 root              adm   210730 Feb 23 13:51 debug.1
-rw-------  1 root              utmp    1152 Feb 23 19:18 btmp
-rw-r-----  1 root              adm   136127 Feb 24 02:39 auth.log.1
-rw-r-----  1 root              adm  1727437 Feb 24 02:41 kern.log.1
drwxr-xr-x  2               119  125    4096 Feb 24 03:08 clamav
-rw-r-----  1 root              adm  1041864 Feb 24 03:08 daemon.log.1
-rw-r-----  1 root              adm   133376 Feb 24 03:08 user.log.1
-rw-r-----  1 root              adm   128037 Feb 24 03:08 syslog.3.gz
-rw-r-----  1 root              adm  1643189 Feb 24 03:08 messages.1
-rw-r--r--  1 root              root   32111 Feb 24 11:31 Xorg.2.log.old
-rw-r--r--  1 root              root   30462 Feb 24 12:44 Xorg.2.log
-rw-r-----  1 root              adm    57155 Feb 25 03:06 syslog.2.gz
drwxr-x---  2 root              adm     4096 Feb 26 03:07 apache2
drwxr-xr-x  2 root              root    4096 Feb 26 03:07 cups
-rw-r-----  1 root              adm   718763 Feb 26 03:07 syslog.1
drwxr-s---  2               118 adm     4096 Feb 26 03:07 mysql
-rw-r--r--  1 root              root   30002 Feb 26 20:05 Xorg.1.log.old
-rw-r--r--  1 root              root   53701 Feb 26 20:37 Xorg.0.log.old
-rw-r-----  1 root              adm   126670 Feb 26 20:38 debug
drwxr-xr-x 13 root              root    4096 Feb 26 21:41 .
drwx--x--x  2 root              root    4096 Feb 26 21:41 lightdm
-rw-r--r--  1 root              root   30001 Feb 26 22:41 Xorg.1.log
-rw-r-----  1 root              adm    42648 Feb 27 02:05 user.log
-rw-r-----  1 root              adm   905620 Feb 27 02:39 messages
-rw-r-----  1 root              adm   997117 Feb 27 02:39 kern.log
-rw-r-----  1 root              adm    64229 Feb 27 02:39 auth.log
-rw-rw-r--  1 root              utmp  104832 Feb 27 02:39 wtmp
-rw-r-----  1 root              adm   563463 Feb 27 02:39 syslog
-rw-r-----  1 root              adm   663042 Feb 27 02:39 daemon.log
-rw-r--r--  1 root              root   65656 Feb 27 02:39 Xorg.0.log

 

root@debian:~# grep -i "error\|warning\|panic" /var/log/syslog
Feb 27 23:40:11 localhost kernel: [    0.348055] acpi PNP0A08:00: _OSC failed (AE_ERROR); disabling ASPM
Feb 27 23:40:11 localhost kernel: [    8.131872] random: 7 urandom warning(s) missed due to ratelimiting
Feb 27 23:40:11 localhost kernel: [   37.766759] usb 2-1.2: device descriptor read/64, error -32
Feb 27 23:40:11 localhost kernel: [   38.370763] usb 2-1.2: device not accepting address 4, error -32
Feb 27 23:40:17 localhost kernel: [   57.979722] ACPI Warning: SystemIO range 0x0000000000000428-0x000000000000042F conflicts with OpRegion 0x0000000000000400-0x000000000000047F (\PMIO) (20160831/utaddress-247)
Feb 27 23:40:17 localhost kernel: [   57.979736] ACPI Warning: SystemIO range 0x0000000000000540-0x000000000000054F conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20160831/utaddress-247)
Feb 27 23:40:17 localhost kernel: [   57.979742] ACPI Warning: SystemIO range 0x0000000000000530-0x000000000000053F conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20160831/utaddress-247)
Feb 27 23:40:17 localhost kernel: [   57.979749] ACPI Warning: SystemIO range 0x0000000000000500-0x000000000000052F conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20160831/utaddress-247)
Feb 27 23:40:17 localhost lightdm[737]: Error getting user list from org.freedesktop.Accounts: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.Accounts was not provided by any .service files
Feb 27 23:40:18 localhost lightdm[737]: Could not enumerate user data directory /var/lib/lightdm/data: Error opening directory '/var/lib/lightdm/data': No such file or directory
Feb 27 23:40:43 localhost lightdm[931]: Error getting user list from org.freedesktop.Accounts: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.Accounts was not provided by any .service files
Feb 27 23:53:41 localhost kernel: [  861.965399] sr 2:0:0:0: [sr0] tag#1 Sense Key : Medium Error [deferred] 
Feb 27 23:53:41 localhost kernel: [  861.965406] sr 2:0:0:0: [sr0] tag#1 Add. Sense: L-EC uncorrectable error
Feb 27 23:53:41 localhost kernel: [  861.965419] blk_update_request: I/O error, dev sr0, sector 256564
Feb 27 23:53:48 localhost kernel: [  868.983900] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: errors=remount-ro

 

root@debian:~# grep -i "error\|warning\|panic" /var/log/kern.log
Feb 27 23:40:11 localhost kernel: [    0.348055] acpi PNP0A08:00: _OSC failed (AE_ERROR); disabling ASPM
Feb 27 23:40:11 localhost kernel: [    8.131872] random: 7 urandom warning(s) missed due to ratelimiting
Feb 27 23:40:11 localhost kernel: [   37.766759] usb 2-1.2: device descriptor read/64, error -32
Feb 27 23:40:11 localhost kernel: [   38.370763] usb 2-1.2: device not accepting address 4, error -32
Feb 27 23:40:17 localhost kernel: [   57.979722] ACPI Warning: SystemIO range 0x0000000000000428-0x000000000000042F conflicts with OpRegion 0x0000000000000400-0x000000000000047F (\PMIO) (20160831/utaddress-247)
Feb 27 23:40:17 localhost kernel: [   57.979736] ACPI Warning: SystemIO range 0x0000000000000540-0x000000000000054F conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20160831/utaddress-247)
Feb 27 23:40:17 localhost kernel: [   57.979742] ACPI Warning: SystemIO range 0x0000000000000530-0x000000000000053F conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20160831/utaddress-247)
Feb 27 23:40:17 localhost kernel: [   57.979749] ACPI Warning: SystemIO range 0x0000000000000500-0x000000000000052F conflicts with OpRegion 0x0000000000000500-0x0000000000000563 (\GPIO) (20160831/utaddress-247)
Feb 27 23:53:41 localhost kernel: [  861.965399] sr 2:0:0:0: [sr0] tag#1 Sense Key : Medium Error [deferred] 
Feb 27 23:53:41 localhost kernel: [  861.965406] sr 2:0:0:0: [sr0] tag#1 Add. Sense: L-EC uncorrectable error
Feb 27 23:53:41 localhost kernel: [  861.965419] blk_update_request: I/O error, dev sr0, sector 256564
Feb 27 23:53:48 localhost kernel: [  868.983900] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: errors=remount-ro

 

Missing /bin directory

The system does not boot because its /bin directory is missing (at least). That directory normally contains vital binaries, for example mount, which are necessary for bootup.

 

Solution

Some ways to try to recover /bin:

  • look at /lost+found (unfortunately is empty)
  • copying /bin from Live CD

Amazingly, after copying the /bin from Live CD the system booted nicely without any display error message.
Although copying binaries from the Live CD worked, it is alien stuff. Therefore, it is recommended a reinstall instead.

 

Current Status

Currently it is unknown why the /bin directory went missing. One possible explanation is that, a while ago, there was an issue of full disk crashing the system (trash grew too large unnoticed: didn't know lxde doesn't limit that). I've thought I'd already gotten over it: I've emptied the trash and deleted some unimportant corrupted files. After that the computer run fine for a couple of days until, suddenly, the network interface crashed forcing me to do a hard reset as explained in the initial post.

Could this full disk crash be the reason of the missing /bin?
If not, what else could be? How to find out?

  • Check /hddtest/var/log/syslog for messages from the last HDD boot. Is the /hddtest/bin/mount binary present? freedesktop.org/wiki/Software/systemd/Debugging gives you further debugging hints, also try following those. – Ferenc Wágner Feb 28 at 9:13
  • @FerencWágner I've just tried to boot via HDD right now. However, the last entry at /hddtest/var/log/syslog is still from feb 26 (2 days ago). Therefore, it seems that the system is no longer able to write into the syslog. – Mark Messa Feb 28 at 17:11
  • @FerencWágner There is no /hddtest/bin directory. The closest one is /hddtest/sbin, which happens to have several mount* files. – Mark Messa Feb 28 at 17:14
  • @FerencWágner > "freedesktop.org/wiki/Software/systemd/Debugging gives you further debugging hints" Unfortunately I'm not able to boot via HDD, the best I could do was booting via Live CD. Therefore, most of the recommended debugging commands with journalctl and systemctl are not available to me. – Mark Messa Feb 28 at 17:27
1

Your system does not boot because its /bin directory is missing (at least). That directory normally contains vital binaries, for example mount, which are necessary for bootup. How it went missing is hard to tell, but your syslog (which really should end at Feb 27 02:39 according to the directory listing in your question) or other neighboring log files might shed some light into that (these are not touched by the Live CD). Exactly when did you start experiencing problems? If there's absolutely nothing in the logs, then I'd suspect an intermittent HDD or controller failure. That fsck reports /dev/sda1 clean is not consistent with this theory, but it may have been repaired during an earlier bootup attempt. You might try to recover /bin from /lost+found (if there is anything in that directory), but a reinstall may be simpler. Also, I'd give some stress test to that HDD from the Live system before trusting it with anything important. (Think extracting and removing the Linux source tree a couple of times and so on.)

  • > "Exactly when did you start experiencing problems?" A week ago I had an issue of full disk crashing the system (trash grew too large unnoticed: didn't know lxde doesn't limit that). I've thought I'd already gotten over it: I've emptied the trash and deleted some unimportant corrupted files. After that the computer run fine for a couple of days until, suddenly, the network interface crashed forcing me to do a hard reset as explained in the initial post. Do you think this full disk crash might be the reason? Could it make an entire /bin directory disappear? – Mark Messa Mar 1 at 16:34
  • > "your syslog [...] or other neighboring log files might shed some light into that" Which neighboring files would you recommend starting with? What else do you recommend look for in syslog? – Mark Messa Mar 1 at 16:55
  • > "That fsck reports /dev/sda1 _clean is not consistent with this theory"_ Why not? Does fsck check for /bin directory? Maybe it only check the data structure consistency ... – Mark Messa Mar 1 at 17:02
  • > "You might try to recover /bin from /lost+found (if there is anything in that directory)" You are correct, /lost+found is empty. – Mark Messa Mar 1 at 17:03
  • > "I'd give some stress test to that HDD from the Live system" Do you recommend any tool to perform automatically such stress test? – Mark Messa Mar 1 at 17:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.