Huge packet loss on openwrt

Question

I have a strange issue on mikrotik rb951-2hnd router. I built image a few years ago using revision 39392 and patch firmwared it and everything worked fine. Just a few months ago I decided to update firmware. So I did it and discovered that physical network is completely broken. I have over 90% packet loss via ethernet ports, though wifi works perfect . I thought that I messed up while build. So I firmwared 2 different images from download.openwrt and a couple more that I built by myself, but symptoms are always the same. I wanted to try worked for me svn revision but unfortunately this patch is unavailable so I can't firmare image back. The funny thing that everything including physical ports works while netbooting via vmlinux-initramfs (bootp) the same build revision. Since vmlinux works fine I suspect that flash is damaged so I made myself sure that files from rootfs.tar.gz and firmwared ones are the same. On the next step I compared the loaded demons on vmlinux-initramfs and firmwared system, the firmwared one has extra listed below:

nf_log_common.ko
nf_log_ipv4.ko
nf_log_ipv6.ko
nf_nat_masquerade_ipv4.ko
nf_reject_ipv4
nf_reject_ipv4.ko
nf_reject_ipv6
nf_reject_ipv6.ko
nls_base.ko

Preventing them from loading doesn't help. Furthermore I get no errors from dmesg or logread. Here's my configuration:
top
iptables -L -n
/etc/config/network - 1 wan. 2-5 lan (wifi in sta client mode)

scenario for dhcp:

After plugging laptop (dhcp client) to lan port for 15 seconds ( plug out after) I see the followings:

• Laptop send: 3 dhcp request and 9 icmpv6
• Laptop receive: 0 packets
• router sends: None ? (ifconfig displays 4 packets but tcpdump doesn't catch them)
• router receives: 2 icmp packets from Laptop sent list (listed below)

I also checked tcpdump on router, and it doesn't show lost packets. Seems like the problem is somewhere on the driver level. But wait, vmlinux works and drivers (kernel modules) are the same.

root@OpenWrt:/# tcpdump -vv -i eth0.3
tcpdump: WARNING: eth0.3: no IPv4 address assigned
tcpdump: listening on eth0.3, link-type EN10MB (Ethernet), capture size 65535 bytes
[ 1042.060000] Atheros AR8216/AR8236/AR8316 ag71xx-mdio.0:00: Port 2 is up
09:35:24.172637 IP6 (hlim 1, next-header Options (0) payload length: 36) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, m]
09:35:25.872843 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 16) fe80::b2e3:928a:66b2:ff43 > ff02::2: [icmp6 sum ok] ICMP6, router6
          source link-address option (1), length 8 (1): 5c:f9:dd:48:9e:89
            0x0000:  5cf9 dd48 9e89
5c:f9:dd:48:9e:89 /fe80::b2e3:928a:66b2:ff43 - laptop,  d4:ca:6d:92:a4:7e / fe80::d6ca:6dff:fe92:a47e: - router

scenario for static ip:

openwrt (static 192.168.2.1)
root@OpenWrt:/# ping 192.168.2.2
PING 192.168.2.2 (192.168.2.2): 56 data bytes
64 bytes from 192.168.2.2: seq=4 ttl=64 time=0.505 ms
64 bytes from 192.168.2.2: seq=21 ttl=64 time=0.489 ms
64 bytes from 192.168.2.2: seq=34 ttl=64 time=0.528 ms
64 bytes from 192.168.2.2: seq=39 ttl=64 time=0.512 ms
64 bytes from 192.168.2.2: seq=45 ttl=64 time=0.527 ms
64 bytes from 192.168.2.2: seq=48 ttl=64 time=0.549 ms
64 bytes from 192.168.2.2: seq=51 ttl=64 time=0.813 ms
^C
--- 192.168.2.2 ping statistics ---
56 packets transmitted, 7 packets received, 87% packet loss
round-trip min/avg/max = 0.489/0.560/0.813 ms
laptop (static 192.168.2.2)
14:50:08:andrew:/home/andrew:0
: ping 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
From 192.168.2.2 icmp_seq=13 Destination Host Unreachable
From 192.168.2.2 icmp_seq=14 Destination Host Unreachable
From 192.168.2.2 icmp_seq=15 Destination Host Unreachable
^C
--- 192.168.2.1 ping statistics ---
100 packets transmitted, 0 received, +3 errors, 100% packet loss, time 99022ms
pipe 3
14:51:53:andrew:/home/andrew:1
: ping 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
^C
--- 192.168.2.1 ping statistics ---
29 packets transmitted, 0 received, 100% packet loss, time 28080ms

So here is my question:

What steps should I take to dive deeper and find out what the problem is? I want to find at least the error. Should I turn a debug level somewhere? What can cause such a huge packet loss?

EDIT
Netbooting vmlinux produces the same
Netbooting kerneldebug - works perfectly, unfortunately its size is 11mb, mtd1 is to small to hold it.

Answer 1

Ok I finally found the reason. Initramfs wasn't the key of success. It turns out that routerboard works if it was booted with pressed reset button. But it seems like flashed driver can't last for long. Router reboots with kernel panic after a few hours of using depending on loading rate.