I have a strange issue on mikrotik rb951-2hnd router. I built image a few years ago using revision 39392 and patch firmwared it and everything worked fine. Just a few months ago I decided to update firmware. So I did it and discovered that physical network is completely broken. I have over 90% packet loss via ethernet ports, though wifi works perfect . I thought that I messed up while build. So I firmwared 2 different images from download.openwrt and a couple more that I built by myself, but symptoms are always the same. I wanted to try worked for me svn revision but unfortunately this patch is unavailable so I can't firmare image back. The funny thing that everything including physical ports works while netbooting via vmlinux-initramfs (bootp) the same build revision. Since vmlinux works fine I suspect that flash is damaged so I made myself sure that files from rootfs.tar.gz and firmwared ones are the same. On the next step I compared the loaded demons on vmlinux-initramfs and firmwared system, the firmwared one has extra listed below:
nf_log_common.ko nf_log_ipv4.ko nf_log_ipv6.ko nf_nat_masquerade_ipv4.ko nf_reject_ipv4 nf_reject_ipv4.ko nf_reject_ipv6 nf_reject_ipv6.ko nls_base.ko
Preventing them from loading doesn't help. Furthermore I get no errors from dmesg or logread. Here's my configuration:
iptables -L -n
/etc/config/network - 1 wan. 2-5 lan (wifi in sta client mode)
scenario for dhcp:
After plugging laptop (dhcp client) to lan port for 15 seconds ( plug out after) I see the followings:
- Laptop send: 3 dhcp request and 9 icmpv6
- Laptop receive: 0 packets
- router sends: None ? (ifconfig displays 4 packets but tcpdump doesn't catch them)
- router receives: 2 icmp packets from Laptop sent list (listed below)
I also checked tcpdump on router, and it doesn't show lost packets. Seems like the problem is somewhere on the driver level. But wait, vmlinux works and drivers (kernel modules) are the same.
root@OpenWrt:/# tcpdump -vv -i eth0.3 tcpdump: WARNING: eth0.3: no IPv4 address assigned tcpdump: listening on eth0.3, link-type EN10MB (Ethernet), capture size 65535 bytes [ 1042.060000] Atheros AR8216/AR8236/AR8316 ag71xx-mdio.0:00: Port 2 is up 09:35:24.172637 IP6 (hlim 1, next-header Options (0) payload length: 36) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, m] 09:35:25.872843 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 16) fe80::b2e3:928a:66b2:ff43 > ff02::2: [icmp6 sum ok] ICMP6, router6 source link-address option (1), length 8 (1): 5c:f9:dd:48:9e:89 0x0000: 5cf9 dd48 9e89 5c:f9:dd:48:9e:89 /fe80::b2e3:928a:66b2:ff43 - laptop, d4:ca:6d:92:a4:7e / fe80::d6ca:6dff:fe92:a47e: - router
scenario for static ip:
openwrt (static 192.168.2.1) root@OpenWrt:/# ping 192.168.2.2 PING 192.168.2.2 (192.168.2.2): 56 data bytes 64 bytes from 192.168.2.2: seq=4 ttl=64 time=0.505 ms 64 bytes from 192.168.2.2: seq=21 ttl=64 time=0.489 ms 64 bytes from 192.168.2.2: seq=34 ttl=64 time=0.528 ms 64 bytes from 192.168.2.2: seq=39 ttl=64 time=0.512 ms 64 bytes from 192.168.2.2: seq=45 ttl=64 time=0.527 ms 64 bytes from 192.168.2.2: seq=48 ttl=64 time=0.549 ms 64 bytes from 192.168.2.2: seq=51 ttl=64 time=0.813 ms ^C --- 192.168.2.2 ping statistics --- 56 packets transmitted, 7 packets received, 87% packet loss round-trip min/avg/max = 0.489/0.560/0.813 ms laptop (static 192.168.2.2) 14:50:08:andrew:/home/andrew:0 : ping 192.168.2.1 PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data. From 192.168.2.2 icmp_seq=13 Destination Host Unreachable From 192.168.2.2 icmp_seq=14 Destination Host Unreachable From 192.168.2.2 icmp_seq=15 Destination Host Unreachable ^C --- 192.168.2.1 ping statistics --- 100 packets transmitted, 0 received, +3 errors, 100% packet loss, time 99022ms pipe 3 14:51:53:andrew:/home/andrew:1 : ping 192.168.2.1 PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data. ^C --- 192.168.2.1 ping statistics --- 29 packets transmitted, 0 received, 100% packet loss, time 28080ms
So here is my question:
What steps should I take to dive deeper and find out what the problem is? I want to find at least the error. Should I turn a debug level somewhere? What can cause such a huge packet loss?
Netbooting vmlinux produces the same
Netbooting kerneldebug - works perfectly, unfortunately its size is 11mb, mtd1 is to small to hold it.