3

I have a strange issue on mikrotik rb951-2hnd router. I built image a few years ago using revision 39392 and patch firmwared it and everything worked fine. Just a few months ago I decided to update firmware. So I did it and discovered that physical network is completely broken. I have over 90% packet loss via ethernet ports, though wifi works perfect . I thought that I messed up while build. So I firmwared 2 different images from download.openwrt and a couple more that I built by myself, but symptoms are always the same. I wanted to try worked for me svn revision but unfortunately this patch is unavailable so I can't firmare image back. The funny thing that everything including physical ports works while netbooting via vmlinux-initramfs (bootp) the same build revision. Since vmlinux works fine I suspect that flash is damaged so I made myself sure that files from rootfs.tar.gz and firmwared ones are the same. On the next step I compared the loaded demons on vmlinux-initramfs and firmwared system, the firmwared one has extra listed below:

nf_log_common.ko
nf_log_ipv4.ko
nf_log_ipv6.ko
nf_nat_masquerade_ipv4.ko
nf_reject_ipv4
nf_reject_ipv4.ko
nf_reject_ipv6
nf_reject_ipv6.ko
nls_base.ko

Preventing them from loading doesn't help. Furthermore I get no errors from dmesg or logread. Here's my configuration:
top
iptables -L -n
/etc/config/network - 1 wan. 2-5 lan (wifi in sta client mode)

scenario for dhcp:

After plugging laptop (dhcp client) to lan port for 15 seconds ( plug out after) I see the followings:

  • Laptop send: 3 dhcp request and 9 icmpv6
  • Laptop receive: 0 packets
  • router sends: None ? (ifconfig displays 4 packets but tcpdump doesn't catch them)
  • router receives: 2 icmp packets from Laptop sent list (listed below)

I also checked tcpdump on router, and it doesn't show lost packets. Seems like the problem is somewhere on the driver level. But wait, vmlinux works and drivers (kernel modules) are the same.

root@OpenWrt:/# tcpdump -vv -i eth0.3
tcpdump: WARNING: eth0.3: no IPv4 address assigned
tcpdump: listening on eth0.3, link-type EN10MB (Ethernet), capture size 65535 bytes
[ 1042.060000] Atheros AR8216/AR8236/AR8316 ag71xx-mdio.0:00: Port 2 is up
09:35:24.172637 IP6 (hlim 1, next-header Options (0) payload length: 36) :: > ff02::16: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, m]
09:35:25.872843 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 16) fe80::b2e3:928a:66b2:ff43 > ff02::2: [icmp6 sum ok] ICMP6, router6
          source link-address option (1), length 8 (1): 5c:f9:dd:48:9e:89
            0x0000:  5cf9 dd48 9e89
5c:f9:dd:48:9e:89 /fe80::b2e3:928a:66b2:ff43 - laptop,  d4:ca:6d:92:a4:7e / fe80::d6ca:6dff:fe92:a47e: - router

scenario for static ip:

openwrt (static 192.168.2.1)
root@OpenWrt:/# ping 192.168.2.2
PING 192.168.2.2 (192.168.2.2): 56 data bytes
64 bytes from 192.168.2.2: seq=4 ttl=64 time=0.505 ms
64 bytes from 192.168.2.2: seq=21 ttl=64 time=0.489 ms
64 bytes from 192.168.2.2: seq=34 ttl=64 time=0.528 ms
64 bytes from 192.168.2.2: seq=39 ttl=64 time=0.512 ms
64 bytes from 192.168.2.2: seq=45 ttl=64 time=0.527 ms
64 bytes from 192.168.2.2: seq=48 ttl=64 time=0.549 ms
64 bytes from 192.168.2.2: seq=51 ttl=64 time=0.813 ms
^C
--- 192.168.2.2 ping statistics ---
56 packets transmitted, 7 packets received, 87% packet loss
round-trip min/avg/max = 0.489/0.560/0.813 ms
laptop (static 192.168.2.2)
14:50:08:andrew:/home/andrew:0
: ping 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
From 192.168.2.2 icmp_seq=13 Destination Host Unreachable
From 192.168.2.2 icmp_seq=14 Destination Host Unreachable
From 192.168.2.2 icmp_seq=15 Destination Host Unreachable
^C
--- 192.168.2.1 ping statistics ---
100 packets transmitted, 0 received, +3 errors, 100% packet loss, time 99022ms
pipe 3
14:51:53:andrew:/home/andrew:1
: ping 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
^C
--- 192.168.2.1 ping statistics ---
29 packets transmitted, 0 received, 100% packet loss, time 28080ms

So here is my question:

What steps should I take to dive deeper and find out what the problem is? I want to find at least the error. Should I turn a debug level somewhere? What can cause such a huge packet loss?

EDIT
Netbooting vmlinux produces the same
Netbooting kerneldebug - works perfectly, unfortunately its size is 11mb, mtd1 is to small to hold it.

  • Which version of OpenWRT were you using before and which one are you using now? I have the exact same router and using the stable Barrier Breaker (14.07) release I've never had a single networking issue. Haven't tried the Chaos Calmer (15.05) RC images yet - are these the ones you're playing with? As you wrote you are comfortable with building OpenWRT images yourself, try building a kernel with CONFIG_AG71XX_DEBUG set to enable debug messages. Note you might have to play with the ag71xx_msg_level parameter as well to get any output. Also, does top or free show anything suspicious? – Kempniu Jul 10 '15 at 8:32
  • I checked barrier breaker and chaos calmer from downloads.openwrt. I think i had been running some attitude_adjustment revision built by myself when everything worked. Free says that only 17M ram is used. Also I added top but cpu loaded only for <3%. I need to add CONFIG_AG71XX_DEBUG to .config and ag71xx_msg_level belongs to /package/boot/uboot-ar71xx/files/cpu/mips/ar71xx_serial.c ? – deathangel908 Jul 11 '15 at 8:14
  • Well, the sole fact that Barrier Breaker gives you issues while it's rock stable on an identical piece of hardware elsewhere is odd. I understand there is no chance you can revert to the previously used OpenWRT version? I'm wondering whether you'd be getting the same behavior after a downgrade. As for ag71xx_msg_level, I meant the one in target/linux/ar71xx/files/drivers/net/ag71xx/ag71xx_main.c. – Kempniu Jul 11 '15 at 10:37
  • Ah, I forgot to mention. When I updated openwrt (after 1st firmwared worked image) I move back to RouterOs ( to set boot option to LAN 1st to boot from initramfs) and then I firmwared openwrt. I'm not exactly sure but this could update the boot partition and cause the issue. I'm compiling the image with debug=16 for now, see what it changes. – deathangel908 Jul 11 '15 at 11:01
  • Well, I'm definitely not a Mikrotik expert, so I can't tell you whether it could break something, but what I know for sure is that I never did a single thing in RouterOS, using the alternative method described in the OpenWRT wiki to force a one-time netboot. – Kempniu Jul 11 '15 at 11:13
0

Ok I finally found the reason. Initramfs wasn't the key of success. It turns out that routerboard works if it was booted with pressed reset button. But it seems like flashed driver can't last for long. Router reboots with kernel panic after a few hours of using depending on loading rate.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.