2

I have a server running on Debian Wheezy 3.2.68-1+deb7u1 x86_64. Lately strange records in /var/log/messages appeared. First messages of this log starts just after server reboot.

6.222.18 LEN=44 TOS=0x00 PREC=0x00 TTL=58 ID=61546 PROTO=TCP SPT=80 DPT=42918 WINDOW=16384 RES=0x00 ACK SYN URGP=0 
Jun  6 08:02:49 s02 kernel: [   29.615405] iptables-input-drop: IN=eth0 OUT= MAC=d4:3d:7e:ed:0a:9a:3c:94:d5:4a:e5:03:08:00 SRC=104.31.176.10 DST=144.
76.222.18 LEN=44 TOS=0x00 PREC=0x00 TTL=58 ID=0 DF PROTO=TCP SPT=80 DPT=24586 WINDOW=29200 RES=0x00 ACK SYN URGP=0 
Jun  6 08:03:02 s02 rsyslogd-2177: imuxsock begins to drop messages from pid 6754 due to rate-limiting
Jun  6 08:03:06 s02 kernel: [   46.752749] iptables-input-drop: IN=lxc OUT= MAC=01:00:5e:00:00:01:fe:12:86:6b:0f:b8:08:00 SRC=0.0.0.0 DST=224.0.0.1 L
EN=32 TOS=0x00 PREC=0xC0 TTL=1 ID=0 DF PROTO=2 
Jun  6 08:03:11 s02 kernel: 472]pbsntr:Nt TM=::::::::::::: C01.58D=4621L= Sx E00T5I0FROCS= T46IO20R=0A NR=
Jun  6 08:03:43 s02 kernel: 4 89 cnolf=o,doe
Jun  6 08:03:43 s02 kernel: 8apamneo<82:, 2a3-#a6b>.]T<84ff6da7d 6[f4?_3
Jun  6 08:03:43 s02 kernel: 1<f8 ngx9 6[f1?slqex 7 fb kox7 7 f3 lh/430f8feuor/433f88eul0x>.r+f 6[f6?__a3d 7 fb ree7f 6[f6?ghoxc 6[f1?ea1b 7 fc ok03 7
 f5 al0
Jun  6 08:03:43 s02 kernel: 9<f6 a/436f85al/639 c.dtdrotxhoe 7m 1ki5f 634r: i04kl<82-<87e u 7C0 t1 [9   b  >.] :  :438 i, d<84 h0:s
Jun  6 08:03:43 s02 kernel: 6U  hu0 7P: c   7C7 t1 [9  p: 7P:1c 6 7C1 t1 [9   b3 >.] :  :437 i, d<88 h6:s
Jun  6 08:03:43 s02 kernel: 1U 8hu9 8P:1c   7NNp: 8P:1c 2 7C1 t1 [9   b3 >.] :  :433 i, d<86 h6:s
Jun  6 08:03:43 s02 kernel: 6U 8hu5 8P:1c 5 7aa7cn3tn[9 e3n_5s_4 8ua4t6e5a<81e4bi1leb7 8m3h3t5u<80e 5mB0haaBinci v:elid:ofke8okyrkakmlleleb _0e:s0n a
:gnaeb
Jun  6 08:04:01 s02 rsyslogd-2177: imuxsock lost 1237 messages from pid 6754 due to rate-limiting
Jun  6 08:04:02 s02 rsyslogd-2177: imuxsock begins to drop messages from pid 6754 due to rate-limiting
Jun  6 08:04:11 s02 rsyslogd-2177: imuxsock lost 1322 messages from pid 6754 due to rate-limiting
Jun  6 08:05:02 s02 rsyslogd-2177: imuxsock begins to drop messages from pid 6754 due to rate-limiting
Jun  6 08:05:11 s02 rsyslogd-2177: imuxsock lost 1446 messages from pid 6754 due to rate-limiting
Jun  6 08:08:11 s02 kernel: 2ws:2 4>  1l_e0 <89eme3i4:B9 _5 v: _4 v:2vekao e) t6l4 1ke4a4 7scl0blekekBa2ueokempc0nmn 8lr[ >.]00*1361*0 k44=k 8ND18075
9 Bk22864B 2oo2*1361:ea:95:30C.27=72L 0R0==PTTP2W 0SP:0 0N=ETDD==PN< .1anr=U=7a:43 46T684xCTI T==W2E N06 et OSFA:f3:8=20LOP0 5TP LhA:0:10C.42=0CT=FIE
=6 = v9e6:76 0S06x=L3 D9=4C2=.L xE062DT 3PL DU=DL
Jun  6 08:10:40 s02 kernel: :0939da5:0121T784x=L0TPTW9=KGv e6f2098R0D. 30R0== OP3D 340]tst:c =Of:83:0.S6 T 049TTO=Qe35:051D..==R015OP93 -=UYe7=8f:27 
.D..==R049PC=08
Jun  6 08:11:40 s02 kernel: 0M3d:d5:12S6 T 08FT85D00Y0cT IexM:::::::C.T.NSP069 U=DL
Jun  6 08:14:11 s02 kernel: hAe:bE066 D8=4<4>[  620.699091] iptables-input-drop: IN=xc OUT HSNezO C:8::0:0.S6 T 048O CDEUNx=6b:900D.=0CT= D5=4>60ilnd
 xTYvxM:::::::C.S.2ET0C 44FO 8=3EM3e99400=48=61=S 0TIFOS 6I2R0 R
Jun  6 08:22:40 s02 kernel: 01. 1.L3= = 6=8P=P45= et  He7f6:b1:S0S7180E  6RM==61>056plppxPeL:b5:7S.=2ESR I OYD5=4 867pe-IUIrC8:ea:1 0N=ETDD==PNrO IhM
e:8:700 0N=ETDD==PNaiudpIl =HNz e:be:8=2.ESR I OTDE07]tsu:cPvO=8fb10C.0LOP0 9TP L7:68:a8C0=.  0=TD T   3 UNrC8:ea:1 0N=ETDD==PNPtLe:be:0R.T684xCTI0OP
8001.016=PxLD OP5D 3pO=e9:5071=2ESR IPP94OES=eTde:9a:S4D78 0x58R 03 71:22:R01 T 043OS8 
Jun  6 08:23:40 s02 kernel: 0 7. 42ESR IPP 4WR  
Jun  6 08:25:13 s02 kernel: a .D0N=ETDD==PNE044OTDE::a8:R01 T 047OS6 
Jun  6 08:28:11 s02 kernel: 0L3= =T43DT 5TN430C23D4.86=P053==PN cUPIerLC268e:0R.T68400=9RP I =6f2098R0D. 30R0== OP4D 3r=UYeOC2b8e:0C.42=0CT=FIE=11N=NrA2:53a0S0D0EO x=5F=T
Jun  6 08:37:40 s02 kernel: pr===Of:83:0.S06x=L5 D2=44]aidNTNx=6b:900D.=0CT=FU4T=66tir=TIzM1b:39:=.12LOP0 2TTO=E8TTO=EerO Ce26bf::31:00C..S462 =T0 C0T44 RIT8ED01:a:e8C. .1400= OT=I20 P hM20:10C.42=0CT=FIE=01TeL:b5:7S.=1 0x60R 33 :27 .D..==R049PCEE1EE=PxLD1RD= 34Uv7e6:05 0T.LOP0  OYD1Q 8:ea:1 0N=ETDD==PNcSzA:fb:8=2.ESR I OTDE::0a:0=.D0.NTxRxT 4DODT9TL3Of8fb:8=24.8x=L1 C=03 bnrNTIrC8:ea:1 .1400=3PM8 5
Jun  6 08:38:14 s02 kernel: 99bnpcHt 1:22:R012N=ETDFT3TI60 
Jun  6 08:45:35 s02 kernel: [ 2178.791699] ipabei-:lUH=wM:::6:8=30LOP0 0TP LD6RUT85=t- x=IrC8:ea:1 0N=ETDD==PN<4>[ 2299.711247] iptables-input-drop: IN=lxcO=Y=zL=26:23:70=0T7.NO0=T 2PCEE1Eo=OPNh7C160:e25:C.S0L xE062DT 9PL 71R0SPsd===Hf:8c:0.D.=0CT=FU4T=SL:b5:7S.=2ESR I OYD1Q4d3:3S1742=0CT=OSP 2SCRz=:fb:8=2.ESR I OTDERL5 D4=4===of:8e:0.S06x=L4 D7=46:::5 0T.300=7PP55327ap: YhM20:10C.0LOP0 1TP LN7f6:b1:R.=.60E  7RP9TN456rl v7e6:05 0T.300=7PP353OHNtx =16:b2:29:0R00D1. =T0 C0L 7 O 5 5N

I have a rule in iptables for droping packets:

-A INPUT -m limit --limit 2/min -j LOG --log-prefix "iptables-input-drop: " --log-level 4

This rule causes first few lines, afterwards the log is pretty much messy. Could it be caused by some malformed packets? Is it some sort of attack or rather hardware issue (corrupted memory etc.)?

UPDATE:

I turned off rsyslog limiting $SystemLogRateLimitInterval 0 to avoid dropping messages. But still aren't many readable messages:

Jun  8 04:32:15 s02 kernel: 05[f7?h01[.]f1>c+x[.]f1>seu/404ff0_ice0
Jun  8 04:32:15 s02 kernel: 05[f7?ic4166 f3 lh/401ff1mo_o4<14<ff]___1816 f0]_ue0>13f87rnksx<15<fa ctr2266 ff u_+x[.]f0>aek+xt[.]f1>mp_
+x[.]f1>dged0>10f8f_c+x[.]f1>dgex666 fb p1
Jun  8 04:32:15 s02 kernel: 07[f5?_g2
Jun  8 04:32:15 s02 kernel: 07[ff?obr1 
Jun  8 04:32:15 s02 kernel: 07[f5?i_ee/<12<fd ahax1[.]f1>_+x[.]f1>ni_x5[.]f0>t_0xx>13f84orb
Jun  8 04:32:15 s02 kernel: 07[fb?ef<13<f6 a/407ff7scs0116 nr..l li cmn[.]ye2i4,n5[.]y 0m77Bc408-<15dAp[.] :  :403 h0:s
Jun  8 04:32:15 s02 kernel: 07P: c  16   b  >19 i, d<16U  hu066C6 t1 [.] :  :407e3c>17 i, d<17U 8hu366C2 t11[.] :  :407 h6:s
Jun  8 04:32:15 s02 kernel: 07P:1c 816   b3 >17 i, d<18drr<18U 8hu166C1 t11[.] :  :403 h6:s
Jun  8 04:32:15 s02 kernel: 07P:1c 516   b3 >15 i, d<10U 8hu466aa7n_0ln403tl9ci3of
Jun  8 04:32:15 s02 kernel: 08ua6y1b6a<10r4_m1leb216 d 2ae6e[.]0r0nlB4toie0ien_ki:oakal tkk 0t0p 0bi0bleetBakaBerkkselle<12ws:2 <13dAe6: 0g4in4cnkv:ktl3nb1oakalke4m:B:Bb2p2ek_m4sra6rakt2ueokempc0nmn16 _e0 <15dre4:B5h6cn5ie2 _9Bie6nb0lnBteBn7m: 1ke9m2B:Bra4sra5et2ae0tkcwc0enlcl<16ws: 16  *862422*1B80 B66ND2 B6*23Bk213B91B66NN B8*3 k855* k97
Jun  8 04:32:15 s02 kernel: 095tae
Jun  8 04:32:15 s02 kernel: 09 i <10ae:,en<10e 
Jun  8 04:32:15 s02 kernel: 09oaB<4>[160101.733911] 8383984 pages RAM
Jun  8 04:32:15 s02 kernel: <>[160101.733929] 164294 ae eee<67]2ga41313ss
Jun  8 04:32:15 s02 kernel: 67]dugt ruamen614[]082 3 0  i[190 1 9     0
Jun  8 04:32:15 s02 kernel: 67]1 9  25- -s614[]053 5 0  t>021          s<03 6 4       u614[]06  0 0  l614[]349 9 0  s>031   4      g[138 1 8     0i1.03 8 38     
Jun  8 04:32:15 s02 kernel: 67]7 3  86   u<03 9376 4    w614[]309 6 0  s>061   8      g[158 1 4     0i1.63 8 06     
Jun  8 04:32:15 s02 kernel: 67]8 3  06   u<03 5386 9    w614[]360 4 0  s>081   8      g[168 1 4     0i1.23 8 40     
Jun  8 04:32:15 s02 kernel: 67]9 3  73   u<03 1397 6    w615[]321 7 0  s>001   5      g[188 1 01    0i1.03 8 29     
Jun  8 04:32:15 s02 kernel: 67]9 3  44   u<03 7398 6    w615[]382 4 0  s>021   5      g[108 1 11    0i1.64 8 72     
Jun  8 04:32:15 s02 kernel: 67]0 4  65   u<03 9309 2    w615[]326 5 0  s>041   2      g[118 1 01    0i1.34 8 14     
Jun  8 04:32:15 s02 kernel: 67]1 4  95   u<03 2348 2    w615[]356 9 0  s>012   3      g[127 2 31    0i1.84 7 33     
Jun  8 04:32:24 s02 kernel: 67]7 0  13   u<03 9 5  8    r>04    0      <03 5 1  4    ut[185   4     0o<03 7 7  0    ot[165 1 2     0i1.0orum: s1sc9acl[15lr  ia3ko:kl:B062]oc=pgt_e413P8oot e.m1a.d
Jun  8 04:32:24 s02 kernel: 62] :1.3ff7 pe8d1.5ff4 s+0413 f1ctnc+0413 f17_plroe<08 f84 io0x413 f18s_e/<08 f81 go_y/<08 f8f gh_xx414 f1apas+0414 f17_plroe<08 f81 _pc+/

I've noticed similar behaviour on other 2 servers (same hardware). It's possible that all 3 servers have corrupted memory, but all applications are still running fine and repeated memtester checks didn't show any issues (on other servers I was always able to reproduce memory issues during tests).

UPDATE 2: I forgot to mention that on all 3 machines were LXC containers running. I didn't see similar issue on machine without LXC containers.

  • If you haven't just started to use an experimental driver, I'd definitely do a RAM test. – Gilles Jun 6 '15 at 18:29
  • No, there wasn't any driver change lately. All drivers are stable, the server had uptime over 200 days until yesterday. I'm using memtester utility for checking memory on running servers. I did a test run memtester 8GB 1 which didn't show any problems. But there are 32GB RAM on that server, I can try multiple runs on bigger amounts of memory. Are there other options how to test memory? – Tombart Jun 7 '15 at 11:47
  • Memtest86+ is the industry standard. But it can't be user live, you have to reboot into it and let it run for one full pass. Does your server have ECC? What are its correction statistics? – Gilles Jun 7 '15 at 12:03
  • I know Memtest86+, but I'd like to avoid downtime, if possible. Modules are non-ECC 8GB Crucial. – Tombart Jun 7 '15 at 13:57
  • I'd definitely check out pid 6754 -- rsyslogd is complaining about getting buried by it. – Jeff Schaller Jun 8 '15 at 2:10
1

After reading this article: Linux kernel bug delivers corrupt TCP/IP, I thought these issues might be connected. Original veth driver appeared in kernel 2.6.24-rc1, the bug was introduced in 2010 with this patch (v2.6.37-rc8). Now a fix for this issue should be backported into kernels 3.14+, thus upgrading to one of those kernels might resolve this issue.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.