4

I have a process (the NewRelic infra agent) that is hung on a RHEL 7, xfs filesystem host. I'm thinking that it's trying to watch a file or maybe a socket that another process already has in use.

When I run strace on it, I get this:

strace: Process 18333 attached
read(6,

If I run lsof on it, I see this:

COMMAND     PID USER   FD      TYPE             DEVICE SIZE/OFF     NODE NAME
newrelic- 18333 root  cwd       DIR              253,3      236       64 /
newrelic- 18333 root  rtd       DIR              253,3      236       64 /
newrelic- 18333 root  txt       REG              253,3 15672136  9280340 /usr/bin/newrelic-infra
newrelic- 18333 root  mem       REG              253,3    68192 16892409 /usr/lib64/libbz2.so.1.0.6
newrelic- 18333 root  mem       REG              253,3   157400 16892537 /usr/lib64/liblzma.so.5.2.2
newrelic- 18333 root  mem       REG              253,3    90248 16891935 /usr/lib64/libz.so.1.2.7
newrelic- 18333 root  mem       REG              253,3   100008 16891900 /usr/lib64/libelf-0.172.so
newrelic- 18333 root  mem       REG              253,3    19896 16892403 /usr/lib64/libattr.so.1.1.0
newrelic- 18333 root  mem       REG              253,3    88776 16798006 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
newrelic- 18333 root  mem       REG              253,3    19288 16891827 /usr/lib64/libdl-2.17.so
newrelic- 18333 root  mem       REG              253,3   330464 16891809 /usr/lib64/libdw-0.172.so
newrelic- 18333 root  mem       REG              253,3    43776 17005232 /usr/lib64/librt-2.17.so
newrelic- 18333 root  mem       REG              253,3  1137024 16891840 /usr/lib64/libm-2.17.so
newrelic- 18333 root  mem       REG              253,3    20032 16892412 /usr/lib64/libcap.so.2.22
newrelic- 18333 root  mem       REG              253,3    86544 16891748 /usr/lib64/libnss_myhostname.so.2
newrelic- 18333 root  mem       REG              253,3   105824 17005229 /usr/lib64/libresolv-2.17.so
newrelic- 18333 root  mem       REG              253,3    31408 17005206 /usr/lib64/libnss_dns-2.17.so
newrelic- 18333 root  mem       REG              253,3    61624 17005208 /usr/lib64/libnss_files-2.17.so
newrelic- 18333 root  mem       REG              253,3  2151672 16891807 /usr/lib64/libc-2.17.so
newrelic- 18333 root  mem       REG              253,3   141968 17005223 /usr/lib64/libpthread-2.17.so
newrelic- 18333 root  mem       REG              253,3   163400 16798041 /usr/lib64/ld-2.17.so
newrelic- 18333 root    0r      CHR                1,3      0t0     1042 /dev/null
newrelic- 18333 root    1u     unix 0xffff9350a1c09c00      0t0  1317098 socket
newrelic- 18333 root    2u     unix 0xffff9350a1c09c00      0t0  1317098 socket
newrelic- 18333 root    3u     IPv4           11711276      0t0      TCP myfunhost:52494->infra-api.newrelic.com:https (ESTABLISHED)
newrelic- 18333 root    4u  a_inode               0,10        0     6464 [eventpoll]
newrelic- 18333 root    6r  a_inode               0,10        0     6464 inotify
newrelic- 18333 root    7r  a_inode               0,10        0     6464 inotify
newrelic- 18333 root    8u     unix 0xffff93518321b400      0t0  1318644 socket

If I go look at the fdinfo for that file descriptor under the process:

pos:    0
flags:    00
mnt_id:    10
inotify wd:1 ino:77f7 sdev:14 mask:fce ignored_mask:0 fhandle-bytes:c fhandle-type:1 f_handle:3e61c95cf777000000000000

Can I take the information from fdinfo or lsof and find what it's trying to watch for?

  • Have a look at sysdig also. – Rui F Ribeiro May 14 at 8:34
  • @RuiFRibeiro I'm not sure that helps much here. It also just says FD 6 is inotify. – tubaguy50035 May 15 at 4:14
  • How easy is the problem to reproduce? If it's easy to reproduce, running under strace and examining the output to figure out what the process is watching via fd 6/inotify should be possible. You should be able to see what files/directories get added. – Andrew Henle May 15 at 9:43
  • It is not easy to reproduce unfortunately. I have a hunch that this was caused by the NR Infra agent and the NR Node agent (inside a container on the same box) trying to maybe look at the same files. They would both hang at the same time. It looks like someone mounted all of /var/log to the container so we're going to try and stop doing that. – tubaguy50035 May 15 at 18:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.