5

I have a process (the NewRelic infra agent) that is hung on a RHEL 7, xfs filesystem host. I'm thinking that it's trying to watch a file or maybe a socket that another process already has in use.

When I run strace on it, I get this:

strace: Process 18333 attached
read(6,

If I run lsof on it, I see this:

COMMAND     PID USER   FD      TYPE             DEVICE SIZE/OFF     NODE NAME
newrelic- 18333 root  cwd       DIR              253,3      236       64 /
newrelic- 18333 root  rtd       DIR              253,3      236       64 /
newrelic- 18333 root  txt       REG              253,3 15672136  9280340 /usr/bin/newrelic-infra
newrelic- 18333 root  mem       REG              253,3    68192 16892409 /usr/lib64/libbz2.so.1.0.6
newrelic- 18333 root  mem       REG              253,3   157400 16892537 /usr/lib64/liblzma.so.5.2.2
newrelic- 18333 root  mem       REG              253,3    90248 16891935 /usr/lib64/libz.so.1.2.7
newrelic- 18333 root  mem       REG              253,3   100008 16891900 /usr/lib64/libelf-0.172.so
newrelic- 18333 root  mem       REG              253,3    19896 16892403 /usr/lib64/libattr.so.1.1.0
newrelic- 18333 root  mem       REG              253,3    88776 16798006 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
newrelic- 18333 root  mem       REG              253,3    19288 16891827 /usr/lib64/libdl-2.17.so
newrelic- 18333 root  mem       REG              253,3   330464 16891809 /usr/lib64/libdw-0.172.so
newrelic- 18333 root  mem       REG              253,3    43776 17005232 /usr/lib64/librt-2.17.so
newrelic- 18333 root  mem       REG              253,3  1137024 16891840 /usr/lib64/libm-2.17.so
newrelic- 18333 root  mem       REG              253,3    20032 16892412 /usr/lib64/libcap.so.2.22
newrelic- 18333 root  mem       REG              253,3    86544 16891748 /usr/lib64/libnss_myhostname.so.2
newrelic- 18333 root  mem       REG              253,3   105824 17005229 /usr/lib64/libresolv-2.17.so
newrelic- 18333 root  mem       REG              253,3    31408 17005206 /usr/lib64/libnss_dns-2.17.so
newrelic- 18333 root  mem       REG              253,3    61624 17005208 /usr/lib64/libnss_files-2.17.so
newrelic- 18333 root  mem       REG              253,3  2151672 16891807 /usr/lib64/libc-2.17.so
newrelic- 18333 root  mem       REG              253,3   141968 17005223 /usr/lib64/libpthread-2.17.so
newrelic- 18333 root  mem       REG              253,3   163400 16798041 /usr/lib64/ld-2.17.so
newrelic- 18333 root    0r      CHR                1,3      0t0     1042 /dev/null
newrelic- 18333 root    1u     unix 0xffff9350a1c09c00      0t0  1317098 socket
newrelic- 18333 root    2u     unix 0xffff9350a1c09c00      0t0  1317098 socket
newrelic- 18333 root    3u     IPv4           11711276      0t0      TCP myfunhost:52494->infra-api.newrelic.com:https (ESTABLISHED)
newrelic- 18333 root    4u  a_inode               0,10        0     6464 [eventpoll]
newrelic- 18333 root    6r  a_inode               0,10        0     6464 inotify
newrelic- 18333 root    7r  a_inode               0,10        0     6464 inotify
newrelic- 18333 root    8u     unix 0xffff93518321b400      0t0  1318644 socket

If I go look at the fdinfo for that file descriptor under the process:

pos:    0
flags:    00
mnt_id:    10
inotify wd:1 ino:77f7 sdev:14 mask:fce ignored_mask:0 fhandle-bytes:c fhandle-type:1 f_handle:3e61c95cf777000000000000

Can I take the information from fdinfo or lsof and find what it's trying to watch for?

  • Have a look at sysdig also. – Rui F Ribeiro May 14 '19 at 8:34
  • @RuiFRibeiro I'm not sure that helps much here. It also just says FD 6 is inotify. – tubaguy50035 May 15 '19 at 4:14
  • How easy is the problem to reproduce? If it's easy to reproduce, running under strace and examining the output to figure out what the process is watching via fd 6/inotify should be possible. You should be able to see what files/directories get added. – Andrew Henle May 15 '19 at 9:43
  • It is not easy to reproduce unfortunately. I have a hunch that this was caused by the NR Infra agent and the NR Node agent (inside a container on the same box) trying to maybe look at the same files. They would both hang at the same time. It looks like someone mounted all of /var/log to the container so we're going to try and stop doing that. – tubaguy50035 May 15 '19 at 18:09
0

If you have the ability to restart the process then start it with strace and monitor the inotify_add_watch systemcall.

inotify does not support recursive watches so each single directory that is watched will be printed to your screen.

strace -fe trace=inotify_add_watch ./NewRelic

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.