3

Background

I am running BusyBox in the remote server.

I have a bash script that does two things:
1. via ssh, starts a sub process to monitor tcp traffic using tcpdump command. Save results to a file - either on remote machine or local machine. Tried both.
2. starts a second sub process to generate tcp traffic.

Code Snippet:

#html_tcpdumpfile="$(ssh remotemachine.mydomain.net \"mktemp\")"
html_tcpdumpfile=$(mktemp)

test_steps=(
    #"{ ssh remotemachine.mydomain.net \"timeout -t 20 tcpdump -nvi eth0 port 5060 > "$html_tcpdumpfile" \" ; }" 
    "{ ssh remotemachine.mydomain.net \"timeout -t 20 tcpdump -i eth0 port 5060 \"> $html_tcpdumpfile; }"   
    "{ ssh remotemachine.mydomain.net \"timeout -t 15 cat /tmp/htmlemail.txt | /etc/postfix/process_email.py \"; }"
 )
pids=()
for index in ${!test_steps[@]}; do       
      (echo "${test_steps[$index]}" | bash) &
      pids[${index}]=$!
      echo "$pids[${index}] is the pid"
done

#shouldn't really need this because of the timers but... just in case...
for pid in ${pids[*]}; 
do   
  wait $pid; 
done;
# ============ ANALYZE TEST RESULTS
echo "========== html_tcpdumpfile CONTENTS ============="
cat $html_tcpdumpfile
echo "========== html_tcpdumpfile CONTENTS ============="

Problem

Sometimes, the tcpdump command doesn't capture anything, and at other times it does. There are no error messages when it fails to capture.

What I've tried So far

  1. As you can see, I've tried to change the location of the dump file between the remote machine and the local one. That doesn't seem to make a difference.

  2. I've proven that TCP traffic is ALWAYS generated... each time I run the script because I have another ssh session open and i can see the traffic being generated. It's just that my script intermittently fails to capture it.

  3. I've tried to increase the timeout value on the tcp session to something huge to make sure I give it enough time. But I don't think that's the problem.

Any suggestions would be appreciated. Thanks.

EDIT 1

I tried to introduce a sleep in between launching each subprocess:

pids=()
for index in ${!test_steps[@]}; do       
      (echo "${test_steps[$index]}" | bash) &
      sleep 5
      pids[${index}]=$!
      echo "$pids[${index}] is the pid"
done

But that doesn't make a difference either.

EDIT 2

I changed the tcpdump command to look like this:

test_steps=(     
    "{ ssh remotemachine.mydomain.net \"timeout -t 30 tcpdump -nlc 100 -i eth0 port 5060 \"> $rtf_tcpdumpfile; }" 
    "{ ssh remotemachine.mydomain.net \"timeout -t 20 tail -f /var/log/messages \" > $syslog; }"    
    "{ ssh remotemachine.mydomain.net \"timeout -t 15 cat /tmp/htmlemail.txt | /etc/postfix/process_email.py \"; }"
 )

The tcpdump still fails to capture intermittently, but ... what's interesting is that the syslog is always successfully captured. (the python script actually writes to the syslog when it's invoked and so I can see /prove that the script is working)

  • No space in remotemachine.mydomain.net\"timeout where I'd expect there should be one (between net and \"). – Kusalananda Feb 15 '18 at 14:30
  • @Kusalananda hi that was just a typo while I created the post. The script doesn't have that issue. Otherwise, I would get a syntax error. I have updated my post to fix the typo! Thanks! – dot Feb 15 '18 at 14:37
  • 1
    Try giving tcpdump the -l (lowercase ell) option, to make it line buffered. Otherwise you'll only get output every 4KB or so. – Mark Plotnick Feb 15 '18 at 16:48
  • @MarkPlotnick Good idea, I often forget that. – Rui F Ribeiro Feb 15 '18 at 16:51
  • @MarkPlotnick I just tried that but it's still failing. :( But thanks for that suggestion. I'm keeping that option in just to be safe. – dot Feb 15 '18 at 16:58
2

First off, if you are dealing with an appliance/iOT with a limited space, I would deal with the output in the calling side, i.e. using the > after the ssh commands as in

ssh "command" > output.txt

As for tcpdump I would not kill it as a policy all the time, risking losing buffers. You might have not output maybe because of that.

I would place a limit on packets captured. I would also try not to solve DNS. As in, for capturing 100 packets:

tcpdump -nc 100 -i eth0 port 5600

When you are storing the capture file on the local system, you should only run cat locally and not remotely and locally.

Likewise, when you are running both tcpdump and cat remotely, you are launching both at the same time, and both the remote and local cat won't have nothing to show.

Following the suggestion of @MarkPlotnick, I also added -lto tcpdump to make it line buffered. That may obviate the need for the -c option. I would use both.

So I would change that script for:

#!/bin/bash
html_tcpdumpfile=$(mktemp)

ssh remotemachine.mydomain.net "timeout -t 20 tcpdump -nlc 100 -i eth0  port 5060 " > $html_tcpdumpfile

cat $html_tcpdumpfile

rm $html_tcpdumpfile

Or we might not even need to create explicitly a temporary file:

#!/bin/bash

ssh remotemachine.mydomain.net "timeout -t 20 tcpdump -nlc 100 -i eth0  port 5060 " \
| less

Lastly, I would advise deleting all the temp files created, specially on the remote side.

PS: the OP mentioned in comments the remote system is BusyBox and as such the timeout options are different than in the coretutils package. I also edit the question for it to mention BusyBox.

  • Hi there. So unfortunately, your suggestion to add "-nlc 100" doesn't improve the situation. I am saving the dump file locally and cat'ing the local file- only tried remote in case it made a difference but it didn't. Also I am cleaning up after myself. Thanks for the reminders though. Regarding DNS - sorry, that one has to stay. – dot Feb 15 '18 at 18:53
  • ssh remotemachine.mydomain.net "timeout -t 20 tcpdump -lc 100 -i eth0 port 5060 " | less does it work? – Rui F Ribeiro Feb 15 '18 at 18:55
  • I didn't tack on " | less" because I don't want to change more code than necessary right now... but i did change the tcpdump to REMOVE the "n" and just have "tcpdump -lc 100 -i eth0 port 5060" but that doesn't work – dot Feb 15 '18 at 19:03
  • Did you see my EDIT 2? Specifically, wondering if you noticed my findings with syslog? Just wondering if this proves that the issue is with tcpdump command instead of some logic issue in the script per se. – dot Feb 15 '18 at 19:17
  • comment the "&" and comment/delete the waitpid cycle, at the end of the day is mostly what my answer is about. – Rui F Ribeiro Feb 15 '18 at 19:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.