3

How do you test whether a particular congestion algorithm is working for you? I ask because it's not as if I can easily re-create a representative workload for most algorithms.

I'm current looking at two things, but I'm open to more suggestions:

  • The "segments retransmitted" from the output of netstat -s

My current thinking is that congestion could result in dropped packets some percentage of the time so while there may not be a 1:1 relationship between a packet being dropped and the server being notified of a congestion event, there may be a loose correlation to be drawn if switching to one algorithm resulted in fewer dropped packets.

Does this figure show segments that were retransmitted due to congestion or is it limited to lossy links? If so (and I'm thinking it might be the case) that might muddy the waters even more to the point of this not being a good metric to use.

  • Is there a metric available to measuring the average age for TCP connections?

My thinking here is that TCP connections that finish earlier (absent a spike in errors and dropped packets) might indicate that data is being pushed through with less delay.

  • TCP connections finishing earlier indicates higher throughput, not lower latency. Ideally, you could get latency and/or throughput figures from your TCP-using applications (some games may show latency figures, http/ftp downloads show throughput,...) – ninjalj Jun 27 '14 at 20:24
  • Is there some scenario where increased throughput per-connection doesn't correspond with decreased latency? (really asking) – Bratchley Jun 27 '14 at 20:36
  • Increased througput for a connection can mean increased latency for the rest of the connections. – ninjalj Jun 27 '14 at 20:38
  • Yeah but wouldn't that tend to balance out if I'm looking at average figures? – Bratchley Jun 27 '14 at 20:39
  • I'm not sure. I think a long running non-fair high-throughput connection could increase the latency of short-lived connections, while not needing to recover from "lending" bandwidth to those other connections. – ninjalj Jun 27 '14 at 20:45
4

Does this figure show segments that were retransmitted due to congestion or is it limited to lossy links? If so (and I'm thinking it might be the case) that might muddy the waters even more to the point of this not being a good metric to use.

segments retransmitted in netstat -s includes all the kernel's TCP retransmissions for any reason, including those listed in your question. Those reasons could also include:

  • Link errors
  • Ethernet switch congestion
  • Local host drops due to qos or resource depletion
  • Remote host drops (perhaps due to some form of qos/resource depletion on the remote host)

Performance test engineers routinely deal with all these variables and ensure that they can measure them appropriately. One of the first tests you should do is to ensure the cabling / network runs clean at the packet sizes and traffic rates in question. This is normally done with a dedicated test appliance, such as those from Ixia or Spirent.

Once you baseline network performance, you're in a position to run the test you're asking about. Even if the network tested clean, you should still monitor switch interface errors/drops during the host TCP test to ensure they don't skew your results.

As to your thoughts about creating congestion conditions, you may find it helpful to intentionally generate iperf UDP background traffic just below the qos class' queueing threshold before you run your TCP traffic. If you find that you can't saturate the link you have, you also may find it helpful to negotiate the NIC down to 1GE or 100M.

All this may sound complicated, and in some ways perhaps it is; however qos testing is entirely doable with proper focus and visibility to all the system components.

| improve this answer | |
1

Concise Version:

As @ninjalj pointed out, the workload application should probably be considered the authoritative source on whether any given adjustment was beneficial to workload performance. Depending on whether your requirements are latency or only overall throughput on the system, you can make the judgment call as to whether changes in behavior better meet your performance requirements.

In this case, it would be making the change and noticing whether httpd's overall latency went down.


Longer Version With Specific Examples:

To elaborate, I'll put this in context. Let's look at Apache httpd. You can log the time to complete a request in microseconds and the size of each request using the LogFormat and CustomLog directives. For instance:

LogFormat "%h %m %D %b" perflog
CustomLog /var/log/httpd/performance.log perflog

Which produces output similar to this:

xxx.xxx.28.20 GET 41140 86
xxx.xxx.28.20 GET 34468 28
xxx.xxx.28.20 GET 47612 1434
xxx.xxx.28.20 GET 54860 868
xxx.xxx.28.20 POST 97055 6283
xxx.xxx.28.20 GET 33754 53
xxx.xxx.28.20 GET 68964 8416
xxx.xxx.28.20 GET 1143827 11528
xxx.xxx.28.20 GET 38055 61
xxx.xxx.64.208 HEAD 6255 -
xxx.xxx.28.20 GET 36922 142
xxx.xxx.28.20 GET 51871 5581

I'm going to just concern myself with GET requests for this:

root@xxxxxxvlp14 /tmp $ grep GET /var/log/httpd/performance.log > work.log
root@xxxxxxvlp14 /tmp $ sed -i 's/-$/0/g' work.log
root@xxxxxvlp14 /tmp $ 

(for whatever reason httpd give you a hyphen instead of an integer 0).

We can then programatically pick it apart:

#!/bin/bash

totalRequests=$(cat /tmp/work.log | wc -l )

totalTime=$(awk 'BEGIN{ count=0 } {count = count + $3} END { print count }' /tmp/work.log)
averageTime=$( printf "%.2f" $(bc -l <<< "$totalTime / $totalRequests "))
minTime=$(sort -nk 3 work.log | head -1 | awk '{print $3}')
maxTime=$(sort -rnk 3 work.log | head -1 | awk '{print $3}')

totalBytes=$(awk 'BEGIN{ count=0 } {count = count + $4} END { print count }' /tmp/work.log)
minBytes=$(sort -nk 4 work.log | head -1 | awk '{print $4}')
maxBytes=$(sort -rnk 4 work.log | head -1 | awk '{print $4}')

echo "Total Requests In Sample: $totalRequests"

echo

echo "Total Time Taken: $totalTime ms (average: $averageTime ms)"
echo "Longest Request: $maxTime ms"
echo "Shortest Request: $minTime ms"

echo

echo "Total Data Moved: $totalBytes bytes"
echo "Largest Request: $maxBytes bytes"
echo "Smallest Request: $minBytes bytes"

Please no comments on the script, it's written for clarity, not efficiency. The above produces the following:

Total Requests In Sample: 207

Total Time Taken: 15138613 ms (average: 73133.40 ms)
Longest Request: 1143827 ms
Shortest Request: 1788 ms

Total Data Moved: 409825 bytes
Largest Request: 20476 bytes
Smallest Request: 0 bytes

Obviously, the above illustrates why it's important to get a lengthy sample. The numbers are correct though (the minute and a half long request was someone generating a report in Word format that included images/graphs, for the curious).

So you would coax apache into giving you a lengthy sample (probably over the course of a entire day) of normal activity, make your change, rotate the logs, then begin collecting logs again (e.g waiting through another 24 hour period).

Each service (NFS, other HTTP servers, Samba, FTP servers, etc) will have its own way of gathering information but generally there will be some means of recording time and throughput.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.