Performance evaluation of OpenFlow Controllers: what's a reliable evaluation?

Question

SDN controllers are diverse and complex, but they have common goals and features. An important aspect of knowing which SDN controller to pick is whether it performs well, and whether its performance bottlenecks are reasonably high.

In light of wisdom, I stumbled on this evaluation paper, which essentially criticizes the performance of OpenFlow, giving many insightful reasons on why the OpenFlow protocol was subobtimally implemented. Most interesting, I found, was the comparison of many SDN controllers on similar benchmarks.

I use the OpenDaylight controller for research, and I found it pretty damning that this paper claims that ODL is so inefficient that providing experimental data is futile. This seems like an unreal claim, considering how large and active ODL is.

While the paper gives many reasons why the other OpenFlow controllers could perform less well, it is annoying that strictly nothing is given on OpenDaylight. Additionally, I note that the general logical architecture of these SDN controllers is not given. This is worrying, to me, because programmability is the name of the game with SDN, and so using mostly default behavior (which I assume is what's going on in the paper) is probably not the most reliable way to compare the capacities of SDN controllers.

Say, Controller A uses Technique A to automatically discover the topology, and Controller B uses Technique B. If Technique B is more efficient, regardless of implementation, this would result in a clear bias in the performance evaluations of both Controllers. If both controllers used Technique B (something reasonable considering the highly configurable nature of SDN), then the evaluation would be more fair.

Another point that bothers me is the characteristics that are evaluated. In my mind, latency is as important as the bottleneck performance of processing a certain amount of messages per second on a given hardware node. This, to me, is much less implementation dependent, seeing as there are many different techniques to perform similar tasks with SDN, but those techniques do not have the same 'complexity' in terms of messaging overhead, or packet-in/packet-out rate.

Does this make sense? Is this right, or am I missing something? Are the performances given in the paper to be taken with a grain of salt? If yes, then what's an implementation-independent way of evaluating a controller technology?

Answer 1

One of my friends, Daniel Farrel of RedHat, sent me a URL to your Question. Daniel led the OPNFV CPerf project for several years: https://wiki.opnfv.org/display/cperf CPerf is a cross project collaboration with members from ODL Integration/Test, OPNFV, and IETF Benchmarking Methodology Working Group. We've learned quite a bit through our testing efforts and weekly discussions (and you are welcome to join us of course).

I read the conference paper you cited, and share many of your concerns. Many of your interests and questions are on-target, and I'll share a few links and add some short answers that might help.

I too was surprised that the authors excluded ODL results due to low performance. There are plenty of test results around for ODL, including CI test results: https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Performance_Test:Results One of the interesting comparative studies was performed by members of CPerf, a few years back: https://www.linux.com/news/sdn-developers-report-key-lessons-testing-opendaylight-performance

One of the surprises was that the persistent datastore (which is a valuable reliability feature of ODL and enabled by default) is a performance handicap, and other controllers did not offer this feature or did not enable it by default. The persistent datastore has to be disabled in ODL to make a fair comparison. The benefits of using a SSD for the datastore were also assessed. More recent testing of the ODL controller is described here: https://www.opendaylight.org/blog/2017/10/24/how-performance-testing-improved-the-nitrogen-release

Another discussion topic has centered around the "right" set of metrics for controller benchmarking. CPerf project thoughts are captured in this JIRA ticket: https://jira.opnfv.org/projects/CPERF/issues/CPERF-3?filter=allopenissues One of the points we all agreed was that Loss of OpenFlow Packet-IN to Flow-MOD responses was a key metric for benchmarking. In other words, many tests of controller Throughput also measure response latency concurrently, and a Throughput where latency is high for many responses may not be represent a useful operating condition. This is likewise true when the Packet-INs produce no response, and CPerf agreed that Throughput measurements should report the level where no responses are lost. One of our team members wrote a golang tool that deploys as a probe on the controller: https://github.com/anastop/latte and makes both Loss and Latency measurements on the OF interface.

I also mentioned that the IETF Benchmarking Methodology WG has worked on specs for controller benchmarks, and the lead author of these Internet Drafts has also participated in CPerf: https://datatracker.ietf.org/doc/html/draft-ietf-bmwg-sdn-controller-benchmark-term-06 https://datatracker.ietf.org/doc/html/draft-ietf-bmwg-sdn-controller-benchmark-meth-06

I hope you'll find this material helpful in your research.

Al