[quagga-dev 10959] Re: high-bandwidth, low-latency quagga necessary ?
ricky.charlet at hp.com
Wed Dec 11 18:15:02 GMT 2013
In reply to a few questions from Alexis, I'll summarize and reply in the main body here rather than reply inline.
Also, in case it helps you to understand where I am coming from and how far to believe the things I say (not to far really) , here is a word about my personal experience level with DPDK...
I've never integrated or programed DPDK. I've worked in two different companies that use DPDK and been a programmer quite near it.
At one place, I've used DPDK to source/sync fast traffic and been involved in architectural decisions to migrate to using DPDK. We transformed our app from the ability to source/sync 4 Gb traffic to 30 Gb of traffic.
At another place, they were using DPDK when I got there for network traffic forwarding. I personally work on keeping the route tables up to date (using quagga) in the DPDK fast path. The (top version of the) thing forwards 40Gb of traffic.
Now for my replies:
Q: how does dpdk interact with netfilter/iptables (those are in the kernel)?
A: DPDK is a user-mode tcp/ip stack and replaces the entirety of the kernel stack including netfilter and iptables. On the other hand DPDK itself is strongly based on BSD and includes an iptables feature in it. You should speak with your DPDK sales / professional-services engineer to understand exactly what features it can offer. I can confidently say to you that DPDK has IPv6 and iptables with conntrack and it works at surprisingly great speed. Your realization of speed will depend upon your hardware, but will not depend upon your kernel.
Q: (from testing observations) how big an impact does DPDK have?
>From experience, I saw it provide a, nearly 10x improvement in application source/sync throughput on same hardware. But that is not too relevant to our discussion here. We are more interested in packet forwarding improvement. Even though I do work on a DPDK packet forwarder currently, I don't have any before/after comparison. Our first release of product has been DPDK based from the beginning. I'm sure that DPDK vendors will be more than happy to provide you with slides and studies.
The point I'm trying to make to this list is not "DPDK is better", but "having quagga support a DPDK user stack rather than the kernel stack is fairly easy". DPDK can drop in and replace kernel stack in a rather un-noticible way.
Software Dev / Routing Dude: Aries team, Roseville CA
ricky.charlet at hp.com
From: Alexis Rosen [mailto:quagga-users at alexis.users.panix.com]
Sent: Monday, December 09, 2013 7:12 PM
To: Charlet, Ricky
Cc: Jose Gavine Cueto; quagga-dev at lists.quagga.net Dev
Subject: Re: [quagga-dev 10953] Re: high-bandwidth, low-latency quagga necessary ?
On Dec 9, 2013, at 12:24 PM, "Charlet, Ricky" <ricky.charlet at hp.com> wrote:
> Don't view DPDK as replacing the drivers, view DPDK as replacing the TCPIP stack (and drivers). But there is a trick... All communication that zebra has with the kernel, over ioctl or over netlink, stays the same and DPDK snoops it and just "does the right thing". The kernel interface is unmodified.
Thanks for the followup.
So how does this work with, say, netfilter and xtables? Those live in kernel space... Is there a shared memory region? Does it work with v6? Conntrack (not that you likely want to be using that in a high-throughput environment)?
> [...] If you agree with my reasoning, very close to *nothing* in quagga needs to be modified to adapt to a DPDK system. Well, why should I equivocate... OK... I claim that truly nothing in quagga need be modified to adapt to a DPDK system that leaves quagga daemon (control plane) sockets to the kernel.
That sounds wonderful. Now, the question is, how big an impact will this have? Have you tested this at all? A couple of years ago we set up a lab for throughput testing (some of the results of which I posted here) but that's long gone and I don't expect to be able to do that again in the next couple of months.
Can anyone here give some numbers for a running quagga system, before and after installing DPDK? Either max PPS throughput (hopefully per-core), or CPU usage per core for a given level of traffic? And, can anyone confirm that they did this without modifying the Quagga config at all?
More information about the Quagga-dev