[quagga-dev 10960] Re: high-bandwidth, low-latency quagga necessary ?

Jose Gavine Cueto pepedocs at gmail.com
Thu Dec 12 03:17:41 GMT 2013

This could be my last reply before getting kicked out (sorry).

I have a personal experience with DPDK development.  Particularly, I've
coded a "fast path" for a software switch so that included layer 2 packet
processing through poll-mode drivers API.  Besides these experiences, I've
inspected similar proprietary applications.

My understanding is that DPDK does not replace the kernel network stack,
its just enables one to create a fast stack that could replace the kernel
stack.  With regard to performance, by experience it is difficult to say
which is better without experimentation and testing even though you could
theoretically infer about its juicy benefits.  I've also seen 10x
performance gains from DPDK enhanced applications, but they are highly
specialized in terms of their architecture.

Now I've got some possible use cases here, I'm really glad to include
quagga in my list and continue hunting some possible open applications that
I can possibly experiment with DPDK enhancements.


On Thu, Dec 12, 2013 at 2:15 AM, Charlet, Ricky <ricky.charlet at hp.com>wrote:

> Howdy,
> In reply to a few questions from Alexis, I'll summarize and reply in the
> main body here rather than reply inline.
> Also, in case it helps you to understand where I am coming from and how
> far to believe the things I say (not to far really) , here is a word about
> my personal experience level with DPDK...
> I've never integrated or programed DPDK. I've worked in two different
> companies that use DPDK and been a programmer quite near it.
>         At one place, I've used DPDK to source/sync fast traffic and been
> involved in architectural decisions to migrate to using DPDK. We
> transformed our app from the ability to source/sync 4 Gb traffic to 30 Gb
> of traffic.
>         At another place, they were using DPDK when I got there for
> network traffic forwarding. I personally work on keeping the route tables
> up to date (using quagga) in the DPDK fast path. The (top version of the)
> thing forwards 40Gb of traffic.
> Now for my replies:
> Q: how does dpdk interact with netfilter/iptables (those are in the
> kernel)?
> A: DPDK is a user-mode tcp/ip stack and replaces the entirety of the
> kernel stack including netfilter and iptables. On the other hand DPDK
> itself is strongly based on BSD and includes an iptables feature in it. You
> should speak with your DPDK sales / professional-services engineer to
> understand exactly what features it can offer. I can confidently say to you
> that DPDK has IPv6 and iptables with conntrack and it works at surprisingly
> great speed. Your realization of speed will depend upon your hardware, but
> will not depend upon your kernel.
> Q: (from testing observations) how big an impact does DPDK have?
> From experience, I saw it provide a, nearly 10x improvement in application
> source/sync throughput on same hardware. But that is not too relevant to
> our discussion here. We are more interested in packet forwarding
> improvement. Even though I do work on a DPDK packet forwarder currently, I
> don't have any before/after comparison. Our first release of product has
> been DPDK based from the beginning. I'm sure that DPDK vendors will be more
> than happy to provide you with slides and studies.
> Final note:
> The point I'm trying to make to this list is not "DPDK is better", but
> "having quagga support a DPDK user stack rather than the kernel stack is
> fairly easy".  DPDK can drop in and replace kernel stack in a rather
> un-noticible way.
> --
> Ricky Charlet
> Software Dev / Routing Dude: Aries team, Roseville CA
> ricky.charlet at hp.com
> USA: 916.785.2090
> -----Original Message-----
> From: Alexis Rosen [mailto:quagga-users at alexis.users.panix.com]
> Sent: Monday, December 09, 2013 7:12 PM
> To: Charlet, Ricky
> Cc: Jose Gavine Cueto; quagga-dev at lists.quagga.net Dev
> Subject: Re: [quagga-dev 10953] Re: high-bandwidth, low-latency quagga
> necessary ?
> On Dec 9, 2013, at 12:24 PM, "Charlet, Ricky" <ricky.charlet at hp.com>
> wrote:
> >                 Don't view DPDK as replacing the drivers, view DPDK as
> replacing the TCPIP stack (and drivers). But there is a trick... All
> communication that zebra has with the kernel, over ioctl or over netlink,
> stays the same and DPDK snoops it and just "does the right thing". The
> kernel interface is unmodified.
> Thanks for the followup.
> So how does this work with, say, netfilter and xtables? Those live in
> kernel space... Is there a shared memory region? Does it work with v6?
> Conntrack (not that you likely want to be using that in a high-throughput
> environment)?
> > [...] If you agree with my reasoning, very close to *nothing* in quagga
> needs to be modified to adapt to a DPDK system. Well, why should I
> equivocate... OK... I claim that truly nothing in quagga need be modified
> to adapt to a DPDK system that leaves quagga daemon (control plane) sockets
> to the kernel.
> That sounds wonderful. Now, the question is, how big an impact will this
> have? Have you tested this at all? A couple of years ago we set up a lab
> for throughput testing (some of the results of which I posted here) but
> that's long gone and I don't expect to be able to do that again in the next
> couple of months.
> Can anyone here give some numbers for a running quagga system, before and
> after installing DPDK? Either max PPS throughput (hopefully per-core), or
> CPU usage per core for a given level of traffic? And, can anyone confirm
> that they did this without modifying the Quagga config at all?
> /a

To stop learning is like to stop loving.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.quagga.net/pipermail/quagga-dev/attachments/20131212/669a68e4/attachment-0001.html>

More information about the Quagga-dev mailing list