[quagga-users 12134] Re: State of the Quagga?

Alexis Rosen quagga-users at alexis.users.panix.com
Thu Mar 3 10:40:05 GMT 2011

On Mar 1, 2011, at 10:15 AM, tommy at the-coffeeshop.dk wrote:
> Citat af Alexis Rosen <quagga-users at alexis.users.panix.com>:
>> Things that can help you keep up with a couple million PPS? We  haven't found anything so far in consumer-level gear. But 8 to 12 cores in an Intel Nehalem dual-chip server, along with really  high-end ethernet adapters (the 10GbE ones) doing multiqueue should be able to do it pretty comfortably.
> Okay. That's one to consider seeing as it's not really that expensive either way. A dual Nehalem box with 12gigs of RAM isn't much these days, and I've just seen Intel 10gig cards for $400-500 at my local distributor, so that's not really that frightening. It is, after all, critical infrastructure and none of my other stuff works without it :)

Yeah. BTW, 6GB is more than enough unless you're running a *ridiculous* number of full BGP views (you're not, that's what router servers are for) or doing a lot of other nonrouting things on the box.

Make sure you can return the cards if you can't get them working the way you want. This isn't rocket science, but it's also apparently not often done. YMMV and you should definitely lab this stuff up before you put it into production.

> So you're proposing running 1gig on the 10gig cards to get the advantage multiqueue etc? (I ask that way since I have no 10gig infrastructure at this point, except internally in my core switch stacks) :)

Yes. The 82756s also do multiqueue but they don't have as much hardware assist as the 82599s. There are apparently non-Intel adapters that also work well, but I don't know much about them.

Depending on what kernel you wind up with, look into IRQ affinity, that's a big problem if you get it wrong. You want to hardcode things, not run irqbalanced, or at least that's what worked in our setup. And did I already say YMMV?? I don't know if that's still true if you're running a kernel with RFS or the transmit flow patches (not even sure if those are in a released kernel yet).

Another thing you can look as is asymmetric local routing; run all inbound traffic through one router and all outbound through another. We didn't try that, and I can see some serious issues you might have with that depending on what you're doing. (Keeping any kind of session state would probably be a nonstarter, though that may change in the next year.)

BTW, doing anything stateful is a big hit to performance. If we want NAT on anything, we do it on a separate box. Firewall stuff can be equally threatening.

>> With a quad-core Nehalem and a  midrange multiqueue server adapter like the Intel 82576, which you  can do for comfortably under $1k, we've been able to get around a  million PPS without dropping a ton of packets. However, that's  running with large buffers, and as people are beginning to learn  (again), large buffers are problematic in other ways.
> Okay, we're not much further along than a million PPS? I remember that being the semi-holy (but obtainable) grale 10 years ago, I'd honestly have thought we were further along.

Um, what? Do you mean on a cisco? Because I can pretty much guarantee you couldn't route a million pps ten years ago on Intel hardware.

>> We've tried  the Opteron octocore chips too; at least so far, they don't present  an improvement over the quad-core Nehalems, but neither are they  much worse. That was almost a year ago, though, and it's entirely  possible that the kernel can take better advantage of them now. At  least in theory, they should be able to do better.
> Okay. I haven't any Opteron stuff at all these days, and pretty much prefer to keep my hardware alike if possible - so I'm kinda glad to hear that they don't boast a big gain ;)

I'm thinking of looking at it again when their next gen hits. There is potential for a big price/performance win there, if they can tune the software right. It's mostly a NUMA thing. And it should be doable, I think. The big question is, can you consistently assign packets to the right CPU so you don't have a ridiculous amount of cache-line bouncing on a many-cores server? We'll see.

>> The most recent Linux kernels have a ton of networking improvements  like RFS that may make a server adapter much less critical. I  haven't experimented to see if that's so or not.
> Okay. Any chance any of it is utilized in Vyatta?

Some. The latest devel image is a fairly recent kernel. Not new enough for all the goodies that just hit in the last kernel though.

Unless you're within a few percent of your goal that's probably not where you want to be tuning things. As I mentioned, you want more of a hardware assist than just the multiqueue, and multiqueue by itself is very cheap these days - he most recent server boards often have a 576 already on board (like the latest gen of super micros, for example).

>> I'd really like to know what other people are doing for high PPS  routing. I posted last year and got virtually nothing useful back. I  wonder if people are even bothering sizing their routers based on  attack scenarios?
> I wonder if maybe people are a bit afraid to speak of the known limits of their network in public?

Maybe. Lame. It won't help.


More information about the Quagga-users mailing list