Solaris

Hello all,

Needed a suggestion from you all, if you know anything about this stuff.
We have a high network traffic application. Close to around 700Meg /sec on one NIC.

When the traffic is around 200Meg on the NIC, the VCPU(not the CPU, cause we have 24 VCPU) utilization by the NIC driver hits 88%. I am afraid we cannot scale it to 700Meg, because the driver does not seem to migrate to any other VCPU and stickts to one VCPU all the time.

Any suggestions, so we could make the driver float around?

What type of network traffic is it?

If you are dealing with a lot of encrypted traffic, you could probably do something more efficient...ie offload to the crypto accelerator, because you may be seeing a problem related to FPU queuing, rather than actual CPU load.

reborg..
its plain UDp data getting dumped on to the interface. As soon as the number of open sockets/bindings(either multicast or unicast), the single cpu usage which the bge driver is stuck to keeps hitting max and then starts losing packets.

Is there anyway we could make the driver get distributed among the other 20 vcpus which are lying idle.

Looks like the bge driver always gets stuck with one vcpu and doesnt move around other idle cpus.

You could try upping the number of send and receive rings if you are using a T1000 it's bge's you should have these settings in /kernel/drv/bge.conf

bge-rx-rings = 1;
bge-tx-rings = 1;

You could try increasing the number of rings.

If it's a t2000, you have e1000g nics, and I'm not sure if equivalent code has been putback yet.

Reborg,

Ur guess is right. its t1k.
I had tried it earlier and made it 4 each, but did not help. The driver seemed to be stuck on one VCPU and when the load increased we started losing packets.

I have tried the below settings too.
set ip:ip_squeue_bind=0
set ip:ip_squeue_fanout=1

But the driver seems to be having some cpu affinity(with or without creation of processor sets).

It does not budge. Any suggestions?

Hi Naanu,


We're running into the same problem, and I was wondering if you had been able to resolve it. We're looking at 200Mbps RTP traffix, with some Sigtran signaling interspersed. Because the RTP packets are all 214 bytes long, we're getting about 100K packets/second in. This saturates the virtual CPU and we're dropping packets.

The thread didn't show a resolution, but I'd be very interested to know if you were able to work around it.

Thanks very much!

Jaap.