Visit Our UNIX and Linux User Community


10gbe ixgbe slow at 420mbyte/s max, p2p lan cat8 1m, tried ethtool and ifconfig options to no avail


 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers 10gbe ixgbe slow at 420mbyte/s max, p2p lan cat8 1m, tried ethtool and ifconfig options to no avail
# 1  
Old 04-10-2020
[SOLVED] 10gbe ixgbe slow at 420mbyte/s max, p2p lan cat8 1m, tried ethtool and ifconfig options

Hey guys, first time dealing with 10gbe.
I have two boxes, one is older with i7 2.8ghz from 2009 generation, pcie 2.0 and I just put the Intel X550T there to make use of absolutely similar but builtin adapter on my newer box with AMD EPYC gen 1.
I'm dling file from tmpfs on either box and never go above 415mbyte/s which is about half of potential bandwidth there, right?
I tried most stuff from kernel . org/doc/ols/2009/ols2009-pages-169-184.pdf with a little improvement from 393mbyte/s to current 415.
that's the result of me setting mtu 9000, txqueuelen 10000.
Also tried this to no improvements over current result: darksideclouds wordpress com/2016/10/10/tuning-10gb-nics-highway-to-hell/
And a few other pages out there on google, most stating same options.
I don't see a 100% load burst on i7 box when I dl file, it's around 10% across the cores. So CPU shouldn't be a bottleneck, cable(it's 1meter cat8, with a nice thick shielding) shouldn't be a bottleneck and ram speed and pcie 2.0 speed shouldn't be.
iperf also reports same speed.
Tried these tests in both directions.
There's gotta be something obvious that I'm missing, right?

Pretty sure it doesn't matter but it's lfs on i7 and debian 10 on epyc, both sharing same sysctl settings and ethtool-controlled options.
lspci -n for it on both computers:
Code:
62:00.0 0200: 8086:1563 (rev 01)

so it's similar device. I bought this particular one for compatibility but didn't expect these issues.

Maybe someone here would suggest something, I'm out of ideas.
TIA
--
solved see my own response

Last edited by tinfoil3d; 04-11-2020 at 09:48 AM.. Reason: solved, it's PCIe lane count, usb3, sata3, 6 disks, one GPU downgraded to x8 and no more space to have x4 on pcie 2.0 mb with max 16 lanes i7 CPU
# 2  
Old 04-10-2020
Hi tinfoil,

a) check the configured port speed on both sides

Code:
ethtool <yourinterfacename> | grep -iE "(speed|duplex)"

It should look like this:

Code:
ethtool enp216s0f1 | grep -Ei "(speed|duplex)"
        Speed: 10000Mb/s
        Duplex: Full

b) check the raw network speed with iperf:

Server

Code:
iperf -s -p 12345

Client

Code:
iperf -c server.ip.add.ress -p 12345

and after that in reverse direction with client/server exchanged.

iperf should be available in any package repository under linux. if not: download & compile

regards,
stomp

Last edited by stomp; 04-10-2020 at 09:15 AM..
# 3  
Old 04-10-2020
Hey stomp,
both sides show valid full 10gig, full duplex, I checked that first thing. As stated, txq and mtu match. Raising them to 10000 and 9000 respectively increased speed from 393 to 415 but not any further.
Swapped ports, tried flight-proven cat7 cable: no change.

iperf reports same speed as wget from nginx, apache, even busybox.
The interesting note is when dling to epyc (newer) box it shows 421mbyte/s, downloading to i7 from epyc it shows 415mbyte/s max. I don't know if that may suggest you anything.

also, updating post with more details:
ethtool info for interfaces differs slightly but i'm pretty sure it doesn't matter: you can just see the diff.

this is for i7: https : // paste. ubuntu. com/p/JmDQcYWkbs/
this is epyc: https : / / paste ubuntu .com/p/ZThRF6kcqs/
and this is diff(embedded):
Code:
# diff i7 epyc 
1,2c1
< Cannot get device udp-fragmentation-offload settings: Operation not supported
< Features for eth101:
---
> Features for eth1:
8c7
<     tx-checksum-fcoe-crc: off
---
>     tx-checksum-fcoe-crc: on
32c31
< tx-fcoe-segmentation: off
---
> tx-fcoe-segmentation: on

and ethtool -g for both boxes match:
Code:
Ring parameters for eth1:
Pre-set maximums:
RX:        4096
RX Mini:    0
RX Jumbo:    0
TX:        4096
Current hardware settings:
RX:        4096
RX Mini:    0
RX Jumbo:    0
TX:        4096

# 4  
Old 04-10-2020
Please provide output of
Code:
inxi -v7

from both boxes to have some basic system information for your setup. Inxi has to be installed first. anonymize output before posting if necessary.
# 5  
Old 04-11-2020
what info are you interested in, i can post it from conventional tools like ethtool, ip, ifconfig, lshw, lspci.
# 6  
Old 04-11-2020
Hi,

I'm not sure if I can be of help. I just want to see the basic hardware and software environment. inxi does a good job collecting that and presenting it in a compact way. There may be some specific memories of problems, when I see your environment. Seems not very useful to mention everything of what most is probably not relevent in your case.

Inxi is a script relying on the basic system tools and perl. If you're sceptical to install or even only download an run it it's ok for me.

Further more I have the suggestion to review the kernel startup log(debian: /var/log/kern.log) and dmesg for any error/warning regarding driver module loading(especially complaints about missing firmware files) or networking errors/warnings.
This User Gave Thanks to the-architect For This Post:
# 7  
Old 04-11-2020
no firmware related stuff but on i7 box i see this
Code:
[    3.784158] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 5.1.0-k
[    3.784159] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[    4.460945] ixgbe 0000:02:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[    4.559030] ixgbe 0000:02:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x1 link at 0000:00:05.0 (capable of 31.504 Gb/s with 8 GT/s x4 link)
[    4.670873] ixgbe 0000:02:00.0: MAC: 4, PHY: 0, PBA No: -
[    4.830343] ixgbe 0000:02:00.0: Intel(R) 10 Gigabit Network Connection
[    4.830387] libphy: ixgbe-mdio: probed
[    5.512853] ixgbe 0000:02:00.1: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[    5.611006] ixgbe 0000:02:00.1: 4.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x1 link at 0000:00:05.0 (capable of 31.504 Gb/s with 8 GT/s x4 link)
[    5.722787] ixgbe 0000:02:00.1: MAC: 4, PHY: 0, PBA No: -
[    5.882245] ixgbe 0000:02:00.1: Intel(R) 10 Gigabit Network Connection
[    5.882292] libphy: ixgbe-mdio: probed

does it say it sits on x1 not x4?
because on EPYC box it says this:
Code:
[    2.364368] ixgbe 0000:62:00.0: 32.000 Gb/s available PCIe bandwidth (5 GT/s x8 link)

kernel driver version matches, all other messages match too... strange, i thought this slot is x8, as manual states but then again i have 6 sata devices, full x16 gpu and both PCI slots used so that's probably all the lanes CPU has... you gave me an idea, thanks.
i guess i'll just pull it out later and try any of my other boxes and see if they report it differently under livecd deb 10.
i don't know how CPUs handle the lane selection but i guess if i have x16 card then x16 it would be unless it needs to be balanced out. this particular cpu is i7 860, with 16 lanes max, so if GPU is downgraded to x8, the rest goes to sata devices and two pci(legacy) devices(1gigabit lan and sound card).
there's no way it would be given x4 bandwidth by the cpu right. thanks, i will try it but i guess i'll have to suck it up and bear with it on 4gbit which would be a rather good result huh.


i guess there's really no way i can solve this and there's no better CPU for this socket so this would just never work on full speed unless i can get rid of some drives which isn't really possible. thanks intel for making such low-lane cpus. take amd, this box has 128 lanes with lot of them used by a bunch of nvmes and other stuff and i can plug in as much as physically possible and it would still have room for more in this case.

Last edited by tinfoil3d; 04-11-2020 at 09:07 AM.. Reason: lanes...
This User Gave Thanks to tinfoil3d For This Post:

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Ethtool command not working

Hi All, I am trying to find my NIC card speed in centos 5. But i am unable to get the info using ethtool command. # ethtool eth0 Settings for eth0: Link detected: yes and even tried to look for the file # cd /sys/class/net/eth0/speed files which is also not available.... (8 Replies)
Discussion started by: sravani25
8 Replies

2. UNIX and Linux Applications

VSFTPD slow download outside of LAN

Hello all. I have VSFTP up and ruining on ubuntu 10.04. It works great inside LAN, however when it comes to using it outside i have some problems. If i try to connect from filezila i get the listing and if i try to download it shows it will take forever to receive a file which is larger then 1mb,... (2 Replies)
Discussion started by: mantas1
2 Replies

3. Red Hat

Ethtool script not running on boot.

Hi, there is an incompatibility between my NIC and my Western Digital Powerline adapters which appears to cause autonegotiation to fail intermittently, most often following a reboot. Running the following Ethtool script will establish a connection immediately: #!/bin/sh # chkconfig: 345 85... (12 Replies)
Discussion started by: torley
12 Replies

4. Shell Programming and Scripting

cut, sed, awk too slow to retrieve line - other options?

Hi, I have a script that, basically, has two input files of this type: file1 key1=value1_1_1 key2=value1_2_1 key4=value1_4_1 ... file2 key2=value2_2_1 key2=value2_2_2 key3=value2_3_1 key4=value2_4_1 ... My files are 10k lines big each (approx). The keys are strings that don't... (7 Replies)
Discussion started by: fzd
7 Replies

5. IP Networking

Local Lan, no-ip directed DNS forward, surf within lan

Hi, We have a website running on a local centos 5.4 surfer, static IP. The domain.com uses no-ip.com to take care of the DNS, it forwards all to my server. My router receives the port 80 call, routes it to my server and the world can see domain.com perfectly fine. However, we cannot see... (3 Replies)
Discussion started by: lawstudent
3 Replies

6. IP Networking

Slow LAN FTP Transfers with FreeBSD...

Hello, I've got a FreeBSD 6 server running on my network that I used as a file server for my main computer. However I've really become frustrated with the slow up/down speeds I'm getting, about 100KB/s, which is about as bad as if I was using a "real" server. My setup is as follows: ISP |... (4 Replies)
Discussion started by: CyberCat
4 Replies

7. Programming

ethtool problem

Hi, I'm using ethtool to change network settings: Just for the experiment, i want to change speed to 10, instead of 100, I type: ethtool -s eth0 speed 10 duplex full autoneg off and to view settings i type: ethtool eth0, and get this: Settings for eth0: Supported ports: ... (4 Replies)
Discussion started by: alex889
4 Replies

8. Shell Programming and Scripting

ethtool : command not found

Hi, I am uising Suse linux on my system and i tried : man ethtool and i got the manual for ethtool but when i tried to use the command as: ethtool -a eth0 it says Command not found. also, i tried rpm -qa|grep ethtool i got: ethtool-3-15.2 so,it means my system has ethtool server or... (4 Replies)
Discussion started by: salil2012
4 Replies

9. Solaris

How to configure private LAN and coporate LAN on the same machine-Solaris10

Hi , I am trying to configure a private LAN and corporate LAN on the same machien on Solaris 10. How can I achieve this? Thanks (1 Reply)
Discussion started by: deedee
1 Replies

Featured Tech Videos