Extremely low throughput between AIX 7.2 and RHEL Maipo


 
Thread Tools Search this Thread
Operating Systems Linux Red Hat Extremely low throughput between AIX 7.2 and RHEL Maipo
# 1  
Old 02-25-2020
Extremely low throughput between AIX 7.2 and RHEL Maipo

I am currently having problems with AIX and network performance.

I have an AIX 7.2 on-premises and a Linux with RH 7.7 Maipo in Google Cloud. Between the 2 machines I have a guaranteed bandwith of 1 Gbit/s.

Firstly I started with a trivial traceroute: what I noticed is that after some hops appears the following error "fragmentation required, trying new MTU=1492". Basically this appears 4 times: MTU changes from the starting 1500 to 1492, then to 1480, then to 1472 and finally to 1006. This brings also a very very low throughput (around 16 Mbit/s). Do you have any idea of what's going on? If I try the same experiment from a Linux machine I don't get this issue and the throughtput is quite good (around 320 Mbit/sec).

Could you help me please? Thank you
# 2  
Old 02-25-2020
Have you considered to set following mentioned in below document if you are using VPN between ?

MTU considerations | Cloud VPN | Google Cloud

Can you show output of :
Code:
ip l sh

On that google VM.

Regards
Peasant.
This User Gave Thanks to Peasant For This Post:
# 3  
Old 02-25-2020
Hi, thank you for your kind reply

This is my output, as required:

Code:
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 42:01:0a:99:c8:ad brd ff:ff:ff:ff:ff:ff

Actually I don't have a VPN but a Partner Interconnect: I don't know if this may change something.
What I made in these days was working on the AIX-side: basically I focused on TCP performance/congestion window, modifying some parameters in the "no -a" (let me know if you need the output). After that, I made some scp transfers between AIX and Google GCP and I got a higher throughput (around 240 Mbit/s). Unfortunately in some of my attemps the throughput decreases very quickly, especially when I try sending larger files (around 1 or 2GB), and so I end up with my usual and previous 16 Mbit/s. In other attemps, instead, my speed connection keeps increasing (max 400 Mbit/s) or is stable.

Concerning the traceroute, I tried a traceroute toward the same Google GCP machine using many Linux on-premises but I didn't get any "fragmentation required" issue. On the other side, I always get the "fragmentation required" issue if I traceroute from an AIX.. I'm focusing on this issue because I feel like this problem is correlated to the low throughput one.

Thank you all for your time Smilie
# 4  
Old 02-25-2020
Well unfortunately, i'm no AIX guy, but it if those other boxes in your LAN are connected to same network gear and everything works as expected / paid...
I would put my 2 cents on AIX.

Hopefully, someone will more in depth AIX network stack knowledge will jump in with some metrics / dump commands.

How is the performance in your local LAN when you copy something to AIX box from other linux box ?

Regards
Peasant.
# 5  
Old 02-25-2020
I wonder why the shown MTU is 1460 while the standard is 1500.
But if your LAN switch/router works better with 1460 then try to set it on the other box, too.

I remember a similar issue (severe packet loss), where all Linux systems had the standard MTU 1500. The LAN guy changed the MTU on the LAN switch (or router?), and that fixed it.
# 6  
Old 02-26-2020
Quote:
Originally Posted by MadeInGermany
I wonder why the shown MTU is 1460 while the standard is 1500.
But if your LAN switch/router works better with 1460 then try to set it on the other box, too.

I remember a similar issue (severe packet loss), where all Linux systems had the standard MTU 1500. The LAN guy changed the MTU on the LAN switch (or router?), and that fixed it.
Google Cloud mandates this MTU:

Quote:
Gateway MTU vs. system MTU

You must configure your peer VPN gateway to use a MTU of no greater than 1460 bytes. A value of 1460 bytes is recommended because that matches the default MTU setting for Google Cloud VM instances.

The effective MTU for peer systems and Google Cloud VMs is typically lower than the MTU of your VPN gateway:

For TCP traffic, MSS clamping rewrites the SYN packet of the initial TCP handshake. This allows systems to dynamically adjust Maximum Segment Size (MSS) to accommodate encapsulation.

For UDP traffic, Path MTU Discovery (PMTUD) can negotiate smaller MTU sizes, under certain circumstances, provided that your firewall permits ICMP traffic.
REF: MTU considerations | Cloud VPN | Google Cloud
# 7  
Old 02-26-2020
Hi all,
when I copy from AIX to Linux in my local LAN I get an acceptable throughput (around 400 Mbit/s).
Same thing from AIX to AIX.

I think that my problem from LAN to Cloud might be related to some misconfiguration on the network, probably in some switch or router, but how could you explain the fact that if I transfer from a Linux in the same LAN I don't get any issue while transfering?

I analyzed some tcpdump (AIX-Cloud communication) with Wireshark and I found a lot of "Duplicate Ack", "ACKed segment that wasn't captured (common at capture start)" and "Previous segment(s) not captured (common at capture start)".

Any idea or thought would be useful for me,
Thank you again Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. AIX

Question about shared filesystem btw AIX and RHEL

We found out that the Spectrum Scale (GPFS) doesnt support mix nodes (AIX and RHEL) on direct attached storage. Is there any other options besides NFS for mix O/S? Trying to avoid network type of shared filesystem which might end up high traffic on IO because we do run backup jobs on those... (0 Replies)
Discussion started by: kiasu
0 Replies

2. Shell Programming and Scripting

AIX to RHEL migration - awk treating 0e[0-9]+ as 0 instead of string issue

Greetings Experts, We are migrating from AIX to RHEL Linux. I have created a script to verify and report the NULLs and SPACEs in the key columns and duplicates on key combination of "|" delimited set of big files. Following is the code that was successfully running in AIX. awk -F "|" 'BEGIN {... (5 Replies)
Discussion started by: chill3chee
5 Replies

3. UNIX for Advanced & Expert Users

Help understanding differences between AIX and RHEL

I have started a new job which requires AIX admin skills, which I have, and RHEL skills. Does anyone have a cheat sheet that if I know how to solve the problem in AIX how would I do that in RHEL? I was an IBM pre-sales technical trying to keep sales guys honest - not possible. Any other links to... (5 Replies)
Discussion started by: SpenceSnyder
5 Replies

4. Red Hat

High RAM usage, extremely low swapping

Hi team I have three physical servers running on Red Hat Enterprise Linux Server release 6.2 with the following memory conditions: # cat /proc/meminfo | grep -i mem MemTotal: 8062888 kB MemFree: 184540 kB Shmem: 516 kB and the following swap conditions: ... (6 Replies)
Discussion started by: hedkandi
6 Replies

5. AIX

High Runqueue (R) LOW CPU LOW I/O Low Network Low memory usage

Hello All I have a system running AIX 61 shared uncapped partition (with 11 physical processors, 24 Virtual 72GB of Memory) . The output from NMON, vmstat show a high run queue (60+) for continous periods of time intervals, but NO paging, relatively low I/o (6000) , CPU % is 40, Low network.... (9 Replies)
Discussion started by: IL-Malti
9 Replies

6. AIX

Migrating RHEL 5 UNIX users to AIX

Hi, I'm able to migrate UNIX users/groups from Solaris to AIX (with same password using 13-char encrypted password from shadow file) but no luck with RHEL 5 to AIX. I see encrypted password in RHEL 5 is bit lengthier than 13-char. Is there any way to convert encrypted password such that same... (1 Reply)
Discussion started by: reddyr
1 Replies

7. UNIX for Dummies Questions & Answers

SOLARIS vs AIX vs RHEL

Hi I'm wondering what are the differences between SOLARIS, AIX and RHEL ? I would like to know in which operating system is best for what kind of implementation ? Why some companies use Solaris instead of e.g. AIX and etc. ? thx for help. (1 Reply)
Discussion started by: presul
1 Replies

8. IP Networking

How to improve throughput?

I have a 10Gbps network link connecting two machines A and B. I want to transfer 20GB data from A to B using TCP. With default setting, I can use 50% bandwidth. How to improve the throughput? Is there any way to make throughput as close to 10Gbps as possible? thanks~ :) (3 Replies)
Discussion started by: andrewust
3 Replies

9. IP Networking

Gigabit Link throughput

As a rule of thumb in doing calculations, what figure would you use in Mbytes/sec? I know the answer varies grealty on the topolgy of the network but I wonde what newteok engineers use a rough rule of thumb? Many thanks. (1 Reply)
Discussion started by: debd
1 Replies
Login or Register to Ask a Question