AIX

Is there any option to increase that 2048 to 2148??

Sometimes when a particular host does not present BIOS entry options for HBA's I've known people put the cards in another machine, set and save the BIOS parameters, and then reinstall the HBA into the required machine.

I may well be wrong but at the moment my money is on the HBA's being unable to communicate because of their configuration settings, and nothing to do with the O/S's, drivers, or anything like that. I say this because you get a "LINK ERROR" which I interpret as a low level communication screw up between the cards. I think that simple interactions work okay (perhaps sending smaller packets to each other) but not bigger operations (perhaps maxing out the packet size) also leads me to think the cards are the issue.

Yes, O/S provided buffer sizes being exceeded could also cause an issue but I wouldn't expect to see "LINK ERROR" in this case.

Also, please confirm that you don't see anything about jumbo packet support in the CTRL-Q setup?

Confirmed, nothing for jumbo packets and I'll reconfirm tonight about the max frame size again and post here.

---------- Post updated at 07:28 PM ---------- Previous update was at 04:19 PM ----------

@agent.kgb
How would I set the LDEV here?

I've never worked with SCST - I don't know.

@hicksd8
Only options for the Max Frame Size is 512, 1024 or 2048 on the QLogic card. However what you said makes perfect sense and fits in with the issue well. Need to get the details on the LDEV setting from agent.kgb next. What command do I run to set the LDEV?

---------- Post updated at 08:21 PM ---------- Previous update was at 08:13 PM ----------

Oh you meant on the target, thought you meant on AIX. Here's the config and it looks fine per the developers:

Code:

        TARGET 21:01:00:1b:32:a1:81:21 {
                HW_TARGET

                enabled 1
                rel_tgt_id 2

                GROUP IBM01 {
                        LUN 0 MDSVIOroot01

                        INITIATOR 10:00:00:00:C9:62:1B:82

                        INITIATOR 10:00:00:00:C9:62:1B:83
                }
        }

So it's set to LUN 0. Then what you are saying agent.kgb is that this line here:

000002 0000000000000000 2101001b32a18121 2001001b32a18121

should really read:

000000 .....

?

Cheers,
DH

---------- Post updated at 09:59 PM ---------- Previous update was at 08:21 PM ----------

Anyway to change the Max Frame Size on AIX then? If I can match the two I could find out if they were at play.

---------- Post updated at 10:05 PM ---------- Previous update was at 09:59 PM ----------

Perhaps the Emulex frame size reported of 2112 is 2048 + 64 bytes for the headers and the QLogic doesn't add the 64 bytes to the 2048 it shows in the QLogic Boot Menus? If so they would match. Or am I wrong about this math? The ./hbainfo utility is dated and the devscan doesn't display the frame size. So I'm not 100% here.

Cheers,
DH

---------- Post updated at 11:56 PM ---------- Previous update was at 10:05 PM ----------

Apologies, in case I didn't mention, this is a HBA to HBA FC configuration / test.

---------- Post updated 03-18-16 at 03:20 AM ---------- Previous update was 03-17-16 at 11:56 PM ----------

The PCI entries do not show up for me in the Power5 firmware any longer. I'm wondering if there is anything out of the norm with it. How can I check the Power5 PCI bus for any issues?

I was talking about the packet size at link level. The standard FC (original) max frame size is 2148 which is a 2112 payload (data handed over the bus from the OS) plus 36 bytes of frame construction (header, footer, checksum, et al) put on by the adapter in order to transmit to the target. That was always standard.

Then, because the link could become a bottle neck with packets queuing for transmission, the standard was amended to allow "jumbo" packets (basically the amalgamation of multiple packets) of up to 9000 bytes which after frame construction added 36 bytes to that giving a max frame size of 9036.

Some adapters support jumbo and some don't. Some allow you to switch on/off support in the HBA BIOS, and some don't.

Needless to say, if one HBA is supporting jumbo and the other not, then a frame can arrive that cannot be unpacked.

Now, further up the netstack (in the OS) the payload (just the data element) will be a lower number, perhaps 2048, I'm not sure. That in turn will be included inside a larger packet (perhaps 64 bytes larger) to construct the payload (2112) to be handed down to the HBA for transmission. Therefore, as I said before, the size and number of buffers allocated by the OS also plays a part in this. If the buffer size is configured too small then the incoming packets won't fit in and the packet cannot be captured and unpacked.

However, with a LINK ERROR reported (by the HBA through the hostbus to the OS) I think that the HBA's are having difficulty. Do you have any other cards of the same type on site? We cannot rule out a hardware fault with one of the HBA's. If you have spares I would certainly give it a shot.

So to answer your question, no, I don't understand how the QL is offering just 512, 1024 and 2048 as max frame size options. I'll do some research on that. Perhaps you should give QL support a call to discuss that point.

I'm actually chatting with QL support on this. They've been helping me out in last 1-2 months and we're having difficulty identifying.

I'm about to order some QL cards for the Power5 (older ones) just to even things out a bit and isolate better.

I'm still looking at the Jumbo Frame size on both cards to see if I haven't missed anything. That makes alot of sense.

---------- Post updated at 09:50 AM ---------- Previous update was at 09:39 AM ----------

I reverted to looking on the web and I just don't see 'Jumbo' listed alongside QLA2464 anywhere. Nor do I see anything for Emules LPE11002. I'm doing direct Fiber to Fiber here. There is no switch or iSCSI or FCoE here.

I've done some research on the web and found this page regarding the OS buffer for fibre channel configuration:

AIX Health Check - AIX fibre channel error - FCS_ERR6

although I appreciate that this is not the error you are getting (FCP_ERR4) it does talk about FC crashing under pressure which is what you're experiencing. Configuring the OS availability of FC buffers is like managing a railway station with lots of trains arriving; you need platforms to accommodate the longest trains (max buffer size) and enough platforms to accommodate all the trains in rush hour (total number of buffers). If either isn't sufficient traffic will get dropped.

Also, if you Google your actual error "FCP_ERR4" there's loads to read about.