IP Networking

Port forwarding is usually for firewall traversal. Two flavors are possible: the tcp connection may actually terminate on the intermediate host with the forwarder making a second connection and shuffling all the data (a.k.a. tcp_relay), or something similar to NAT can adjust the port, host and checksum on a packet by packet basis. It is not a solution that scales well, expecially with the first flavor, which begs the question of what you are trying to achieve. Protocols that put port or host informaiton into the data present problems. NAT has logic for FTP, one of the first such, and rewrites data where it seems appropriate, but maybe not for your app.

Any host with more than one IP device can become a router if packet forwarding is turned on. Incoming packets not for this host are routed out when they hit the middle of the IP stack. This is often not desirable, as the second NIC may be for fail-over or another somewhat incompatible sort of traffic, like backups.

You need two static routes because the flow of the packets in one direction is completely independent of the other direction. Sometimes packets in the other direction have a different path to increase throughput or protect one network from the higher traffic on the other, e.g., a web server might download on a second network so the first is free of the many large packets and high volume. For a 80/20 i/o volume split, it gives your network 25% more bandwidth before the busy side chokes. However, this can be a problem for dynamic NAT, as the second firewall does not know what IP the first has reassigned.

I think, using a router between S2 &S3 can resolved this issue easily. And I think a router already exists between S2 & S3 cause the P2P connectivity is maintaining by a vendor who provide remote fiber connection. Lets see, what happen. I will update you guys.

Thanks a lot everyone for your valuable comments.

Out of sheer interest on my part: could you try the interface alias on S3 and post the result?