Shell Programming and Scripting

Hello everyone,
As the title suggests, I am attempting to test adding gzip compression to a connection to an application I am testing. Currently I have the application set up with httptunnel, which forwards the connection to the remote host.

I would like to use a script to intercept the connection between the application and httptunnel, gzip the data that is there, and then forward it on. On the other side, the opposite will happen with gzip -d, with the resulting data passed on to the server application.

However I need some help, as I am fairly new to some of the things which are required to do this, like mkfifo, as well as piping outputs. I think there may be some other problems implementing such an idea with multiple connections, but that is a problem for later.

I have created a connection which works fine for sending gz files, and a while loop to keep things going.

Code:

while true ; do nc -l 3333 | gzip -d ; done

and on the other side, simply

Code:

 nc 192.168.20.84 3333 < a.gz

Can someone assist in developing this so that I can use this method to compress the connection on one side ,and decompress on the other side?

The following is an attempt to do what what needs to be done on one side, but on other, I am not so sure about decompressing and passing data on. httptunnel is listening on 8080 of course.

Code:

#!/bin/sh
mkfifo /tmp/gz.tmp
while true; do
cat gz.tmp | gzip | nc -l -k localhost 9123 | nc localhost 8080 ;
done

Secondly, I know that this will only compress data travelling in one direction. I would like to have the same idea operating in reverse, from the server back to the host. Would this be possible? This might be quite complicated, due to dynamic source ports. Having gzip know what to do with the data might be another problem. Perhaps attempting "gzip -d" first, then if it gives a fail code, just run "gzip". Is there a cleaner way of detecting gzip compression, if it is needed?

Any help would be greatly appreciated.

Perhaps other tools can be used for the job? ssh, for instance, can forward ports, and can be told to compress data as well with -S...

---------- Post updated at 01:16 PM ---------- Previous update was at 01:07 PM ----------

For instance:

Code:

ssh -S -L 3333:hostname:8080

This will open port 3333 on 127.0.0.1 on your local machine, which you can connect to at will.

Whenever you do, the far side of the ssh connection will connect to hostname on 8080.

The -S will cause all this traffic to be compressed inbetween.

If you're concerned about performance or CPU use you can fiddle with the cipher options, the blowfish chiper is meant to be very fast.

Thanks for your response.

Yes, ssh is a possible solution, however I can't use it in this case.

I should have added more detail. The problem is that I can't use protocols like ssh or even other applications like stunnel for using ssl. The assumption is that the connection the program is being used on is heavily firewalled. In such a case, an http connection may be the only kind that works. The idea is that by using gzip, any data that goes through the http connection will be compressed, and also, the data will be transformed in a way that will allow it to escape detection from things like regex filtering.

Well that is in theory. If I can get this part working, I will then start onto using suricata to perform on-the-fly gzip decompression and regex filtering.

I have a normal tunnel now working correctly, with

Code:

 while true ; do cat /tmp/rev | nc -l 3333 | nc localhost 8080  > /tmp/rev ; done

I just need to figure out where to put the gzip commands. Furthermore, would a similar command be used on the server, or would it need altering?

Thanks again in advance for any help.

So, port 23 is blocked, but port 8080 isn't?

...Oh, I see.

Since the protocol you're running over port 80 is most blatantly not going to be HTTP anyway, why not run an ssh server on port 80 or 443 instead of netcat? You can do that.

Then just

Code:

ssh -p 80 username@host -L 3333:hostname:8080

Hi again
I know that in most cases using ssh or ssl would be fine. Indeed I have both stunnel (ssl) and ssh versions of this connection working fine, with ssl being slightly faster.

However the overall objective is to demonstrate suricata's ability to decompress gzip data on the fly and detect patterns vs regular methods like iptables/layer7, which just look at the plaintext pattern rather than putting it together.

To do this, I will of course need a connection which uses gzip compression.

If I can get this nc/gzip connection working, I then show that a regular layer7 rule will not be able to detect regex pattens that the test program ouputs. However it is expected that suricata will be able to detect the patterns.

In the case of using http to form the connection, you are right, it is obvious from the traffic pattern that it isn't a standard http connection. However this isn't relevant as yet.

I am attempting firstly to get the nc port forwarding working on both the server and host. Once this is working I will look again at putting in gzip commands. I have a similar command on both machines, however on the server, things do not work properly. I have the server set up with the http tunnel server to receive, then forward packets to port 8008. I am then using this command (after mkfifo /tmp/rev)

Code:

while true ; do cat /tmp/rev | nc -l 8008 | nc localhost 80 > /tmp/rev ; done

with the test server program listening on port 80.

I notice when I enter this command, cpu usage increases dramatically and I get a lot of errors; something isn't right

Code:

TCP: time wait bucket table overflow

If I use this command on the host (with different ports), it forwards correctly between the test program and the tunnel application.

Edit: Well that part is fixed. Turns out the server was using a different version of nc. I had to add "-p" to the listening nc. Now I just need to add gzip/gzip -d in the right places.

---------- Post updated at 08:30 PM ---------- Previous update was at 05:47 PM ----------

Well I have made a little progress. It seems as though there is a problem, possibly to do with gzip not knowing when to stop compressing, although I could be wrong. I am using the following command on both sides. The link works fine if I remove the gzip parts.

Code:

while true ; do cat /tmp/rev | gzip -v | nc -l (-p )8080 | gzip -v | nc localhost 80 | gzip -d > /tmp/rev ; done

Can anyone assist here? If its necessary to do so, is there a way to inform gzip to compress until x bytes, then output? If what I am attempting to achieve is not possible, please let me know so I don't waste too much time!

Again, any help would be greatly appreciated.Thanks

Hmmmm. How about two, separate netcat connections then, one going each way?

I'm not sure how the ability to decompress gzip on the fly is useful. Wouldn't that make it bigger?

So send a gzip over netcat... I have not been able to make nc do what you want yet, despite much fiddling and trying. I have tried to do it the way you want. Unfortunately the way you want is silly. Anything which takes >1 named pipes to kludge into place cannot be the right tool for the job.

Do you have the C language available for a custom solution to be written?