Howdy Ygor and summer_cherry

Thanks for trying to help!


Okie, lets see now:

When I tried Ygor's script:

Code:

gawk -f test.awk cbl-block-list.txt > cbl-block-list.p2p

After a few minutes my puter froze and bam... this error popped up.

Code:

gawk: test.awk:5: (FILENAME=cbl-block-list.txt FNR=3102944) fatal: format_tree:
obuf: can't allocate memory (Not enough space)

I believe I didn't mention the hardware specs of my PC and the OS:
3GB (thats 2x1GB and 2x512MB, not 4 GB on a 32bit OS), 2x dual core Intel proc (that's dual proc mobo), 4 HDDs of which two are in mirror and two are striped for better filesystem performance.

I still believe the prog should've had enough free mem to perform... but as I mentioned before - the file I need to run aggregation on is around 300-400MB in size (after being converted from initial 100-150MB of pure unique IPs) - that's not a small chunk =) - so it might have attempted a huge allocation of memory as it worked its way through array creations?
If you want, I can provide you the file via some file-exchange service so you can test it on or something?

The OS is WinXP 32bit since that's where protowall is located, but scripts are all running via unxutils.

I tried running the script on a 6MB file containing just IP addresses:

Code:

gawk -f test.awk small-chunk.txt > small-chunk.p2p
gawk: test.awk:9: (FILENAME=small-chunk.txt FNR=443897) fatal: function `asorti'
 not defined

Will take a peek and see whats with the asorti function.


When I tried to run summer_cherry's script:

Code:

 
gawk -f test2.awk cbl-block-list.txt > cbl-block-list.p2p
gawk: test2.awk:3: while(<DATA>){
gawk: test2.awk:3: ^ parse error
gawk: test2.awk:3: while(<DATA>){
gawk: test2.awk:3:              ^ parse error
gawk: test2.awk:5:      my @tmp=split("[.]",$_);
gawk: test2.awk:5:         ^ invalid char '@' in expression

Oopsie... did I just do that? Guess my gawk doesn't like it Perhaps my perl would like it better?

I am running gawk via unxtools and not on a native unix so environment, so that might be a slight problem in some cases.

Lets see if we (well - you is more correct ) can make these scripts work. Har Har
I still gotta properly figure out the fine magic behind those lines.

And again - thanks a bunch for helping me work this one out!

Cheers,

Matt

PS. My appologies on broken English .)