Shell Programming and Scripting

My original code spits out about 2.5 Gb (which is less than the full dataset) after about 70 min and then never completes.

I tried Chubler_X's code above and terminated it after 16 hours with no output.

One way I can definitely split without duplicates spanning split files is to split on date. It only takes about one minute to split the data in two with the following code:

Code:

awk -F, -v startDT="$startDate" -v endDT="$endDate" '
            BEGIN { s=mktime(startDT); e=mktime(endDT)}
            NR == 1 {print}
            NR > 1 { t = $6; gsub(/\-|:/, " ", t); t = mktime(t); if ( s <= t && t <= e) {print} }'

Mike

I take it sorting is absolutely out of the question...? It would safely handle files of arbitrary size.

Perl would be better for comparing via hashes. Doing an md5 or the like in awk would mean calling an external md5 utility 30 million times, where you can at least get a built-in module for Perl.

Would definately want a built-in esp in a Windows environment where it is very efficient to create new threads but new processes come with a lot of extra overhead.

Sorting is not out of the question unless it has serious performance problems just like the duplicate removal does.

Mike

Did you try my perl code? I have fairly high hopes it could do the job.

Still no reply -- is this 32 bit windows running cygwin running whatever? If so, the 4G address space can make hash tools fail, often not gracefully, and often well before the 4G nominal limit, like 1.7G, stumbling over some signed int4 in the process and the address space usage of code and other data. It sounds like you need a 64 bit CPU and O/S.

Once you run past the RAM, the sequential reading and writing of sort may outperform the random activity of hash. Also, not all hash are written for dynamic expansion of hash bucket count, so the amount of linear searching inside the bucket may increase. In rouguewave, for instance, you should set the bucket count according to the size of the set at the start. Extendible hashing - Wikipedia, the free encyclopedia

Sorry, It did not appear to me that you were asking me a question in your response.
I am running 64-bit windows with dual i7-2860QM CPUs.
It looks like CYGWIN 1.7.17 is 32-bit. 64-bit started with 1.7.22 in july of last year. I will try it.

Mike

Agreed it would be a very tight squeeze to solve in-memory on a 32bit environment with 30M records we would only get about 50bytes per record to play with. This is why large datasets are usually stored in databases.