Solaris

Somebody HELP!

I have a huge log file (TEXT) 76298035 bytes.

It's a logfile of IMEIs and IMSIS that I get from my EIR node.

Here is how the contents of the file look like:

000000,
1 33016382000913 652020100423994
1 33016382002353 652020100430743
1 33017035101003 652020100441736
....
....
....
235800,
1 35725620987678 652020100545862


Problem is, the file is to some degree made huge by repeated entries ( repeated lines - non consecutive).

I have tried this code to eliminate the repeated entries:

cat myfile | sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' | tee mynewfile | wc -l

but it takes forever and stops midway, at 024000 instead of 235800.

No guarantee that this will work for very large files, but this should remove non-consecutive duplicates...

awk '!($0 in a);{a[$0]=1}' logfile

Ygor,

I am getting:

awk: syntax error near line 1

awk: bailing out near line 1

If on Solaris use nawk.

It worked! Salutes to the Moderator.

Thanx man, you saved my day!