Shell Programming and Scripting

Hi Friends !!

I am facing a hash total issue while performing over a set of files of huge volume:

Command used:

Code:

tail -n +2 <File_Name> |nawk -F"|" -v '%.2f' qq='"' '{gsub(qq,"");sa+=($156<0)?-$156:$156}END{print sa}' OFMT='%.5f'

Pipe delimited file and 156 column is for hash totalling.

File 1:

Record count is 254368

Absolute Sum in DB is 23840949436509.39

Absolute Sum using above script is 23840949436510.18750

File 2:

Record count is 2580100

Absolute Sum in DB is 7305817400402102.5619993295

Absolute Sum using above script is 7305817400403184.00000

Kindly help me in resolving this issue and do suggest me if any better way to do absolute hash totalling for huge volume.

Thanks in advance,
Ravi

Moderator's Comments:

edit by bakunin: you are welcome but you would be even more welcome if you would use these CODE-tags for your code. Thank you for using them yourself from now on.

awk uses 32-bit floating point numbers which do not have infinite precision -- they have at best 9 decimal digits precision. If you want infinite precision like a database will do, try the bc utility.

Hi Corona..

Can you help me with bc utility for this scenario ? I am just new to these functions !

Regards,
Ravi

Assuming that you're using the tail in the command line:

Code:

tail -n +2 <File_Name> |nawk -F"|" -v '%.2f' qq='"' '{gsub(qq,"");sa+=($156<0)?-$156:$156}END{print sa}' OFMT='%.5f'

to discard the 1st two lines of your input file because they contain headers that you don't want included in your output, that the '%.2f' didn't really appear in the command line you executed (since that would be a syntax error for nawk), that you don't really want the output rounded to five digits after the decimal point in the output (as would be done in your command line by OFMT='%.5f', and assuming that field #156 in the other lines in your input file contains a double quoted string containing a string of digits with no more than one period and with an optional leading minus sign (which you want to be ignored), you could try something like:

Code:

nawk -F'|' -v dqANDms='["-]' '
BEGIN { f=156
        printf("0")
}       
NR > 2 {gsub(dqANDms, "", $f) 
        printf("+%s", $f)
}
END {   printf("\n")
}' <File_Name> | bc

Depends what your scenario is, I don't know yet, all I have is a program which doesn't do what you want...

Hi Don !

Thanks for the work around solution and it is working fine for small files, but when I execute large files..facing below error:

Code:

 
0705-001: bundling space exceeded on line 1 stdin

Kindly help me in this regard.

Regards,
Ravichander

Making the assumption that that error code is coming from bc, you could try:

Code:

awk -F'|' -v dqANDms='["-]' '
BEGIN { f=156
        printf("s=0\n")
}
NR > 2 {gsub(dqANDms, "", $f)
        printf("s+=%s\n",  $f)
}
END {   printf("s\n")
}' file | bc