Shell Programming and Scripting

Hi Don,
Thanks for your help.The code you have mentioned is working.One clarification is this providing us distinct customer_id by removing duplicates.

Code:

[omnidevint@sftp311 full_01jan13_31aug15]$ awk -F'|' '
> !(($38 + 0) in a) {
> a[$38 + 0]
> c++
> }
> END {print c
> }' *.csv
3051724

As I'm new to awk,could you kindly explain the above block for my understanding.Else apart from using awk can we do this in any other way.

The shell runs awk code in a multi-line 'string'

Code:

awk '
awk code
'

In the awk code:
$38 is column 38.
a[ ] is a string-addressed array (hash).
For each line:
If $38 is not in a (a[$38] does not exist), then the following code in braces is run.
It defines a[$38] (without assigning a value) and increases c (initially variables are 0 or empty).
If all lines of all given files are processed it processes the END section: print c.

c is incremented if a[$38] does not exist. But not if the same $38 is met again because then a[$38] exists.

In addition to what MadeInGermany has already said, you might note that I used $38 + 0 instead of just $38. The awk language allows fields to be interpreted as strings or as numbers. By adding zero to a field's value, we force the field to be treated as a number. So, if some of you field 38 values have leading zeros and some don't, adding 0 will force customer_id 000255 and customer id 255 to be treated as a single value. If we treated field 38 as a string, those two values would be different customer_ids. Note that when I presented the code, you hadn't given us any examples and we didn't know if that field would contain integer values or floating point values. Adding 0 also makes awk treat 123 and 123.000 and 123.0 and 1.23e+2 as a single value even though they are four different strings.