Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Script to count particular type of records

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Script to count particular type of records
# 1  
Old 06-27-2011
Script to count particular type of records
Hi,

I have a huge file containing thousands of records which are of following pattern:

Code:
TYPE1
{
originNodeType : "IVR"
originHostName : "AAIVR"
originTransactionID : "01310559"
originTimeStamp : "20110620192440+0530"
hostName : "hhhh"
voucher : '0'D
rProfileID : "ZZZZ"
Before
{
Flags : "10000000"
Bal : "123"
}
After
{
Flags : "10000000"
Bal : "123"
}
}
TYPE1
{
originNodeType : "UGW"
originHostName : "AAUGW"
originTransactionID : "01310560"
originTimeStamp : "20110620192441+0530"
hostName : "asdf" 
voucher : '1'D
Before
{
Flags : "00000000"
Bal : "123"
}
After
{
Flags : "10000000"
Bal : "124"
}
}
TYPE2
{
originNodeType : "IVR" 
originHostName : "AAIVR"
originTransactionID : "01462112"
originTimeStamp : "20110624001221+0530"
hostName : "aqws"
Counter : '1'D
errorCode : 'invalidNumber (19)' 
}
TYPE2
{
originNodeType : "UGW" 
originHostName : "AAUGW"
originTransactionID : "01462113"
originTimeStamp : "20110624001221+0530"
hostName : "aqws"
Counter : '2'D
errorCode : 'invalidAcc (18)' 
}
TYPE2
{
originNodeType : "IVR" 
originHostName : "AAIVR"
originTransactionID : "01462112"
originTimeStamp : "20110624001222+0530"
hostName : "aqws"
Counter : '1'D
errorCode : 'invaliddate (17)' 
}


I need to extract following data from this type of file:
1. Number of TYPE1 records where
originNodeType : "IVR" AND voucher : '0'D AND rProfileID : "ZZZZ"

2. Count of different type of errorCode from TYPE2 records where originNodeType : "IVR". For data above output should be:

Code:
errorCode : 'invalidNumber (19)' = 1
errorCode : 'invaliddate (17)' = 1


Fileds hostname & rProfileID are optional so they may not be present in all records.

As number of records is large so would request if someone can help with perl based script.

Thanks,
Madhukar
Moderator's Comments:
Mod Comment
Please use code tags when posting data and code samples!
Last edited by vgersh99; 06-27-2011 at 03:36 PM.. Reason: code tags, please!
# 2  
Old 06-27-2011
Hi,

I can't understand what are you looking for.

1.- From your first condition, my count is 1.
2.- From your second condition, my count is 2.

What does your output mean?

Regards,
Birei
# 3  
Old 06-27-2011
Based on your requirements, the solution for step (2) can be very complex as you did not specify how many different
error codes exists, thus I just display the error and you can use another tool to count the different ones.

Here is what I came up with:
Code:
#!/usr/bin/ksh
typeset -i mCnt1=0
mLOrig='originNodeType : "IVR"'
mLVouch="voucher : '0'D"
mLProf='rProfileID : "ZZZZ"'
mLONType='originNodeType : "IVR"'
mLErr="errorCode :"
while read mLine; do
  mP1to4=$(echo ${mLine} | cut -c1-4)
  if [[ "${mP1to4}" = "TYPE" ]]; then
    if [[ "${mLine}" = "TYPE1" ]]; then
      if [[ "${mOrig}" = "Y" && "${mVouch}" = "Y" && "${mProf}" = "Y" ]]; then
        mCnt1=${mCnt1}+1
      fi
      mType="1"
      mOrig="N"
      mVouch="N"
      mProf="N"
    else
      if [[ "${mLine}" = "TYPE2" ]]; then
        mType="2"
        mONType='N'
      else
        mType="?"
      fi
    fi
  fi
  if [[ "${mType}" = "1" ]]; then
    if [[ "${mLine}" = "${mLOrig}" ]]; then
      mOrig='Y'
    else
      if [[ "${mLine}" = "${mLVouch}" ]]; then
        mVouch='Y'
      else
        if [[ "${mLine}" = "${mLProf}" ]]; then
          mProf='Y'
        fi
      fi
    fi
  fi
  if [[ "${mType}" = "2" ]]; then
    if [[ "${mLine}" = "${mLONType}" ]]; then
      mONType='Y'
    else
      if [[ "${mONType}" = "Y" ]]; then
        mP1to11=$(echo ${mLine} | cut -c1-11)
        if [[ "${mP1to11}" = "errorCode :" ]]; then
          echo "TYPE2 IVR ${mLine}"
        fi
      fi
    fi
  fi
done < input_file
echo "Found ${mCnt1} condition in TYPE1"

# 4  
Old 06-28-2011
Thanks shell_life

Is there someway to keep the pattern search mLOrig='originNodeType : "IVR"' to be space insensitive. Like we get match in this case and even if we have 'originNodeType : "IVR"' (multiple blank or tabs)?


Hi birei,

Yes in the sample file the output which you mentioned is ok. I need count.
In sample pattern below, count for TYPE2 record with sample pattern should be 2.
Code:
TYPE1
{
originNodeType : "IVR"
originHostName : "AAIVR"
originTransactionID : "01310559"
originTimeStamp : "20110620192440+0530"
hostName : "hhhh"
voucher : '0'D
rProfileID : "ZZZZ"
Before
{
Flags : "10000000"
Bal : "123"
}
After
{
Flags : "10000000"
Bal : "123"
}
}
TYPE1
{
originNodeType : "IVR"
originHostName : "AAIVR"
originTransactionID : "01310559"
originTimeStamp : "20110620192440+0530"
hostName : "hhhh"
rProfileID : "ZZZZ"
voucher : '0'D
Before
{
Flags : "10000000"
Bal : "123"
}
After
{
Flags : "10000000"
Bal : "123"
}
}
TYPE1
{
originNodeType : "UGW"
originHostName : "AAUGW"
originTransactionID : "01310560"
originTimeStamp : "20110620192441+0530"
hostName : "asdf"
voucher : '1'D
Before
{
Flags : "00000000"
Bal : "123"
}
After
{
Flags : "10000000"
Bal : "124"
}
}
TYPE2
{
originNodeType : "IVR"
originHostName : "AAIVR"
originTransactionID : "01462112"
originTimeStamp : "20110624001221+0530"
hostName : "aqws"
Counter : '1'D
errorCode : 'invalidNumber (19)'
}
TYPE2
{
originNodeType : "UGW"
originHostName : "AAUGW"
originTransactionID : "01462113"
originTimeStamp : "20110624001221+0530"
hostName : "aqws"
Counter : '2'D
errorCode : 'invalidAcc (18)'
}
TYPE2
{
originNodeType : "IVR"
originHostName : "AAIVR"
originTransactionID : "01462112"
originTimeStamp : "20110624001222+0530"
hostName : "aqws"
Counter : '1'D
errorCode : 'invaliddate (17)'
}


Thanks,
Last edited by Franklin52; 06-28-2011 at 03:46 AM.. Reason: Please use code tags for code and data samples, thank you
# 5  
Old 06-28-2011
This code will work on mixed cases and multiple spaces:
Code:
#!/usr/bin/ksh
typeset -i mCnt1=0
mLOrig='originnodetype:"ivr"'
mLVouch="voucher:'0'd"
mLProf='rprofileid:"zzzz"'
mLONType='originnodetype:"ivr"'
mLErr="errorcode:"
tr -A '[:upper:]' '[:lower:]' < input_file | tr -d ' ' |
while read mLine; do
  mP1to4=$(echo ${mLine} | cut -c1-4)
  if [[ "${mP1to4}" = "type" ]]; then
    if [[ "${mLine}" = "type1" ]]; then
      if [[ "${mOrig}" = "Y" && "${mVouch}" = "Y" && "${mProf}" = "Y" ]]; then
        mCnt1=${mCnt1}+1
      fi
      mType="1"
      mOrig="N"
      mVouch="N"
      mProf="N"
    else
      if [[ "${mLine}" = "type2" ]]; then
        mType="2"
        mONType='N'
      else
        mType="?"
      fi
    fi
  fi
  if [[ "${mType}" = "1" ]]; then
    if [[ "${mLine}" = "${mLOrig}" ]]; then
      mOrig='Y'
    else
      if [[ "${mLine}" = "${mLVouch}" ]]; then
        mVouch='Y'
      else
        if [[ "${mLine}" = "${mLProf}" ]]; then
          mProf='Y'
        fi
      fi
    fi
    continue
  fi
  if [[ "${mType}" = "2" ]]; then
    if [[ "${mLine}" = "${mLONType}" ]]; then
      mONType='Y'
    else
      if [[ "${mONType}" = "Y" ]]; then
        mP1to10=$(echo ${mLine} | cut -c1-10)
        if [[ "${mP1to10}" = "${mLErr}" ]]; then
          echo "TYPE2 IVR ${mLine}"
        fi
      fi
    fi
  fi
done
echo "Found ${mCnt1} condition in TYPE1"

This User Gave Thanks to Shell_Life For This Post:
madhukar1anand
# 6  
Old 06-28-2011
ThanksSmilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Count occurrence of string (based on type) in a column using awk

Hello, I have a table that looks like what is shown below: AA BB CC XY PQ RS AA BB CC XY RS I would like the total counts depending on the set they belong to: if search pattern is in {AA, BB, CC} --> count them as Type1 | wc -l (3 Replies)
Discussion started by: Gussifinknottle
3 Replies

2. Shell Programming and Scripting

Count records in a block

Hi, We have a file that has a structure like this: H10 1 2 3 D10 1 D20 1 2 3 D20 3 4 5 D20 4 5 6 D10 2 D20 1 2 3 D20 3 4 5 D20 4 5 6 S10 10 H10 1 2 3 D10 1 D20 1 2 3 D20 3 4 5 D20 4 5 6 D10 2 (2 Replies)
Discussion started by: jerome_rajan
2 Replies

3. Shell Programming and Scripting

UNIX Script required for count the records in table

Hi Friends, I looking for the script for the count of the records in table. and then it's containg the zero records then should get abort. and should notify us through mail. Can you please help me out in this area i am lacking. (5 Replies)
Discussion started by: victory
5 Replies

4. Shell Programming and Scripting

How to count dup records in file?

Hi Gurus, I need to count the duplicate records in file file abc abc def ghi ghi jkl I want to get below result: abc ,2 abc, 2 def ,1 ghi ,2 ghi, 2 jkl ,1 or abc ,2 def ,1 (3 Replies)
Discussion started by: ken6503
3 Replies

5. Shell Programming and Scripting

how to count how many subdirectory containing more than a certain number of specific file type

hi I want to write a script which count the number of subdirectories in the current root directory that contain more than a specified number of files of a specific type. Is there an easy way to do this? Thanks Robert (2 Replies)
Discussion started by: piynik
2 Replies

6. Shell Programming and Scripting

count and compare no of records in bash shell script.

consider this as a csv file. H,0002,0002,20100218,17.25,P,barani D,1,2,3,4,5,6,7,8,9,10,11 D,1,2,3,4,5,6,7,8,9,10,11 D,1,2,3,4,5,6,7,8,9,10,11 D,1,2,3,4,5,6,7,8,9,10,11 D,1,2,3,4,5,6,7,8,9,10,11 T,5 N i want to read the csv file and count the number of rows that start with D and... (11 Replies)
Discussion started by: barani75
11 Replies

7. Shell Programming and Scripting

Awk to Count Records with not null

Hi, I have a pipe seperated file I want to write a code to display count of lines that have 20th field not null. nawk -F"|" '{if ($20!="") print NR,$20}' xyz..txt This displays records with 20th field also null. I would like output as: (4 Replies)
Discussion started by: pinnacle
4 Replies

8. Shell Programming and Scripting

count characters in specific records

I have a text file which represents a http flow like this: HTTP/1.1 200 OK Date: Fri, 23 Jan 2009 17:16:24 GMT Server: Apache Last-Modified: Fri, 23 Jan 2009 17:08:03 GMT Accept-Ranges: bytes Cache-Control: max-age=540 Expires: Fri, 23 Jan 2009 17:21:31 GMT Vary: Accept-Encoding ... (1 Reply)
Discussion started by: littleboyblu
1 Replies

9. UNIX for Dummies Questions & Answers

Count records in a zip file

Hello, I searched the forums on the keywords in the title I used above, but I did not find the answer: Is it possible to count records in a .zip file on an AIX machine if i don't have pkunzip installed? From all the research I'm reading in google and the reading of pkunzip in Unix.com,... (3 Replies)
Discussion started by: tekster757
3 Replies

10. Shell Programming and Scripting

Count No of Records in File without counting Header and Trailer Records

I have a flat file and need to count no of records in the file less the header and the trailer record. I would appreciate any and all asistance Thanks Hadi Lalani (2 Replies)
Discussion started by: guiguy
2 Replies
Login or Register to Ask a Question