Help building the logic for a script - Shell Programming and Scripting

fahadaizaz · #1 (**permalink**) 07-08-2008

Hi guys,

I am new to shell scripting, i need your help to tackle a problem.
I have a single file, sample is below:

Code:

2008:07:08 07:01:14.360 (tid 4) INFO no bonus notifications to send
2008:07:08 07:01:50.823 (tid 1) INFO Database cleaned of all stale bonus records order than 30 days
2008:07:08 07:01:50.823 (tid 1) INFO No files matching the pattern were found
2008:07:08 07:02:14.770 (tid 4) INFO no bonus records to credit
2008:07:08 07:02:14.770 (tid 4) INFO no bonus notifications to send
2008:07:08 07:02:51.309 (tid 1) INFO Database cleaned of all stale bonus records order than 30 days
2008:07:08 07:02:51.309 (tid 1) INFO Found file: /var/ftp/emm/AnnivBonus_from_20080607_to_20080707_at_20080708_060001.txt
2008:07:08 07:02:51.316 (tid 1) INFO line [ msisdn = 133762, bonus = 56284, total usage = 562842 ]
2008:07:08 07:02:51.323 (tid 1) INFO Recorded bonus
2008:07:08 07:02:51.323 (tid 1) INFO line [ msisdn = 295361, bonus = 12315, total usage = 123153 ]
2008:07:08 07:02:51.324 (tid 1) INFO Recorded bonus
2008:07:08 07:02:51.324 (tid 1) INFO line [ msisdn = 130174, bonus = 16147, total usage = 161471 ]
2008:07:08 07:02:51.324 (tid 1) INFO Recorded bonus
2008:07:08 07:02:51.324 (tid 1) INFO line [ msisdn = 804610, bonus = 16287, total usage = 162873 ]
2008:07:08 07:02:51.324 (tid 1) INFO Recorded bonus
2008:07:08 07:02:51.324 (tid 1) INFO line [ msisdn = 781860, bonus = 26363, total usage = 263636 ]
2008:07:08 07:02:51.324 (tid 1) INFO Recorded bonus
2008:07:08 07:02:51.325 (tid 1) INFO line [ msisdn = 459960, bonus = 12342, total usage = 123426 ]
2008:07:08 07:02:51.325 (tid 1) INFO Recorded bonus
2008:07:08 07:02:51.325 (tid 1) INFO line [ msisdn = 910676, bonus = 17475, total usage = 174758 ]

Now the same file contains the following records

Code:

2008:07:08 07:03:20.655 (tid 4) INFO AIR credit operation success [ msisdn = 773210]
2008:07:08 07:03:20.656 (tid 4) INFO Bonus required [ msisdn = 113220, bonus = 18960, total usage = 189606 ]
2008:07:08 07:03:20.687 (tid 4) INFO AIR credit operation success [ msisdn = 113220]
2008:07:08 07:03:20.688 (tid 4) INFO Bonus required [ msisdn = 334050, bonus = 14184, total usage = 141849 ]
2008:07:08 07:03:20.719 (tid 4) INFO AIR credit operation success [ msisdn = 334050]
2008:07:08 07:03:20.720 (tid 4) INFO Bonus required [ msisdn = 893241, bonus = 64106, total usage = 641063 ]
2008:07:08 07:03:20.750 (tid 4) INFO AIR credit operation success [ msisdn = 893241]
2008:07:08 07:03:20.751 (tid 4) INFO Bonus required [ msisdn = 922615, bonus = 15175, total usage = 151756 ]
2008:07:08 07:03:20.781 (tid 4) INFO AIR credit operation success [ msisdn = 922615]
2008:07:08 07:03:20.782 (tid 4) INFO Bonus required [ msisdn = 150242, bonus = 14307, total usage = 143074 ]
2008:07:08 07:03:20.812 (tid 4) INFO AIR credit operation success [ msisdn = 150242]
2008:07:08 07:03:20.812 (tid 4) INFO Bonus required [ msisdn = 326001, bonus = 10709, total usage = 107092 ]
2008:07:08 07:03:20.842 (tid 4) INFO AIR credit operation success [ msisdn = 326001]
2008:07:08 07:03:20.842 (tid 4) INFO Bonus required [ msisdn = 132403, bonus = 18708, total usage = 187086 ]
2008:07:08 07:03:20.872 (tid 4) INFO AIR credit operation success [ msisdn = 132403]
2008:07:08 07:03:20.872 (tid 4) INFO Bonus required [ msisdn = 917950, bonus = 19989, total usage = 199896 ]

Now I want to get all the msisdns values (after = sign) from two different statements
INFO line [ msisdn = 130174, bonus = 16147, total usage = 161471 ]
INFO AIR credit operation success [ msisdn = 130174]

compare them, in case they are equal i want to output them like displayed below

Code:

20080708, 130174, 16147
.
.
.

The values that are not found in both sets of data can be output in an error file

Guys i need help to create this logix in UNIX/Solaris machine. So far i am unable to use the scripting knowledge that i have to perform this function.

Regards,

quine · #2 (**permalink**) 07-08-2008

So the essence of the problem is that when a USE for the credit (the "success" message) for a particular msisdn, you want to record all the "line" information for that msisdn. But the USE record comes AFTER the recording record. Is that the core of it?

I have a few questions...

1) Is this all being recorded moment by moment in one file like a log, or are the "USE" records in a separate file?

2) It makes sense that if there is a USE record where there has been no line record it would be an error.

3) Can you get two "line records" with the same msisdn without a USE record in between? If yes, you must account for this. How about the reverse? Two or mor uses for each recording? Either that is an error too, or you add up the credits or something, etc.

4) What happens if a line record never gets used? That is, how long do you hold (store) un-used "line records" before they become errors (if they ever do)?

I would start by storing the "line records" in a hash (perl). Basically you read along this file and every time you encounter a "line record" you store it using the msisdn as the index of the hash. As you come across a USE record (I'm assuming one log file here that is growing was you are reading it), you look for it's msisdn already in the hash (exists() in perl). If found, you have the line data as the value of the hash, and you can DELETE that value from the hash since it is accounted for, etc. That would be one way to approach it. If you don't have hashes, you will have to use an array, search it, etc. So long as the number of un-used line records stays low, performance shouldn't degrade.

snowline84 · #3 (**permalink**) 07-08-2008

Hi,
I need few clarifications in the requirements.
In the file, are we supposed to check only the lines that contains

"INFO Bonus required "

to the next line that contains

"INFO AIR credit operation success "???

If yes, we can go for a simpe script like

-----------------------------------------------------------------------

if [ $# -lt 1 ]
then
echo "Usage : " $0 "<file_name>"
exit 1
fi

echo "Going to scan the file : "$1

file_name=`echo $1`

prev_line=2

for cur_line in `cat $file_name | tr ' ' '+'`
do

echo $prev_line | grep "INFO+Bonus+required+" >test_file

if [[ -s test_file ]]
then
prev_msisdn=`echo $prev_line | tr ',' '+' | tr -s '+' | cut -d '+' -f 11`
cur_msisdn=`echo $cur_line | tr ']' '+' | tr -s '+' | cut -d '+' -f 13`

if [ "$prev_msisdn" -ne "$cur_msisdn" ]
then
echo "msisdn mismatch found"
echo "cur_msisdni : "$cur_msisdn
echo "prev_msisdn : "$prev_msisdn
fi
fi

prev_line=`echo $cur_line`

done
-----------------------------------------------------------------------

which works like

$> ./check_msisdn a
Going to scan the file : a
msisdn mismatch found
cur_msisdni : 113220
prev_msisdn : 13220

and the contents of file a were the same that you have provided except for a change in the line
------------------------------------------------------------
2008:07:08 07:03:20.656 (tid 4) INFO Bonus required [ msisdn = 13220, bonus = 18960, total usage = 189606 ]
2008:07:08 07:03:20.687 (tid 4) INFO AIR credit operation success [ msisdn = 113220]
---------------------------------------------------------------

fahadaizaz · #4 (**permalink**) 07-09-2008

@ quine

1) Is this all being recorded moment by moment in one file like a log, or are the "USE" records in a separate file?

Code:

Basically yes this is recorded moment by moment. But most of the time the file contains the polling information that is not required. 
Point is the file contains this data as well as other garbage stuff which is not required. 
Therefore you can consider the file as a static file which contains all the valid records and garbage values/lines.

2) It makes sense that if there is a USE record where there has been no line record it would be an error.

Code:

I can grep the data that i require into a seperate file meaning, the two lines
INFO line [ msisdn = 133762, bonus = 56284, total usage = 562842 ]

and

2008:07:08 07:03:20.655 (tid 4) INFO AIR credit operation success [ msisdn = 773210]

and do the processing.

3) Can you get two "line records" with the same msisdn without a USE record in between? If yes, you must account for this. How about the reverse? Two or mor uses for each recording? Either that is an error too, or you add up the credits or something, etc.

Code:

You see there is a chance that the two records INFO line and INFO AIR credit operation success may not match.
Because what i want to achieve is that the "recorded bonus (INFO line)" and the "credit operation success" are equal. 
if any thing is missing just put it into an error file.

4) What happens if a line record never gets used? That is, how long do you hold (store) un-used "line records" before they become errors (if they ever do)?

Code:

This is why i am creating this script to see that both operations that is recording of bonuses and posting them are consistant. 
Any descripencies could be seen in the error file.

@snowline84

The lines we need to check are:

INFO line
and
INFO AIR credit operation success

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
perl script - adding restart logic	zedex	Shell Programming and Scripting	4	07-08-2008 10:11 AM
building a SET clause in shell script	shalua	Shell Programming and Scripting	14	04-10-2007 03:34 PM
Help required on building the logic	igandu	Shell Programming and Scripting	2	01-21-2007 02:55 PM
Query on a shell script logic	tipsy	Shell Programming and Scripting	1	06-29-2006 02:38 PM
what the logic	ramneek	IP Networking	2	09-05-2005 04:42 AM

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode