Shell Programming and Scripting

Hi

I have a file consists of the following similar lines (10 mb file)

2008-05-15 02:15:38,268 [DEBUG] [AQUEDUCT-WebContainer : 3] RMSConnectionFactory - Setting session state for connection.

2008-05-15 02:15:38,277 [DEBUG] [AQUEDUCT-WebContainer : 3] RMSConnectionFactory - Returning WS connection.

My task is to find out any missing second lines for corresponding first lines.

I have written a script using bash, perl and awk commands but it takes lot of time to complete it. is there a better way to do this task ?

here is my current script:
=====
#FILE contains a list of connection and return statements
FILE=summary_db_conn.txt
#DUP_FILE : Copying as another file so that i can do a loop to find out the missing return statements
DUP_FILE=summary_db_conn_dup.txt
#FINAL_FILE = file contains setting session records which don't have corresponding return statements
FINAL_FILE=final_not_matching_records.txt


# To see whether set connection statements have corresponding return connection statements

while read line; do

temp_output=`echo $line | grep "Setting session state for connection" | awk '{ print $1, $2, $4, $5, $6, $7 }' | perl -wn -e 'print "$1\n" if m{[\[](.*?)[\]]};'`
if [ "$temp_output" != "" ]; then
echo "$line :Processing line "
get_time=`echo $line | grep "Setting session state for connection" | awk '{ print $1,$2 }'`
#echo "get_time $get_time"
flag="0"
while read file_line; do
#echo "Match line : $file_line"

if [ "$flag" = "1" ]; then
temp_file_output1=`echo $file_line | grep "Returning WS connection" | awk '{ print $1, $2, $4, $5, $6, $7 }' | perl -wn -e 'print "$1\n" if m{[\[](.*?)[\]]};'`
if [ "$temp_file_output1" != "" ]; then
if [ "$temp_output" = "$temp_file_output1" ]; then
# echo "Found the matching return statement"
echo "$file_line"
echo "Break: Success"
break
fi
else
temp_file_output1=`echo $file_line | grep "Setting session state for connection" | awk '{ print $1, $2, $4, $5, $6, $7 }' | perl -wn -e 'print "$1\n" if m{[\[](.*?)[\]]};'`
if [ "$temp_file_output1" != "" ]; then
if [ "$temp_output" = "$temp_file_output1" ]; then
echo $line >> $FINAL_FILE
#echo "Searching for:$line"
echo "Sorry found another session start statement with the same thread $line"
echo "The new Line is:$file_line"
break
fi
fi
fi
fi
temp_file_output=`echo $file_line | grep "$get_time"`

if [ "$temp_file_output" != "" ]; then
#echo "Found line"
#echo "$file_line"
flag="1"
fi

done < $DUP_FILE
fi

done < $FILE

echo "Program Ended`date`" >> $FINAL_FILE
====

Please let me know is there a fast and better way of doing this task ....


Thanks

Using awk or Perl, read the "Setting state for connection" keys into an associative array and remove a key from the array when you see the corresponding Returning line. At end of file, any keys still left in the array will be without a pair. I don't suppose you expect to find Returning lines without an opening Setting line.

As a general observation, anything with grep | awk is a waste because awk can usually do anything grep can, and similarly, awk | perl looks like one or the other is redundant.

Era,

thanks for your reply. Initially i thought of using arrays but don't know how to proceed with actual programming. I tried but failed with arrays as I am not a strong shell programmer(Occasional user)

I know it's not good to ask a program but could you please show me the things you mentioned about.

thanks in advance for your help

Can you post a sample with three or four of the buggers to test with?

Here i should use the word between [ ] for comparision for example AQUEDUCT-WebContainer : 0 has set connection as well as return connection.

Here are the sample data file. This file sample contains all kinds of missing things


2008-05-15 02:15:04,057 [DEBUG] [AQUEDUCT-WebContainer : 0] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:04,067 [DEBUG] [AQUEDUCT-WebContainer : 0] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:04,230 [DEBUG] [AQUEDUCT-WebContainer : 1] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:04,239 [DEBUG] [AQUEDUCT-WebContainer : 1] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:04,359 [DEBUG] [AQUEDUCT-WebContainer : 0] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:04,367 [DEBUG] [AQUEDUCT-WebContainer : 0] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:04,513 [DEBUG] [AQUEDUCT-WebContainer : 1] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:04,582 [DEBUG] [AQUEDUCT-WebContainer : 3] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:04,582 [DEBUG] [AQUEDUCT-WebContainer : 1] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:04,590 [DEBUG] [AQUEDUCT-WebContainer : 3] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:04,675 [DEBUG] [AQUEDUCT-WebContainer : 1] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:04,681 [DEBUG] [AQUEDUCT-WebContainer : 1] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:04,731 [DEBUG] [AQUEDUCT-WebContainer : 3] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:04,740 [DEBUG] [AQUEDUCT-WebContainer : 3] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:05,148 [DEBUG] [AQUEDUCT-WebContainer : 1] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:05,155 [DEBUG] [AQUEDUCT-WebContainer : 1] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:05,293 [DEBUG] [AQUEDUCT-WebContainer : 3] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:05,587 [DEBUG] [AQUEDUCT-WebContainer : 1] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:05,596 [DEBUG] [AQUEDUCT-WebContainer : 1] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:05,726 [DEBUG] [AQUEDUCT-WebContainer : 3] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:05,732 [DEBUG] [AQUEDUCT-WebContainer : 3] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:05,839 [DEBUG] [AQUEDUCT-WebContainer : 1] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:05,849 [DEBUG] [AQUEDUCT-WebContainer : 1] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:05,896 [DEBUG] [AQUEDUCT-WebContainer : 3] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:05,905 [DEBUG] [AQUEDUCT-WebContainer : 3] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:05,989 [DEBUG] [AQUEDUCT-WebContainer : 0] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:06,057 [DEBUG] [AQUEDUCT-WebContainer : 0] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:06,127 [DEBUG] [AQUEDUCT-WebContainer : 2] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:06,178 [DEBUG] [AQUEDUCT-WebContainer : 2] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:06,186 [DEBUG] [AQUEDUCT-WebContainer : 2] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:06,323 [DEBUG] [AQUEDUCT-WebContainer : 2] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:06,332 [DEBUG] [AQUEDUCT-WebContainer : 2] RMSConnectionFactory - Returning WS connection.

Save this to a file and chmod +x.

Code:

#!/usr/bin/perl -n

if (m/(.*) \[DEBUG\] \[([^][]+)\] RMSConnectionFactory - Setting session/) {
    $open{$2} = $1;
    next;
}

if (m/(.*) \[DEBUG\] \[([^][]+)\] RMSConnectionFactory - Returning WS /) {
    if ($open{$2}) {
        delete $open{$2};
    }
    else {
        warn "Return without opening session: $_";
    }
}

END {
    for my $k (keys %open)
    {
        print "$open{$k}: $k\n";
    }
}

I tested by removing one of the Returning lines and it printed the corresponding Setting line (or rather, the time stamp and identifier), but that's obviously not very massive testing.

Era,

I am getting the following error

check_test.sh: line 3: syntax error near unexpected token `.*'
check_test.sh: line 3: `if (m/(.*) \[DEBUG\] \[([^][]+)\] RMSConnectionFactory - Setting session/) {'


while executing your script as follows
sh check_test.sh summary_db_conn.txt

thanks