The UNIX and Linux Forums  
Hello and Welcome from United States to the UNIX and Linux Forums! Thank You for Visiting and Joining Our Global Community.

Go Back   The UNIX and Linux Forums > Top Forums > Shell Programming and Scripting
Optimized way of doing the task in shell programming Optimized way of doing the task in shell programming
.
google unix.com

Forums Register Forum RulesLinksAlbums FAQ Members List Calendar Search Today's Posts Mark Forums Read


Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
shell programming nivas Shell Programming and Scripting 8 02-11-2008 12:06 PM
searchin optimized arunsubbhian UNIX for Dummies Questions & Answers 1 09-07-2007 04:31 AM
Optimized Method matrixmadhan UNIX for Dummies Questions & Answers 2 12-14-2006 10:26 AM
Server Optimized Linux computek Linux 3 09-18-2004 03:22 PM
Shell Programming JWK1 UNIX for Dummies Questions & Answers 2 06-01-2001 01:31 PM

Closed Thread
English Japanese Spanish French German Portuguese Italian Dutch Swedish Russian Norwegian Hungarian Hebrew Danish Powered by Powered by Google
 
LinkBack Thread Tools Search this Thread Rate Thread Display Modes
  #1 (permalink)  
Old 05-16-2008
pcjandyala pcjandyala is offline
Registered User
  
 

Join Date: May 2008
Posts: 12
Optimized way of doing the task in shell programming
Hi

I have a file consists of the following similar lines (10 mb file)

2008-05-15 02:15:38,268 [DEBUG] [AQUEDUCT-WebContainer : 3] RMSConnectionFactory - Setting session state for connection.

2008-05-15 02:15:38,277 [DEBUG] [AQUEDUCT-WebContainer : 3] RMSConnectionFactory - Returning WS connection.

My task is to find out any missing second lines for corresponding first lines.

I have written a script using bash, perl and awk commands but it takes lot of time to complete it. is there a better way to do this task ?

here is my current script:
=====
#FILE contains a list of connection and return statements
FILE=summary_db_conn.txt
#DUP_FILE : Copying as another file so that i can do a loop to find out the missing return statements
DUP_FILE=summary_db_conn_dup.txt
#FINAL_FILE = file contains setting session records which don't have corresponding return statements
FINAL_FILE=final_not_matching_records.txt


# To see whether set connection statements have corresponding return connection statements

while read line; do

temp_output=`echo $line | grep "Setting session state for connection" | awk '{ print $1, $2, $4, $5, $6, $7 }' | perl -wn -e 'print "$1\n" if m{[\[](.*?)[\]]};'`
if [ "$temp_output" != "" ]; then
echo "$line :Processing line "
get_time=`echo $line | grep "Setting session state for connection" | awk '{ print $1,$2 }'`
#echo "get_time $get_time"
flag="0"
while read file_line; do
#echo "Match line : $file_line"

if [ "$flag" = "1" ]; then
temp_file_output1=`echo $file_line | grep "Returning WS connection" | awk '{ print $1, $2, $4, $5, $6, $7 }' | perl -wn -e 'print "$1\n" if m{[\[](.*?)[\]]};'`
if [ "$temp_file_output1" != "" ]; then
if [ "$temp_output" = "$temp_file_output1" ]; then
# echo "Found the matching return statement"
echo "$file_line"
echo "Break: Success"
break
fi
else
temp_file_output1=`echo $file_line | grep "Setting session state for connection" | awk '{ print $1, $2, $4, $5, $6, $7 }' | perl -wn -e 'print "$1\n" if m{[\[](.*?)[\]]};'`
if [ "$temp_file_output1" != "" ]; then
if [ "$temp_output" = "$temp_file_output1" ]; then
echo $line >> $FINAL_FILE
#echo "Searching for:$line"
echo "Sorry found another session start statement with the same thread $line"
echo "The new Line is:$file_line"
break
fi
fi
fi
fi
temp_file_output=`echo $file_line | grep "$get_time"`

if [ "$temp_file_output" != "" ]; then
#echo "Found line"
#echo "$file_line"
flag="1"
fi

done < $DUP_FILE
fi

done < $FILE

echo "Program Ended`date`" >> $FINAL_FILE
====

Please let me know is there a fast and better way of doing this task ....


Thanks
  #2 (permalink)  
Old 05-16-2008
era era is offline Forum Advisor  
Herder of Useless Cats (On Sabbatical)
  
 

Join Date: Mar 2008
Location: /there/is/only/bin/sh
Posts: 3,652
Using awk or Perl, read the "Setting state for connection" keys into an associative array and remove a key from the array when you see the corresponding Returning line. At end of file, any keys still left in the array will be without a pair. I don't suppose you expect to find Returning lines without an opening Setting line.

As a general observation, anything with grep | awk is a waste because awk can usually do anything grep can, and similarly, awk | perl looks like one or the other is redundant.
  #3 (permalink)  
Old 05-16-2008
pcjandyala pcjandyala is offline
Registered User
  
 

Join Date: May 2008
Posts: 12
Thanks
Era,

thanks for your reply. Initially i thought of using arrays but don't know how to proceed with actual programming. I tried but failed with arrays as I am not a strong shell programmer(Occasional user)

I know it's not good to ask a program but could you please show me the things you mentioned about.

thanks in advance for your help
  #4 (permalink)  
Old 05-16-2008
era era is offline Forum Advisor  
Herder of Useless Cats (On Sabbatical)
  
 

Join Date: Mar 2008
Location: /there/is/only/bin/sh
Posts: 3,652
Can you post a sample with three or four of the buggers to test with?
  #5 (permalink)  
Old 05-16-2008
pcjandyala pcjandyala is offline
Registered User
  
 

Join Date: May 2008
Posts: 12
Sure
Here i should use the word between [ ] for comparision for example AQUEDUCT-WebContainer : 0 has set connection as well as return connection.

Here are the sample data file. This file sample contains all kinds of missing things


2008-05-15 02:15:04,057 [DEBUG] [AQUEDUCT-WebContainer : 0] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:04,067 [DEBUG] [AQUEDUCT-WebContainer : 0] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:04,230 [DEBUG] [AQUEDUCT-WebContainer : 1] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:04,239 [DEBUG] [AQUEDUCT-WebContainer : 1] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:04,359 [DEBUG] [AQUEDUCT-WebContainer : 0] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:04,367 [DEBUG] [AQUEDUCT-WebContainer : 0] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:04,513 [DEBUG] [AQUEDUCT-WebContainer : 1] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:04,582 [DEBUG] [AQUEDUCT-WebContainer : 3] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:04,582 [DEBUG] [AQUEDUCT-WebContainer : 1] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:04,590 [DEBUG] [AQUEDUCT-WebContainer : 3] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:04,675 [DEBUG] [AQUEDUCT-WebContainer : 1] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:04,681 [DEBUG] [AQUEDUCT-WebContainer : 1] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:04,731 [DEBUG] [AQUEDUCT-WebContainer : 3] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:04,740 [DEBUG] [AQUEDUCT-WebContainer : 3] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:05,148 [DEBUG] [AQUEDUCT-WebContainer : 1] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:05,155 [DEBUG] [AQUEDUCT-WebContainer : 1] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:05,293 [DEBUG] [AQUEDUCT-WebContainer : 3] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:05,587 [DEBUG] [AQUEDUCT-WebContainer : 1] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:05,596 [DEBUG] [AQUEDUCT-WebContainer : 1] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:05,726 [DEBUG] [AQUEDUCT-WebContainer : 3] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:05,732 [DEBUG] [AQUEDUCT-WebContainer : 3] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:05,839 [DEBUG] [AQUEDUCT-WebContainer : 1] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:05,849 [DEBUG] [AQUEDUCT-WebContainer : 1] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:05,896 [DEBUG] [AQUEDUCT-WebContainer : 3] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:05,905 [DEBUG] [AQUEDUCT-WebContainer : 3] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:05,989 [DEBUG] [AQUEDUCT-WebContainer : 0] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:06,057 [DEBUG] [AQUEDUCT-WebContainer : 0] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:06,127 [DEBUG] [AQUEDUCT-WebContainer : 2] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:06,178 [DEBUG] [AQUEDUCT-WebContainer : 2] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:06,186 [DEBUG] [AQUEDUCT-WebContainer : 2] RMSConnectionFactory - Returning WS connection.
2008-05-15 02:15:06,323 [DEBUG] [AQUEDUCT-WebContainer : 2] RMSConnectionFactory - Setting session state for connection.
2008-05-15 02:15:06,332 [DEBUG] [AQUEDUCT-WebContainer : 2] RMSConnectionFactory - Returning WS connection.
  #6 (permalink)  
Old 05-16-2008
era era is offline Forum Advisor  
Herder of Useless Cats (On Sabbatical)
  
 

Join Date: Mar 2008
Location: /there/is/only/bin/sh
Posts: 3,652
Save this to a file and chmod +x.

Code:
#!/usr/bin/perl -n

if (m/(.*) \[DEBUG\] \[([^][]+)\] RMSConnectionFactory - Setting session/) {
    $open{$2} = $1;
    next;
}

if (m/(.*) \[DEBUG\] \[([^][]+)\] RMSConnectionFactory - Returning WS /) {
    if ($open{$2}) {
        delete $open{$2};
    }
    else {
        warn "Return without opening session: $_";
    }
}

END {
    for my $k (keys %open)
    {
        print "$open{$k}: $k\n";
    }
}
I tested by removing one of the Returning lines and it printed the corresponding Setting line (or rather, the time stamp and identifier), but that's obviously not very massive testing.
Closed Thread

Bookmarks

« select a record from one file matching from second file using awk | Iteration through the results from a unix command »
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes Rate This Thread
Hybrid Mode Hybrid Mode
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT -4. The time now is 02:59 PM.

The UNIX and Linux Forums - Learn UNIX and UNIX Commands RSS Feeds - Contact Us - The UNIX and Linux Forums - Learn UNIX and UNIX Commands - Archive - Top

Powered by: vBulletin, Copyright ©2000 - 2006, Jelsoft Enterprises Limited. Language Translations Powered by .
vBCredits v1.4 Copyright ©2007 - 2008, PixelFX Studios
The UNIX and Linux Forums Content Copyright ©1993-2009. All Rights Reserved.Ad Management by RedTyger

Content Relevant URLs by vBSEO 3.2.0