Search patterns in multiple logs parallelly.


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Search patterns in multiple logs parallelly.
# 1  
Old 01-21-2014
Search patterns in multiple logs parallelly.

Hi All,

I am starting a service which will redirect its out put into 2 logs say A and B.
Now for succesful startup of the service i need to search pattern1 in log A and pattern2 in log B which are writen continuosly.

Now my requirement is to find the patterns in the increasing logs A and B parallely.

Need an advice on above requirement
# 2  
Old 01-21-2014
To search two logs simultaneously, run two programs.

What have you tried?
# 3  
Old 01-21-2014
I tried to get the last lines of each log and search for the pattern using sed inside while condition.

Below is the code.
server_lno --> last line no of server log
jboss_lno --> last line of jboss log

Code:
                while [ `$SED -n "$server_lno,$ p" $SERVER_LOG | $GREP -E "Shutdown.*complete" | wc -l` -eq 0 -o `$SED -n "$jboss_lno,$ p" $JBOSS_LOG | $GREP -E "shutdown.*finished" | wc -l` -eq 0 ]
                do
                        sleep 10
                        count=`expr $count + 1`
                        if [ $count -eq $max_count ]; then
                                echo "Error"
                                exit 1
                        fi
                done

Here i am having an itteration of required condition is not met.
Itteration will have a tollerance limit of on max_count

I would like to know whether is there any way to check the in a single script only.
# 4  
Old 01-21-2014
What do you mean by 'single script only'? Do you mean not looping, waiting for the log instead?

I'm sure this can be simplified a lot, please show us the relevant parts of your logfiles.
# 5  
Old 01-21-2014
Below is the sample output of server logs ::


Code:
2013-12-11 11:14:22,893 INFO  [org.jboss.jms.server.destination.QueueService] (JBoss Shutdown Hook) Queue[/queue/CarnotSystemQueue] stopped
2013-12-11 11:14:22,900 INFO  [org.jboss.jms.server.destination.QueueService] (JBoss Shutdown Hook) Queue[/queue/CarnotApplicationQueue] stopped
2013-12-11 11:14:22,905 INFO  [org.jboss.jms.server.destination.QueueService] (JBoss Shutdown Hook) Queue[/queue/CarnotDaemonQueue] stopped
2013-12-11 11:14:22,912 INFO  [org.jboss.jms.server.destination.QueueService] (JBoss Shutdown Hook) Queue[/queue/DLQ] stopped
2013-12-11 11:14:22,918 INFO  [org.jboss.jms.server.destination.QueueService] (JBoss Shutdown Hook) Queue[/queue/ExpiryQueue] stopped
2013-12-11 11:14:22,926 INFO  [org.jboss.jms.server.connectionfactory.ConnectionFactory] (JBoss Shutdown Hook) org.jboss.jms.server.connectionfactory.ConnectionFactory@14079543 undeployed
2013-12-11 11:14:22,932 INFO  [org.jboss.jms.server.connectionfactory.ConnectionFactory] (JBoss Shutdown Hook) org.jboss.jms.server.connectionfactory.ConnectionFactory@285c5e36 undeployed
2013-12-11 11:14:22,940 INFO  [org.jboss.jms.server.connectionfactory.ConnectionFactory] (JBoss Shutdown Hook) org.jboss.jms.server.connectionfactory.ConnectionFactory@5f37fd4b undeployed
2013-12-11 11:14:23,357 INFO  [org.jboss.jms.server.ServerPeer] (JBoss Shutdown Hook) JMS ServerPeer[0] stopped
2013-12-11 11:14:23,378 INFO  [org.jboss.resource.connectionmanager.ConnectionFactoryBindingService] (JBoss Shutdown Hook) Unbound ConnectionManager 'jboss.jca:service=DataSourceBinding,name=Defaul
tDS' from JNDI name 'java:DefaultDS'
2013-12-11 11:14:23,707 INFO  [org.jboss.jdbc.HypersonicDatabase] (JBoss Shutdown Hook) Database standalone closed clean
2013-12-11 11:14:23,734 INFO  [org.jboss.resource.connectionmanager.ConnectionFactoryBindingService] (JBoss Shutdown Hook) Unbound ConnectionManager 'jboss.jca:service=ConnectionFactoryBinding,name
=CarnotConnectionFactory' from JNDI name 'java:CarnotConnectionFactory'
2013-12-11 11:14:23,785 INFO  [org.quartz.core.QuartzScheduler] (JBoss Shutdown Hook) Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutting down.
2013-12-11 11:14:23,785 INFO  [org.quartz.core.QuartzScheduler] (JBoss Shutdown Hook) Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED paused.
2013-12-11 11:14:23,862 INFO  [org.quartz.simpl.SimpleThreadPool] (JBoss Shutdown Hook) There are still 54 worker threads active. See javadoc runInThread(Runnable) for a possible explanation
2013-12-11 11:14:23,868 INFO  [org.quartz.core.QuartzScheduler] (JBoss Shutdown Hook) Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutdown complete.
2013-12-11 11:14:24,076 INFO  [org.jboss.web.tomcat.service.deployers.TomcatDeployment] (JBoss Shutdown Hook) undeploy, ctxPath=/invoker
2013-12-11 11:14:24,116 INFO  [org.jboss.web.tomcat.service.deployers.TomcatDeployment] (JBoss Shutdown Hook) undeploy, ctxPath=/web-console
2013-12-11 11:14:24,172 INFO  [org.apache.catalina.core.StandardService] (JBoss Shutdown Hook) Stopping service jboss.web
2013-12-11 11:14:24,302 INFO  [com.arjuna.ats.jbossatx.jta.TransactionManagerService] (JBoss Shutdown Hook) Stopping transaction recovery manager
2013-12-11 11:14:24,323 INFO  [com.arjuna.ats.jbossatx.jta.TransactionManagerService] (JBoss Shutdown Hook) Destroying TransactionManagerService
2013-12-11 11:14:26,434 INFO  [org.jboss.mail.MailService] (JBoss Shutdown Hook) Mail service 'java:/Mail' removed from JNDI
2013-12-11 11:14:43,760 INFO  [org.jboss.bootstrap.microcontainer.ServerImpl] (JBoss Shutdown Hook) Shutdown complete

Below is the sample output of jboss logs::


Code:
2014-01-06 07:13:57,074 [TaskController_28] INFO  service.TaskController - executeTask "Stop WorkflowModelViewManager" params={physicalNode=PN1, taskCode=Set PN Shutdown} with user system, timeout=
-1s
2014-01-06 07:13:57,112 [TaskController_28] INFO  service.TaskController - executeTask Stop WorkflowModelViewManager success = Service.ServiceTask[Stop WorkflowModelViewManagerv0, type: Shutdown, s
tartup: Automatic, ruleSequence: WorkflowModelViewManager-stop]
2014-01-06 07:13:57,155 [JBoss Shutdown Hook] INFO  impl.SessionFactoryImpl - closing
2014-01-06 07:13:57,172 [JBoss Shutdown Hook] INFO  applicationlog.ApplicationLogHelper - Stopping workers...
2014-01-06 07:13:57,173 [ApplicationLogWorker] INFO  applicationlog.ApplicationLogWorker - Told to stop, quitting...
2014-01-06 07:13:57,173 [JBoss Shutdown Hook] INFO  applicationlog.ApplicationLogHelper - All workers were stopped.
2014-01-06 07:13:57,173 [JBoss Shutdown Hook] INFO  service.AMHServerControl - AMH shutdown
2014-01-06 07:13:57,173 [JBoss Shutdown Hook] INFO  server.ServerControlManager - End stopping the server
2014-01-06 07:13:57,173 [JBoss Shutdown Hook] INFO  server.ServerControlManager - shutdown finished

We can use loops. But i want to check for patterns in both logs to happen in a single script that too simultaneously..

Moderator's Comments:
Mod Comment Code tags for code, please.
# 6  
Old 01-21-2014
Quote:
Originally Posted by Girish19
We can use loops. But i want to check for patterns in both logs to happen in a single script that too simultaneously.
I don't think you can guarantee both logs get checked at precisely the same time.

The rest of your statement did not cross the language barrier, could you try explaining another way?
# 7  
Old 01-21-2014
If checking logs is not possible means ok not a problem.

Is there any way to compare the date and time available in the log with a variable having date value say "dt=2014-01-06 07:00:57".

Since i want to find the lines only after 2014-01-06 07:00:57

Is there any easy way for this??
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Search Multiple patterns and display

Hi, I have scenario like below and need to search for multiple patterns Eg: Test Time Started= secs Time Ended = secc Green test Test Time Started= secs Time Ended = secc Green test Output: I need to display the text starting with Test and starting with Time... (2 Replies)
Discussion started by: weknowd
2 Replies

2. Shell Programming and Scripting

Search and replace multiple patterns in a particular column only - efficient script

Hi Bigshots, I have a pattern file with two columns. I have another data file. If column 1 in the pattern file appears as the 4th column in the data file, I need to replace it (4th column of data file) with column 2 of the pattern file. If the pattern is found in any other column, it should not... (6 Replies)
Discussion started by: ss112233
6 Replies

3. Shell Programming and Scripting

How to search multiple patterns and remove lines from a file?

Hi, I have a file content as below. Table : PAYR Displayed fields: 15 of 15 Fixed columns: 4 List width 0999... (4 Replies)
Discussion started by: shirdi
4 Replies

4. Shell Programming and Scripting

How to search Multiple patterns in unix

Hi, I tried to search multiple pattern using awk trans=1234 reason=LN MISMATCH rec=`awk '/$trans/ && /'"$reason"'/' file` whenevr i tried to run on command promt it is executing but when i tried to implment same logic in shell script,it is failing i.e $rec is empty ... (6 Replies)
Discussion started by: ns64110
6 Replies

5. Shell Programming and Scripting

awk: Multiple search patterns & print in an one liner

I would like to print result of multiple search pattern invoked from an one liner. The code looks like this but won't work gawk -F '{{if ($0 ~ /pattern1/) pat1=$1 && if ($0 ~ /pattern2/) pat2=$2} ; print pat1, pat2}' Can anybody help getting the right code? (10 Replies)
Discussion started by: sdf
10 Replies

6. Shell Programming and Scripting

search multiple patterns

I have two lists in a file that look like a b b a e f c d f e d c I would like a final list a b c d e f I've tried multiple grep and awk but can't get it to work (8 Replies)
Discussion started by: godzilla07
8 Replies

7. Shell Programming and Scripting

Search multiple patterns in multiple files

Hi, I have to write one script that has to search a list of numbers in certain zipped files. For eg. one file file1.txt contains the numbers. File1.txt contains 5,00,000 numbers and I have to search each number in zipped files(The number of zipped files are around 1000 each file is 5 MB) I have... (10 Replies)
Discussion started by: vsachan
10 Replies

8. Shell Programming and Scripting

Perl - How to search a text file with multiple patterns?

Good day, great gurus, I'm new to Perl, and programming in general. I'm trying to retrieve a column of data from my text file which spans a non-specific number of lines. So I did a regexp that will pick out the columns. However,my pattern would vary. I tried using a foreach loop unsuccessfully.... (2 Replies)
Discussion started by: Sp3ck
2 Replies

9. Shell Programming and Scripting

Perl: Match a line with multiple search patterns

Hi I'm not very good with the serach patterns and I'd need a sample how to find a line that has multiple patterns. Say I want to find a line that has "abd", "123" and "QWERTY" and there can be any characters or numbers between the serach patterns, I have a file that has thousands of lines and... (10 Replies)
Discussion started by: Juha
10 Replies

10. UNIX for Dummies Questions & Answers

How to parameterize multiple search patterns and generate a new file

I have one file: 123*100*abcd*10 123*101*abcd*-29*def 123*100*abcd*-10 123*102*abcd*-105*asd I would like to parameterize the search patterns in the following way so that the user could dynamically change the search pattern. *100* and *- (ie *minus) *102* and *- The output that is... (6 Replies)
Discussion started by: augustinep
6 Replies
Login or Register to Ask a Question