Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Search patterns in multiple logs parallelly.

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Search patterns in multiple logs parallelly.
# 1  
Old 01-21-2014
Search patterns in multiple logs parallelly.
Hi All,

I am starting a service which will redirect its out put into 2 logs say A and B.
Now for succesful startup of the service i need to search pattern1 in log A and pattern2 in log B which are writen continuosly.

Now my requirement is to find the patterns in the increasing logs A and B parallely.

Need an advice on above requirement
# 2  
Old 01-21-2014
To search two logs simultaneously, run two programs.

What have you tried?
# 3  
Old 01-21-2014
I tried to get the last lines of each log and search for the pattern using sed inside while condition.

Below is the code.
server_lno --> last line no of server log
jboss_lno --> last line of jboss log

Code:
                while [ `$SED -n "$server_lno,$ p" $SERVER_LOG | $GREP -E "Shutdown.*complete" | wc -l` -eq 0 -o `$SED -n "$jboss_lno,$ p" $JBOSS_LOG | $GREP -E "shutdown.*finished" | wc -l` -eq 0 ]
                do
                        sleep 10
                        count=`expr $count + 1`
                        if [ $count -eq $max_count ]; then
                                echo "Error"
                                exit 1
                        fi
                done

Here i am having an itteration of required condition is not met.
Itteration will have a tollerance limit of on max_count

I would like to know whether is there any way to check the in a single script only.
# 4  
Old 01-21-2014
What do you mean by 'single script only'? Do you mean not looping, waiting for the log instead?

I'm sure this can be simplified a lot, please show us the relevant parts of your logfiles.
# 5  
Old 01-21-2014
Below is the sample output of server logs ::


Code:
2013-12-11 11:14:22,893 INFO  [org.jboss.jms.server.destination.QueueService] (JBoss Shutdown Hook) Queue[/queue/CarnotSystemQueue] stopped
2013-12-11 11:14:22,900 INFO  [org.jboss.jms.server.destination.QueueService] (JBoss Shutdown Hook) Queue[/queue/CarnotApplicationQueue] stopped
2013-12-11 11:14:22,905 INFO  [org.jboss.jms.server.destination.QueueService] (JBoss Shutdown Hook) Queue[/queue/CarnotDaemonQueue] stopped
2013-12-11 11:14:22,912 INFO  [org.jboss.jms.server.destination.QueueService] (JBoss Shutdown Hook) Queue[/queue/DLQ] stopped
2013-12-11 11:14:22,918 INFO  [org.jboss.jms.server.destination.QueueService] (JBoss Shutdown Hook) Queue[/queue/ExpiryQueue] stopped
2013-12-11 11:14:22,926 INFO  [org.jboss.jms.server.connectionfactory.ConnectionFactory] (JBoss Shutdown Hook) org.jboss.jms.server.connectionfactory.ConnectionFactory@14079543 undeployed
2013-12-11 11:14:22,932 INFO  [org.jboss.jms.server.connectionfactory.ConnectionFactory] (JBoss Shutdown Hook) org.jboss.jms.server.connectionfactory.ConnectionFactory@285c5e36 undeployed
2013-12-11 11:14:22,940 INFO  [org.jboss.jms.server.connectionfactory.ConnectionFactory] (JBoss Shutdown Hook) org.jboss.jms.server.connectionfactory.ConnectionFactory@5f37fd4b undeployed
2013-12-11 11:14:23,357 INFO  [org.jboss.jms.server.ServerPeer] (JBoss Shutdown Hook) JMS ServerPeer[0] stopped
2013-12-11 11:14:23,378 INFO  [org.jboss.resource.connectionmanager.ConnectionFactoryBindingService] (JBoss Shutdown Hook) Unbound ConnectionManager 'jboss.jca:service=DataSourceBinding,name=Defaul
tDS' from JNDI name 'java:DefaultDS'
2013-12-11 11:14:23,707 INFO  [org.jboss.jdbc.HypersonicDatabase] (JBoss Shutdown Hook) Database standalone closed clean
2013-12-11 11:14:23,734 INFO  [org.jboss.resource.connectionmanager.ConnectionFactoryBindingService] (JBoss Shutdown Hook) Unbound ConnectionManager 'jboss.jca:service=ConnectionFactoryBinding,name
=CarnotConnectionFactory' from JNDI name 'java:CarnotConnectionFactory'
2013-12-11 11:14:23,785 INFO  [org.quartz.core.QuartzScheduler] (JBoss Shutdown Hook) Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutting down.
2013-12-11 11:14:23,785 INFO  [org.quartz.core.QuartzScheduler] (JBoss Shutdown Hook) Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED paused.
2013-12-11 11:14:23,862 INFO  [org.quartz.simpl.SimpleThreadPool] (JBoss Shutdown Hook) There are still 54 worker threads active. See javadoc runInThread(Runnable) for a possible explanation
2013-12-11 11:14:23,868 INFO  [org.quartz.core.QuartzScheduler] (JBoss Shutdown Hook) Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutdown complete.
2013-12-11 11:14:24,076 INFO  [org.jboss.web.tomcat.service.deployers.TomcatDeployment] (JBoss Shutdown Hook) undeploy, ctxPath=/invoker
2013-12-11 11:14:24,116 INFO  [org.jboss.web.tomcat.service.deployers.TomcatDeployment] (JBoss Shutdown Hook) undeploy, ctxPath=/web-console
2013-12-11 11:14:24,172 INFO  [org.apache.catalina.core.StandardService] (JBoss Shutdown Hook) Stopping service jboss.web
2013-12-11 11:14:24,302 INFO  [com.arjuna.ats.jbossatx.jta.TransactionManagerService] (JBoss Shutdown Hook) Stopping transaction recovery manager
2013-12-11 11:14:24,323 INFO  [com.arjuna.ats.jbossatx.jta.TransactionManagerService] (JBoss Shutdown Hook) Destroying TransactionManagerService
2013-12-11 11:14:26,434 INFO  [org.jboss.mail.MailService] (JBoss Shutdown Hook) Mail service 'java:/Mail' removed from JNDI
2013-12-11 11:14:43,760 INFO  [org.jboss.bootstrap.microcontainer.ServerImpl] (JBoss Shutdown Hook) Shutdown complete

Below is the sample output of jboss logs::


Code:
2014-01-06 07:13:57,074 [TaskController_28] INFO  service.TaskController - executeTask "Stop WorkflowModelViewManager" params={physicalNode=PN1, taskCode=Set PN Shutdown} with user system, timeout=
-1s
2014-01-06 07:13:57,112 [TaskController_28] INFO  service.TaskController - executeTask Stop WorkflowModelViewManager success = Service.ServiceTask[Stop WorkflowModelViewManagerv0, type: Shutdown, s
tartup: Automatic, ruleSequence: WorkflowModelViewManager-stop]
2014-01-06 07:13:57,155 [JBoss Shutdown Hook] INFO  impl.SessionFactoryImpl - closing
2014-01-06 07:13:57,172 [JBoss Shutdown Hook] INFO  applicationlog.ApplicationLogHelper - Stopping workers...
2014-01-06 07:13:57,173 [ApplicationLogWorker] INFO  applicationlog.ApplicationLogWorker - Told to stop, quitting...
2014-01-06 07:13:57,173 [JBoss Shutdown Hook] INFO  applicationlog.ApplicationLogHelper - All workers were stopped.
2014-01-06 07:13:57,173 [JBoss Shutdown Hook] INFO  service.AMHServerControl - AMH shutdown
2014-01-06 07:13:57,173 [JBoss Shutdown Hook] INFO  server.ServerControlManager - End stopping the server
2014-01-06 07:13:57,173 [JBoss Shutdown Hook] INFO  server.ServerControlManager - shutdown finished

We can use loops. But i want to check for patterns in both logs to happen in a single script that too simultaneously..

Moderator's Comments:
Mod Comment Code tags for code, please.
# 6  
Old 01-21-2014
Quote:
Originally Posted by Girish19
We can use loops. But i want to check for patterns in both logs to happen in a single script that too simultaneously.
I don't think you can guarantee both logs get checked at precisely the same time.

The rest of your statement did not cross the language barrier, could you try explaining another way?
# 7  
Old 01-21-2014
If checking logs is not possible means ok not a problem.

Is there any way to compare the date and time available in the log with a variable having date value say "dt=2014-01-06 07:00:57".

Since i want to find the lines only after 2014-01-06 07:00:57

Is there any easy way for this??
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Search Multiple patterns and display

Hi, I have scenario like below and need to search for multiple patterns Eg: Test Time Started= secs Time Ended = secc Green test Test Time Started= secs Time Ended = secc Green test Output: I need to display the text starting with Test and starting with Time... (2 Replies)
Discussion started by: weknowd
2 Replies

2. Shell Programming and Scripting

Search and replace multiple patterns in a particular column only - efficient script

Hi Bigshots, I have a pattern file with two columns. I have another data file. If column 1 in the pattern file appears as the 4th column in the data file, I need to replace it (4th column of data file) with column 2 of the pattern file. If the pattern is found in any other column, it should not... (6 Replies)
Discussion started by: ss112233
6 Replies

3. Shell Programming and Scripting

How to search multiple patterns and remove lines from a file?

Hi, I have a file content as below. Table : PAYR Displayed fields: 15 of 15 Fixed columns: 4 List width 0999... (4 Replies)
Discussion started by: shirdi
4 Replies

4. Shell Programming and Scripting

How to search Multiple patterns in unix

Hi, I tried to search multiple pattern using awk trans=1234 reason=LN MISMATCH rec=`awk '/$trans/ && /'"$reason"'/' file` whenevr i tried to run on command promt it is executing but when i tried to implment same logic in shell script,it is failing i.e $rec is empty ... (6 Replies)
Discussion started by: ns64110
6 Replies

5. Shell Programming and Scripting

awk: Multiple search patterns & print in an one liner

I would like to print result of multiple search pattern invoked from an one liner. The code looks like this but won't work gawk -F '{{if ($0 ~ /pattern1/) pat1=$1 && if ($0 ~ /pattern2/) pat2=$2} ; print pat1, pat2}' Can anybody help getting the right code? (10 Replies)
Discussion started by: sdf
10 Replies

6. Shell Programming and Scripting

search multiple patterns

I have two lists in a file that look like a b b a e f c d f e d c I would like a final list a b c d e f I've tried multiple grep and awk but can't get it to work (8 Replies)
Discussion started by: godzilla07
8 Replies

7. Shell Programming and Scripting

Search multiple patterns in multiple files

Hi, I have to write one script that has to search a list of numbers in certain zipped files. For eg. one file file1.txt contains the numbers. File1.txt contains 5,00,000 numbers and I have to search each number in zipped files(The number of zipped files are around 1000 each file is 5 MB) I have... (10 Replies)
Discussion started by: vsachan
10 Replies

8. Shell Programming and Scripting

Perl - How to search a text file with multiple patterns?

Good day, great gurus, I'm new to Perl, and programming in general. I'm trying to retrieve a column of data from my text file which spans a non-specific number of lines. So I did a regexp that will pick out the columns. However,my pattern would vary. I tried using a foreach loop unsuccessfully.... (2 Replies)
Discussion started by: Sp3ck
2 Replies

9. Shell Programming and Scripting

Perl: Match a line with multiple search patterns

Hi I'm not very good with the serach patterns and I'd need a sample how to find a line that has multiple patterns. Say I want to find a line that has "abd", "123" and "QWERTY" and there can be any characters or numbers between the serach patterns, I have a file that has thousands of lines and... (10 Replies)
Discussion started by: Juha
10 Replies

10. UNIX for Dummies Questions & Answers

How to parameterize multiple search patterns and generate a new file

I have one file: 123*100*abcd*10 123*101*abcd*-29*def 123*100*abcd*-10 123*102*abcd*-105*asd I would like to parameterize the search patterns in the following way so that the user could dynamically change the search pattern. *100* and *- (ie *minus) *102* and *- The output that is... (6 Replies)
Discussion started by: augustinep
6 Replies
Login or Register to Ask a Question