Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Search patterns in multiple logs parallelly.

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Search patterns in multiple logs parallelly.
# 8  
Old 01-21-2014
Quote:
Originally Posted by Girish19
Is there any way to compare the date and time available in the log with a variable having date value say "dt=2014-01-06 07:00:57".
I see what you mean now.

Yes, that should be straightforward actually. YYYY-MM-DD HH:MM:SS,SSS timestamps are very convenient, you can compare entire lines with < > =. Later times are guaranteed to compare > than earlier times.

You can rewrite that huge pipe chain with the awk utility, including the time comparison.

Code:
TIME="2014-01-06 07:00:57,000"

# If the line is later than T, and contains "shutdown complete", quit with success, else quit with error.
# Then the same on the second file, chained with && to check for the success of both.
if awk -v T="$TIME" '($0 >= T) && /Shutdown.*complete/ { SC=1 } END { EXIT !SC }' $SERVER_LOG &&
        awk -v T="$TIME" '($0 >= T) && /Shutdown.*complete/ { SC=1 } END { EXIT !SC }' $JBOSS_LOG
then
        echo "Shutdown Complete found in both logs after $TIME"
else
        echo "Shutdown Complete missing from one or both"
fi

Last edited by Corona688; 01-21-2014 at 01:39 PM..
# 9  
Old 01-21-2014
That may have been too complex an example. You can filter all the lines greater than a date in your logs like this:

Code:
TIME="2014-01-06 07:00:57,000"

# Print all lines at or later than TIME from filename
awk -v T="$TIME" '$0 >= T' filename

grep or sed do not have these forms of comparison.

This only works because your logs begin with YYYY-MM-DD HH:MM:SS.SSS, which means they sort and compare alphabetically (which is why YYYY-MM-DD dates are standard -- they're just so handy.)
# 10  
Old 01-21-2014
Couldn't you convert TIME to epoch?

Code:
TIME="2014-01-06 07:00:57,000"
EPOCH=`date --date="${TIME}" +%s`

The above didn't work on my FreeBSD server as I don't have gnu date installed, I don't think getting to the same point would be too difficult (or just install gnu date and use that). probably more work than its worth though, if you're just trying to retrieve out of a log.
# 11  
Old 01-21-2014
I am sorry i was not able to view the output

---------- Post updated at 11:39 PM ---------- Previous update was at 11:37 PM ----------

$ EPOCH=`date --date="${TIME}" +%s`
date: illegal option -- date=2014-01-06 07:00:57,000
usage: date [-u] mmddHHMM[[cc]yy][.SS]
date [-u] [+format]
date -a [-]sss[.fff]
# 12  
Old 01-21-2014
Have you tried any of the solutions I suggested yet?

To repeat:

Code:
TIME="2014-01-06 07:00:57,000"

# Print all lines at or later than TIME from filename
awk -v T="$TIME" '$0 >= T' filename

# 13  
Old 01-21-2014
Or else can you please let me know how to compare 2 dates say
dt1=2014-01-06 06:59:57 and dt2=2014-01-06 07:00:57 and know which one is greater

---------- Post updated at 11:45 PM ---------- Previous update was at 11:44 PM ----------

@corona I have tried your solution but output was not displayed
# 14  
Old 01-21-2014
Thanks for letting me know. But it works perfectly with the data you posted:

Code:
$ cat log

2013-12-11 11:14:22,893 INFO  [org.jboss.jms.server.destination.QueueService] (JBoss Shutdown Hook) Queue[/queue/CarnotSystemQueue] stopped
2013-12-11 11:14:22,900 INFO  [org.jboss.jms.server.destination.QueueService] (JBoss Shutdown Hook) Queue[/queue/CarnotApplicationQueue] stopped
2013-12-11 11:14:22,905 INFO  [org.jboss.jms.server.destination.QueueService] (JBoss Shutdown Hook) Queue[/queue/CarnotDaemonQueue] stopped
2013-12-11 11:14:22,912 INFO  [org.jboss.jms.server.destination.QueueService] (JBoss Shutdown Hook) Queue[/queue/DLQ] stopped
2013-12-11 11:14:22,918 INFO  [org.jboss.jms.server.destination.QueueService] (JBoss Shutdown Hook) Queue[/queue/ExpiryQueue] stopped
2013-12-11 11:14:22,926 INFO  [org.jboss.jms.server.connectionfactory.ConnectionFactory] (JBoss Shutdown Hook) org.jboss.jms.server.connectionfactory.ConnectionFactory@14079543 undeployed
2013-12-11 11:14:22,932 INFO  [org.jboss.jms.server.connectionfactory.ConnectionFactory] (JBoss Shutdown Hook) org.jboss.jms.server.connectionfactory.ConnectionFactory@285c5e36 undeployed
2013-12-11 11:14:22,940 INFO  [org.jboss.jms.server.connectionfactory.ConnectionFactory] (JBoss Shutdown Hook) org.jboss.jms.server.connectionfactory.ConnectionFactory@5f37fd4b undeployed
2013-12-11 11:14:23,357 INFO  [org.jboss.jms.server.ServerPeer] (JBoss Shutdown Hook) JMS ServerPeer[0] stopped
2013-12-11 11:14:23,378 INFO  [org.jboss.resource.connectionmanager.ConnectionFactoryBindingService] (JBoss Shutdown Hook) Unbound ConnectionManager 'jboss.jca:service=DataSourceBinding,name=DefaultDS' from JNDI name 'java:DefaultDS'
2013-12-11 11:14:23,707 INFO  [org.jboss.jdbc.HypersonicDatabase] (JBoss Shutdown Hook) Database standalone closed clean
2013-12-11 11:14:23,734 INFO  [org.jboss.resource.connectionmanager.ConnectionFactoryBindingService] (JBoss Shutdown Hook) Unbound ConnectionManager 'jboss.jca:service=ConnectionFactoryBinding,name=CarnotConnectionFactory' from JNDI name 'java:CarnotConnectionFactory'
2013-12-11 11:14:23,785 INFO  [org.quartz.core.QuartzScheduler] (JBoss Shutdown Hook) Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutting down.
2013-12-11 11:14:23,785 INFO  [org.quartz.core.QuartzScheduler] (JBoss Shutdown Hook) Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED paused.
2013-12-11 11:14:23,862 INFO  [org.quartz.simpl.SimpleThreadPool] (JBoss Shutdown Hook) There are still 54 worker threads active. See javadoc runInThread(Runnable) for a possible explanation
2013-12-11 11:14:23,868 INFO  [org.quartz.core.QuartzScheduler] (JBoss Shutdown Hook) Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutdown complete.
2013-12-11 11:14:24,076 INFO  [org.jboss.web.tomcat.service.deployers.TomcatDeployment] (JBoss Shutdown Hook) undeploy, ctxPath=/invoker
2013-12-11 11:14:24,116 INFO  [org.jboss.web.tomcat.service.deployers.TomcatDeployment] (JBoss Shutdown Hook) undeploy, ctxPath=/web-console
2013-12-11 11:14:24,172 INFO  [org.apache.catalina.core.StandardService] (JBoss Shutdown Hook) Stopping service jboss.web
2013-12-11 11:14:24,302 INFO  [com.arjuna.ats.jbossatx.jta.TransactionManagerService] (JBoss Shutdown Hook) Stopping transaction recovery manager
2013-12-11 11:14:24,323 INFO  [com.arjuna.ats.jbossatx.jta.TransactionManagerService] (JBoss Shutdown Hook) Destroying TransactionManagerService
2013-12-11 11:14:26,434 INFO  [org.jboss.mail.MailService] (JBoss Shutdown Hook) Mail service 'java:/Mail' removed from JNDI
2013-12-11 11:14:43,760 INFO  [org.jboss.bootstrap.microcontainer.ServerImpl] (JBoss Shutdown Hook) Shutdown complete

$ TIME="2013-12-11 11:14:24,323"
$ awk -v T="$TIME" '$0 >= T' log

2013-12-11 11:14:24,323 INFO  [com.arjuna.ats.jbossatx.jta.TransactionManagerService] (JBoss Shutdown Hook) Destroying TransactionManagerService
2013-12-11 11:14:26,434 INFO  [org.jboss.mail.MailService] (JBoss Shutdown Hook) Mail service 'java:/Mail' removed from JNDI
2013-12-11 11:14:43,760 INFO  [org.jboss.bootstrap.microcontainer.ServerImpl] (JBoss Shutdown Hook) Shutdown complete

$

Please show exactly what you did when you tried my code -- word for word, letter for letter, keystroke for keystroke. If it doesn't work, probably nothing else will either until we figure out what's changed.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Search Multiple patterns and display

Hi, I have scenario like below and need to search for multiple patterns Eg: Test Time Started= secs Time Ended = secc Green test Test Time Started= secs Time Ended = secc Green test Output: I need to display the text starting with Test and starting with Time... (2 Replies)
Discussion started by: weknowd
2 Replies

2. Shell Programming and Scripting

Search and replace multiple patterns in a particular column only - efficient script

Hi Bigshots, I have a pattern file with two columns. I have another data file. If column 1 in the pattern file appears as the 4th column in the data file, I need to replace it (4th column of data file) with column 2 of the pattern file. If the pattern is found in any other column, it should not... (6 Replies)
Discussion started by: ss112233
6 Replies

3. Shell Programming and Scripting

How to search multiple patterns and remove lines from a file?

Hi, I have a file content as below. Table : PAYR Displayed fields: 15 of 15 Fixed columns: 4 List width 0999... (4 Replies)
Discussion started by: shirdi
4 Replies

4. Shell Programming and Scripting

How to search Multiple patterns in unix

Hi, I tried to search multiple pattern using awk trans=1234 reason=LN MISMATCH rec=`awk '/$trans/ && /'"$reason"'/' file` whenevr i tried to run on command promt it is executing but when i tried to implment same logic in shell script,it is failing i.e $rec is empty ... (6 Replies)
Discussion started by: ns64110
6 Replies

5. Shell Programming and Scripting

awk: Multiple search patterns & print in an one liner

I would like to print result of multiple search pattern invoked from an one liner. The code looks like this but won't work gawk -F '{{if ($0 ~ /pattern1/) pat1=$1 && if ($0 ~ /pattern2/) pat2=$2} ; print pat1, pat2}' Can anybody help getting the right code? (10 Replies)
Discussion started by: sdf
10 Replies

6. Shell Programming and Scripting

search multiple patterns

I have two lists in a file that look like a b b a e f c d f e d c I would like a final list a b c d e f I've tried multiple grep and awk but can't get it to work (8 Replies)
Discussion started by: godzilla07
8 Replies

7. Shell Programming and Scripting

Search multiple patterns in multiple files

Hi, I have to write one script that has to search a list of numbers in certain zipped files. For eg. one file file1.txt contains the numbers. File1.txt contains 5,00,000 numbers and I have to search each number in zipped files(The number of zipped files are around 1000 each file is 5 MB) I have... (10 Replies)
Discussion started by: vsachan
10 Replies

8. Shell Programming and Scripting

Perl - How to search a text file with multiple patterns?

Good day, great gurus, I'm new to Perl, and programming in general. I'm trying to retrieve a column of data from my text file which spans a non-specific number of lines. So I did a regexp that will pick out the columns. However,my pattern would vary. I tried using a foreach loop unsuccessfully.... (2 Replies)
Discussion started by: Sp3ck
2 Replies

9. Shell Programming and Scripting

Perl: Match a line with multiple search patterns

Hi I'm not very good with the serach patterns and I'd need a sample how to find a line that has multiple patterns. Say I want to find a line that has "abd", "123" and "QWERTY" and there can be any characters or numbers between the serach patterns, I have a file that has thousands of lines and... (10 Replies)
Discussion started by: Juha
10 Replies

10. UNIX for Dummies Questions & Answers

How to parameterize multiple search patterns and generate a new file

I have one file: 123*100*abcd*10 123*101*abcd*-29*def 123*100*abcd*-10 123*102*abcd*-105*asd I would like to parameterize the search patterns in the following way so that the user could dynamically change the search pattern. *100* and *- (ie *minus) *102* and *- The output that is... (6 Replies)
Discussion started by: augustinep
6 Replies
Login or Register to Ask a Question