Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

Using grep for finding StackTraces

Shell Programming and Scripting


Reply    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 03-13-2018   -   Original Discussion by mpmichael
mpmichael's Unix or Linux Image
mpmichael mpmichael is offline
Registered User
 
Join Date: Feb 2018
Last Activity: 14 March 2018, 7:03 AM EDT
Posts: 10
Thanks: 6
Thanked 0 Times in 0 Posts
Using grep for finding StackTraces

Hello,

I've a horrible problem using grep. I want to "grep" stacktraces from a Logfile. Caused by performance i don't want to read the files line-by-line.

I allready used a regex-online-tester (succesful) but und the commandline I only get the first line - not the full stack-trace.

My command is:



Code:
 cat sample.log | grep -o -P "(\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d\.\d\d\d ERROR[\s\S]*?)(\d\d\d\d-\d\d-\d\d |\z)"


My Test Data is:



Code:
2018-02-14 09:14:20.050 ERROR 21 --- [batchAsyncTaskExecutor-2] de.mi.test.agent.core.IssueService        : error createExploreIssue:
org.springframework.web.client.HttpClientErrorException: 400 Bad Request
	at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:78)
	at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:700)
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:653)
	at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:613)
	at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:559)
	at sun.reflect.GeneratedMethodAccessor1561.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSite.invoke(PojoMetaMethodSite.java:192)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
	at de.mi.test.agent.core.GitLabRestClient.commitFile(GitLabRestClient.groovy:33)
	at de.mi.test.agent.core.GitLabRestClient$commitFile.call(Unknown Source)
	at de.mi.test.agent.core.IssueService.createIssue(IssueService.groovy:55)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:174)
	at de.mi.test.agent.core.IssueService.createExploreIssue(IssueService.groovy:38)
	at de.mi.test.agent.core.IssueService$createExploreIssue.call(Unknown Source)
	at de.mi.test.agent.core.ExploreExceptionSkipListener.onSkipInProcess(ExploreExceptionSkipListener.groovy:34)
	at de.mi.test.agent.core.ExploreExceptionSkipListener.onSkipInProcess(ExploreExceptionSkipListener.groovy)
	at de.mi.test.agent.core.ExploreExceptionSkipListener$$FastClassBySpringCGLIB$$42ab6582.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
	at de.mi.test.agent.core.ExploreExceptionSkipListener$$EnhancerBySpringCGLIB$$97e4579e.onSkipInProcess(<generated>)
	at org.springframework.batch.core.listener.CompositeSkipListener.onSkipInProcess(CompositeSkipListener.java:87)
	at org.springframework.batch.core.listener.MulticasterBatchListener.onSkipInProcess(MulticasterBatchListener.java:338)
	at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.callProcessSkipListener(FaultTolerantChunkProcessor.java:478)
	at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.callSkipListeners(FaultTolerantChunkProcessor.java:450)
	at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.write(FaultTolerantChunkProcessor.java:438)
	at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:199)
	at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:75)
	at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:406)
	at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:330)
	at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
	at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:272)
	at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:81)
	at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:374)
	at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215)
	at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:144)
	at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:257)
	at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:200)
	at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148)
	at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:64)
	at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67)
	at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169)
	at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144)
	at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:134)
	at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:306)
	at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:135)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

2018-02-14 09:14:24.495  INFO 21 --- [batchAsyncTaskExecutor-2] d.m.m.a.portal.portalExploreService  : -> process [47722,1 (928456)]
2018-02-14 09:14:24.626  INFO 21 --- [batchAsyncTaskExecutor-1] d.m.m.a.portal.portalExploreService  : -> process [48903 (363675)]
2018-02-14 09:14:25.031  INFO 21 --- [batchAsyncTaskExecutor-2] io.github.bonigarcia.wdm.BrowserManager  : Exporting webdriver.gecko.driver as /root/.m2/repository/webdriver/geckodriver/linux64/0.17.0/geckodriver
2018-02-14 09:14:25.086  INFO 21 --- [batchAsyncTaskExecutor-1] io.github.bonigarcia.wdm.BrowserManager  : Exporting webdriver.gecko.driver as /root/.m2/repository/webdriver/geckodriver/linux64/0.17.0/geckodriver
2018-02-14 09:14:28.181  INFO 21 --- [Forwarding newSession on session null to remote] o.o.selenium.remote.Proto

My result is:



Code:
2018-02-14 09:14:20.050 ERROR 21 --- [batchAsyncTaskExecutor-2] de.mi.test.agent.core.IssueService        : error createExploreIssue:

Expected would be:



Code:
2018-02-14 09:14:20.050 ERROR 21 --- [batchAsyncTaskExecutor-2] de.mi.test.agent.core.IssueService        : error createExploreIssue:
org.springframework.web.client.HttpClientErrorException: 400 Bad Request
	at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:78)
	at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:700)
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:653)
	at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:613)
	at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:559)
	at sun.reflect.GeneratedMethodAccessor1561.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSite.invoke(PojoMetaMethodSite.java:192)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
	at de.mi.test.agent.core.GitLabRestClient.commitFile(GitLabRestClient.groovy:33)
	at de.mi.test.agent.core.GitLabRestClient$commitFile.call(Unknown Source)
	at de.mi.test.agent.core.IssueService.createIssue(IssueService.groovy:55)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:174)
	at de.mi.test.agent.core.IssueService.createExploreIssue(IssueService.groovy:38)
	at de.mi.test.agent.core.IssueService$createExploreIssue.call(Unknown Source)
	at de.mi.test.agent.core.ExploreExceptionSkipListener.onSkipInProcess(ExploreExceptionSkipListener.groovy:34)
	at de.mi.test.agent.core.ExploreExceptionSkipListener.onSkipInProcess(ExploreExceptionSkipListener.groovy)
	at de.mi.test.agent.core.ExploreExceptionSkipListener$$FastClassBySpringCGLIB$$42ab6582.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
	at de.mi.test.agent.core.ExploreExceptionSkipListener$$EnhancerBySpringCGLIB$$97e4579e.onSkipInProcess(<generated>)
	at org.springframework.batch.core.listener.CompositeSkipListener.onSkipInProcess(CompositeSkipListener.java:87)
	at org.springframework.batch.core.listener.MulticasterBatchListener.onSkipInProcess(MulticasterBatchListener.java:338)
	at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.callProcessSkipListener(FaultTolerantChunkProcessor.java:478)
	at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.callSkipListeners(FaultTolerantChunkProcessor.java:450)
	at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.write(FaultTolerantChunkProcessor.java:438)
	at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:199)
	at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:75)
	at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:406)
	at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:330)
	at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
	at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:272)
	at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:81)
	at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:374)
	at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215)
	at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:144)
	at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:257)
	at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:200)
	at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148)
	at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:64)
	at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67)
	at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169)
	at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144)
	at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:134)
	at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:306)
	at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:135)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Thank you for any idea or tip.
Best regards
Mpmichael
Sponsored Links
    #2  
Old Unix and Linux 03-13-2018   -   Original Discussion by mpmichael
vgersh99's Unix or Linux Image
vgersh99 vgersh99 is offline Forum Advisor  
Forum Advisor
 
Join Date: Feb 2005
Last Activity: 23 April 2018, 6:51 AM EDT
Location: Foxborough, MA
Posts: 8,256
Thanks: 241
Thanked 813 Times in 759 Posts
how about awk:


Code:
awk '/ERROR/' RS= sample.log

The Following User Says Thank You to vgersh99 For This Useful Post:
mpmichael (03-13-2018)
Sponsored Links
    #3  
Old Unix and Linux 03-13-2018   -   Original Discussion by mpmichael
mpmichael's Unix or Linux Image
mpmichael mpmichael is offline
Registered User
 
Join Date: Feb 2018
Last Activity: 14 March 2018, 7:03 AM EDT
Posts: 10
Thanks: 6
Thanked 0 Times in 0 Posts
Hello,

thank you very much for this amazing tip! It worked for the sample file. Now I tried on real logfiles and the result where that it was delivered to much results. If the Word "error" is in the text the line is also delivered. But there should only be lines where "ERROR" is behind the time.
BR
mpmichael
    #4  
Old Unix and Linux 03-13-2018   -   Original Discussion by mpmichael
vgersh99's Unix or Linux Image
vgersh99 vgersh99 is offline Forum Advisor  
Forum Advisor
 
Join Date: Feb 2005
Last Activity: 23 April 2018, 6:51 AM EDT
Location: Foxborough, MA
Posts: 8,256
Thanks: 241
Thanked 813 Times in 759 Posts
could you attach a small sample of the the file including the line(s) that break.
Sponsored Links
    #5  
Old Unix and Linux 03-13-2018   -   Original Discussion by mpmichael
mpmichael's Unix or Linux Image
mpmichael mpmichael is offline
Registered User
 
Join Date: Feb 2018
Last Activity: 14 March 2018, 7:03 AM EDT
Posts: 10
Thanks: 6
Thanked 0 Times in 0 Posts
thank you for the help. I attached the logfile.

The Command I used was:



Code:
awk '/ERROR/' RS= testlog2.txt

and the (false) Result where i. E. lines 741 - 777

BR
mpmichael
Attached Files
File Type: txt testlog2.txt (72.9 KB, 8 views)
Sponsored Links
    #6  
Old Unix and Linux 03-13-2018   -   Original Discussion by mpmichael
RudiC's Unix or Linux Image
RudiC RudiC is offline Forum Staff  
Moderator
 
Join Date: Jul 2012
Last Activity: 23 April 2018, 6:48 AM EDT
Location: Aachen, Germany
Posts: 12,509
Thanks: 401
Thanked 3,878 Times in 3,565 Posts
How about


Code:
sed -n '/ERROR/p; //,/^[0-9]\{4\}-/ {/^[0-9]\{4\}-/!p}' /tmp/testlog2.txt

Tested on Ubuntu linux 17.10
The Following User Says Thank You to RudiC For This Useful Post:
mpmichael (03-14-2018)
Sponsored Links
    #7  
Old Unix and Linux 03-14-2018   -   Original Discussion by mpmichael
mpmichael's Unix or Linux Image
mpmichael mpmichael is offline
Registered User
 
Join Date: Feb 2018
Last Activity: 14 March 2018, 7:03 AM EDT
Posts: 10
Thanks: 6
Thanked 0 Times in 0 Posts
Hello RudiC,

many thank's it works as desired!
Great help!
BR
mpmichael
Sponsored Links
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
finding keywords in many files using grep raghulrajan UNIX for Dummies Questions & Answers 5 03-14-2012 11:05 AM
finding pattern without grep in unix feint Homework & Coursework Questions 2 05-01-2011 04:08 AM
Finding a pattern with grep moyzZ UNIX for Dummies Questions & Answers 7 03-17-2011 12:51 AM
finding the line number from a grep ? hcclnoodles Shell Programming and Scripting 7 03-03-2009 05:49 AM
Finding number ranges using grep DKNUCKLES Shell Programming and Scripting 2 11-07-2007 01:30 AM



All times are GMT -4. The time now is 07:54 AM.