Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Using grep for finding StackTraces

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Using grep for finding StackTraces
# 1  
Old 03-13-2018
Using grep for finding StackTraces
Hello,

I've a horrible problem using grep. I want to "grep" stacktraces from a Logfile. Caused by performance i don't want to read the files line-by-line.

I allready used a regex-online-tester (succesful) but und the commandline I only get the first line - not the full stack-trace.

My command is:

Code:
 cat sample.log | grep -o -P "(\d\d\d\d-\d\d-\d\d \d\d:\d\d:\d\d\.\d\d\d ERROR[\s\S]*?)(\d\d\d\d-\d\d-\d\d |\z)"


My Test Data is:

Code:
2018-02-14 09:14:20.050 ERROR 21 --- [batchAsyncTaskExecutor-2] de.mi.test.agent.core.IssueService        : error createExploreIssue:
org.springframework.web.client.HttpClientErrorException: 400 Bad Request
	at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:78)
	at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:700)
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:653)
	at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:613)
	at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:559)
	at sun.reflect.GeneratedMethodAccessor1561.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSite.invoke(PojoMetaMethodSite.java:192)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
	at de.mi.test.agent.core.GitLabRestClient.commitFile(GitLabRestClient.groovy:33)
	at de.mi.test.agent.core.GitLabRestClient$commitFile.call(Unknown Source)
	at de.mi.test.agent.core.IssueService.createIssue(IssueService.groovy:55)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:174)
	at de.mi.test.agent.core.IssueService.createExploreIssue(IssueService.groovy:38)
	at de.mi.test.agent.core.IssueService$createExploreIssue.call(Unknown Source)
	at de.mi.test.agent.core.ExploreExceptionSkipListener.onSkipInProcess(ExploreExceptionSkipListener.groovy:34)
	at de.mi.test.agent.core.ExploreExceptionSkipListener.onSkipInProcess(ExploreExceptionSkipListener.groovy)
	at de.mi.test.agent.core.ExploreExceptionSkipListener$$FastClassBySpringCGLIB$$42ab6582.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
	at de.mi.test.agent.core.ExploreExceptionSkipListener$$EnhancerBySpringCGLIB$$97e4579e.onSkipInProcess(<generated>)
	at org.springframework.batch.core.listener.CompositeSkipListener.onSkipInProcess(CompositeSkipListener.java:87)
	at org.springframework.batch.core.listener.MulticasterBatchListener.onSkipInProcess(MulticasterBatchListener.java:338)
	at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.callProcessSkipListener(FaultTolerantChunkProcessor.java:478)
	at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.callSkipListeners(FaultTolerantChunkProcessor.java:450)
	at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.write(FaultTolerantChunkProcessor.java:438)
	at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:199)
	at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:75)
	at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:406)
	at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:330)
	at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
	at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:272)
	at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:81)
	at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:374)
	at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215)
	at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:144)
	at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:257)
	at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:200)
	at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148)
	at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:64)
	at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67)
	at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169)
	at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144)
	at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:134)
	at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:306)
	at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:135)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

2018-02-14 09:14:24.495  INFO 21 --- [batchAsyncTaskExecutor-2] d.m.m.a.portal.portalExploreService  : -> process [47722,1 (928456)]
2018-02-14 09:14:24.626  INFO 21 --- [batchAsyncTaskExecutor-1] d.m.m.a.portal.portalExploreService  : -> process [48903 (363675)]
2018-02-14 09:14:25.031  INFO 21 --- [batchAsyncTaskExecutor-2] io.github.bonigarcia.wdm.BrowserManager  : Exporting webdriver.gecko.driver as /root/.m2/repository/webdriver/geckodriver/linux64/0.17.0/geckodriver
2018-02-14 09:14:25.086  INFO 21 --- [batchAsyncTaskExecutor-1] io.github.bonigarcia.wdm.BrowserManager  : Exporting webdriver.gecko.driver as /root/.m2/repository/webdriver/geckodriver/linux64/0.17.0/geckodriver
2018-02-14 09:14:28.181  INFO 21 --- [Forwarding newSession on session null to remote] o.o.selenium.remote.Proto

My result is:

Code:
2018-02-14 09:14:20.050 ERROR 21 --- [batchAsyncTaskExecutor-2] de.mi.test.agent.core.IssueService        : error createExploreIssue:

Expected would be:

Code:
2018-02-14 09:14:20.050 ERROR 21 --- [batchAsyncTaskExecutor-2] de.mi.test.agent.core.IssueService        : error createExploreIssue:
org.springframework.web.client.HttpClientErrorException: 400 Bad Request
	at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:78)
	at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:700)
	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:653)
	at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:613)
	at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:559)
	at sun.reflect.GeneratedMethodAccessor1561.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSite.invoke(PojoMetaMethodSite.java:192)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
	at de.mi.test.agent.core.GitLabRestClient.commitFile(GitLabRestClient.groovy:33)
	at de.mi.test.agent.core.GitLabRestClient$commitFile.call(Unknown Source)
	at de.mi.test.agent.core.IssueService.createIssue(IssueService.groovy:55)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:210)
	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:174)
	at de.mi.test.agent.core.IssueService.createExploreIssue(IssueService.groovy:38)
	at de.mi.test.agent.core.IssueService$createExploreIssue.call(Unknown Source)
	at de.mi.test.agent.core.ExploreExceptionSkipListener.onSkipInProcess(ExploreExceptionSkipListener.groovy:34)
	at de.mi.test.agent.core.ExploreExceptionSkipListener.onSkipInProcess(ExploreExceptionSkipListener.groovy)
	at de.mi.test.agent.core.ExploreExceptionSkipListener$$FastClassBySpringCGLIB$$42ab6582.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
	at de.mi.test.agent.core.ExploreExceptionSkipListener$$EnhancerBySpringCGLIB$$97e4579e.onSkipInProcess(<generated>)
	at org.springframework.batch.core.listener.CompositeSkipListener.onSkipInProcess(CompositeSkipListener.java:87)
	at org.springframework.batch.core.listener.MulticasterBatchListener.onSkipInProcess(MulticasterBatchListener.java:338)
	at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.callProcessSkipListener(FaultTolerantChunkProcessor.java:478)
	at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.callSkipListeners(FaultTolerantChunkProcessor.java:450)
	at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.write(FaultTolerantChunkProcessor.java:438)
	at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:199)
	at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:75)
	at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:406)
	at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:330)
	at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
	at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:272)
	at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:81)
	at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:374)
	at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215)
	at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:144)
	at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:257)
	at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:200)
	at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148)
	at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:64)
	at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67)
	at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169)
	at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144)
	at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:134)
	at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:306)
	at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:135)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Thank you for any idea or tip.
Best regards
Mpmichael
# 2  
Old 03-13-2018
how about awk:
Code:
awk '/ERROR/' RS= sample.log

This User Gave Thanks to vgersh99 For This Post:
mpmichael
# 3  
Old 03-13-2018
Hello,

thank you very much for this amazing tip! It worked for the sample file. Now I tried on real logfiles and the result where that it was delivered to much results. If the Word "error" is in the text the line is also delivered. But there should only be lines where "ERROR" is behind the time.
BR
mpmichael
# 4  
Old 03-13-2018
could you attach a small sample of the the file including the line(s) that break.
# 5  
Old 03-13-2018
thank you for the help. I attached the logfile.

The Command I used was:

Code:
awk '/ERROR/' RS= testlog2.txt

and the (false) Result where i. E. lines 741 - 777

BR
mpmichael
testlog2.txt (72.9 KB) 
# 6  
Old 03-13-2018
How about
Code:
sed -n '/ERROR/p; //,/^[0-9]\{4\}-/ {/^[0-9]\{4\}-/!p}' /tmp/testlog2.txt

Tested on Ubuntu linux 17.10
This User Gave Thanks to RudiC For This Post:
mpmichael
# 7  
Old 03-14-2018
Hello RudiC,

many thank's it works as desired!
Great help!
BR
mpmichael
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Finding The Complete SQL statement Using PDFGREP Or Grep

Linux Gods, I am simply attempting to parse SQL statements from a PDF doc in creating a base SQL script at a later time but for the life of me, am having a tough time extracting this data.This exact string worked perfectly a couple of months ago and now it doesnt. Below is an example of the data... (4 Replies)
Discussion started by: metallica1973
4 Replies

2. Shell Programming and Scripting

Error in finding the PID by grep and assigning to a variable

Hello All, I am facing difficulty in getting the PID value & then assigning it to a variable, kindly help me in resolving the issue. Thanks a lot in advance. The custom utility used inside the Test2.sh will process the file in a batch of 10 lines at once and for efficient memory management,... (3 Replies)
Discussion started by: duddukuri
3 Replies

3. UNIX for Advanced & Expert Users

Finding/Grep on files with date and hour in the file name

Hi, I have a folder structure as follows, DATA -> 2012-01-01 -> 00 -> ABC_2012-01-03_00.txt -> 01 -> ABC_2012-01-03_01.txt -> 02 -> ABC_2012-01-03_02.txt ... -> 23 -> ABC_2012-01-03_02.txt -> 2012-01-02 ... (1 Reply)
Discussion started by: mihirvora16
1 Replies

4. Shell Programming and Scripting

Finding/Grep on files with date and hour in the file name

Hi, I have a folder structure as follows, DATA -> 2012-01-01 -> 00 -> ABC_2012-01-03_00.txt -> 01 -> ABC_2012-01-03_01.txt -> 02 -> ABC_2012-01-03_02.txt ... -> 23 -> ABC_2012-01-03_02.txt -> 2012-01-02 -> 2012-01-03 So the dir DATA contains the above hierarchy, User input Start and... (6 Replies)
Discussion started by: mihirvora16
6 Replies

5. UNIX for Dummies Questions & Answers

finding keywords in many files using grep

Hi to all Sorry for the confusion because I did not explain the task clearly. There are many .hhr files in a folder There are so many lines in these .hhr files but I want only the following 2 lines to be transferred to the output file. The keyword No 1 and all the words in the next line They... (5 Replies)
Discussion started by: raghulrajan
5 Replies

6. Shell Programming and Scripting

Finding the executable files of a directory using Grep

Hi guys, Can you please help me print all the executable files of a directory(in this case /home) using grep? All i know is that this command should do it but it doesnt... ls -l ~ | grep -..x it shows me the following mesage grep: invalid option -- '.' Χρήση: grep ... ΥΠΟΔΕΙΓΜΑ ... (3 Replies)
Discussion started by: jimas13
3 Replies

7. Homework & Coursework Questions

finding pattern without grep in unix

how can i find related pattern in a text file without using grep command in unix (2 Replies)
Discussion started by: feint
2 Replies

8. UNIX for Dummies Questions & Answers

Finding a pattern with grep

I am interested in finding a pattern that looks like this AABBACC These letters are just for example but they can be any letter in the alphabet as long as they are different from each other, but the letter found in a certain spot should act like in the pattern above. Thanks! (7 Replies)
Discussion started by: moyzZ
7 Replies

9. Shell Programming and Scripting

finding the line number from a grep ?

Hi there does anybody know how i can get the line number from an entry or entries in a file ?? for example if i had a file test1 test2 test3 test1 and i needed to get the line numbers for all instances of test1 in that file with the answer being (1,4) Would anybody be able... (7 Replies)
Discussion started by: hcclnoodles
7 Replies

10. Shell Programming and Scripting

Finding number ranges using grep

Is it possible for me to find numbers in a file by a range using grep? like cat data | cut -f1 | grep <info> Im trying to find information and extract every amount that is less than a number (ie less than 75 or whatever) Is this possible? (2 Replies)
Discussion started by: DKNUCKLES
2 Replies
Login or Register to Ask a Question