Help needed to extract distinct logs from a Log File


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Help needed to extract distinct logs from a Log File
# 1  
Old 07-03-2012
Error Help needed to extract distinct logs from a Log File

Hi,
I urgently need some help how to extract distinct entries from a Log file.
The Log File may have same error occuring many times so how do i count the occurance of an error in file and also extract out distinct errors in a file.
Eg:-
I have a file name A.log
it contains entries as below:-
Error 1
Error 2
Error 1
Error 1
Error 2

I want distinct errors and their occurances.
# 2  
Old 07-03-2012
What you have tried so far ..

---------- Post updated at 04:28 PM ---------- Previous update was at 02:51 PM ----------

@ roro , For your attention : https://www.unix.com/shell-programmin...ol-values.html
# 3  
Old 07-03-2012
Code:
egrep -v "Error 1|Error 2" logfile

this will remove ALL the entries you dont care about from the log file and whatever is outputted will be distinct.
# 4  
Old 07-03-2012
Here's a quick and dirty for you
Code:
#Create your working file, if this isn't a running logfile you can skip this step and just pipe the output from sort on A.log the next step.
cp A.log A.log.tmp

#Sort for unique values and grep on your common Error text
sort A.log.tmp | uniq | grep "Error" >> my_unique_errors.log

# 5  
Old 07-06-2012
You can try this :
Code:
 awk 'BEGIN{i=1}(NR==1){a[i]=$0;b[i]=1;i++;next}
{for (j=1;j<i;j++){if($0==a[j]){b[j]+=1;next}};a[i]=$0;b[i]=1;i++}END{for (j=1;j<i;j++){print a[j]":"b[j]}}' a.log

The output will be like this:
Code:
Error 1:3
Error 2:2


Last edited by Franklin52; 07-06-2012 at 11:05 AM.. Reason: Please use code tags for data and code samples, thank you
# 6  
Old 07-06-2012
Code:
cat file | grep  "Error"  | sort | uniq -c
      3 Error 1
      2 Error 2

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Transfer the logs being thrown into /var/log/messages into another file example /var/log/volumelog

I have been searching and reading about syslog. I would like to know how to Transfer the logs being thrown into /var/log/messages into another file example /var/log/volumelog. tail -f /var/log/messages dblogger: msg_to_dbrow: no logtype using missing dblogger: msg_to_dbrow_str: val ==... (2 Replies)
Discussion started by: kenshinhimura
2 Replies

2. Shell Programming and Scripting

How to extract logs between the current time and the last 15 minutes ?

I want to extract the logs between the current time stamp and 15 minutes before and sent an email to the people configured. I developed the below script but it's not working properly; can someone help me?? I have a log file containing this pattern: Constructor QuartzJob ... (3 Replies)
Discussion started by: puneetkhullar
3 Replies

3. Shell Programming and Scripting

Need help in getting the Last 30 minutes logs from the Log File

I have a log file with the below contents : log_file_updated.txt : Jul 5 03:33:06 rsyslogd: was Jul 5 03:33:09 adcsdb1 rhsmd: This system is registered. Sep 2 02:45:48 adcsdb1 UDSAgent: 2015-07-05 04:24:48.959 INFO Worker_Thread_4032813936 Accepted connection from host <unknown>... (3 Replies)
Discussion started by: rahul2662
3 Replies

4. Shell Programming and Scripting

Search for logs traced between specific date and time from log file

HI, I want to search for a logs which are trace between specific date and time from logs file. My logs are generated like this :- Tue Jun 18 05:00:02 EEST 2013 | file_check.sh| Message:script has files to process. Thu Jun 20 05:00:02 EEST 2013 | file_check.sh| Message:script has files to... (5 Replies)
Discussion started by: ketanraut
5 Replies

5. Shell Programming and Scripting

Extract IP from logs and

story ------------------------ The file is a.out, it consist a set of logs from user access to my system ( email system) question -------------------------- using shell script, how can we extract 2 sets of IP output from the a.out log by separate the IP, determine human and non-human... (21 Replies)
Discussion started by: Mr_47
21 Replies

6. Shell Programming and Scripting

Script needed to extract few lines from file

Hello, I need a utility script or command that will extract the following lines from a file based on a 'word' contain in a line. For example my file contains lot of lines. So if i pass 1800182 to the script/command it should return everything between 1st RequestNetRates tag before it and 1st... (4 Replies)
Discussion started by: jakSun8
4 Replies

7. Shell Programming and Scripting

How to print logs in unix prompt or to a log file?

I am executing a stored procedure through shell script. In the procedure, i have dbms_output.put_line to log the comments. This is working fine and but logs are available only after the execution of the pl/sql procedure's completion. I want to print the log comments while executing the procedure... (2 Replies)
Discussion started by: vel4ever
2 Replies

8. Shell Programming and Scripting

Redirect all logs files contents into a single log file

Hi , I have a Data cleansing process which creates different log file for each step , when the process runs it creates following log files in below order: p1_tranfrmr_log.txt p1_tranfrmr_stats.txt p2_globrtr_log.txt p2_globrtr_stats.txt p3_cusparse_log.txt p3_cusparse_stats.txt ' '... (8 Replies)
Discussion started by: sonu_pal
8 Replies

9. Shell Programming and Scripting

Retrieve logs generated in last 10 mins from a log file using 'grep' command

HI All, I have a log file where the logs will be in the format as given below: 2011-05-25 02:32:51 INFO PROCESS STARTING 2011-05-25 02:32:52 INFO PROCESS STARTED . . . I want to retrieve only the logs which are less than 5 mins older than current time using grep... (3 Replies)
Discussion started by: rvhg16
3 Replies

10. Shell Programming and Scripting

script to constantly read the last 500 new logs in a log file

Hello, I would like to write a bash script that would monitor a log file for a certain number of logs, let's say 500 logs and when it reaches that number to write the last log to another file. For example, I want to watch the /var/adm/messages and everytime, there is 500 new logs that are... (1 Reply)
Discussion started by: Pouchie1
1 Replies
Login or Register to Ask a Question