Range of data from a log file .


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Range of data from a log file .
# 1  
Old 02-24-2010
Range of data from a log file .

I am having a log file in which i need a range of data from specific date range.

Code:
$cat my.log
Jan 07 15:39:03 N/A [error] _M_LocalDirectory     listFiles(): listing files from dir
Jan 07 15:39:03 N/A [error] _w_fm_log:_f_Push()   _w_fm_log_export_ftppush(): Files f
Jan 07 05:58:35 N/A [debug] _w_fm_log_autoexport  1
Jan 08 05:58:35 N/A [debug] _w_fm_log_autoexport  2/var/www/html/sites/default/mods/_
Jan 08 05:58:36 N/A [error] _M_Channel            getIntByNode(): getting _chan from 
Jan 08 05:58:36 N/A [debug] _w_fm_log:_f_Push()   Array
Jan 08 05:58:36 N/A [notice] _w_fm_log:_f_Push()   _w_fm_log_export_active_ftppush__c
Jan 08 05:58:36 N/A [error] _M_LocalDirectory     isDirectory(): directory '/opt/_d_/
Feb 03 14:32:01 N/A [debug] _w_fm_log_mod      Inside _w_fm_log_nodeapi me..
Feb 03 14:32:02 N/A [error] _w_fm_log_mod      Inside _w_fm_log_nodeapi me..
Feb 03 14:32:02 N/A [debug] _w_fm_log_mod      Inside _w_fm_log_nodeapi me..
Feb 05 13:54:32 N/A [debug] _w_fm_log_mod      Inside _w_fm_log_nodeapi me..
Feb 05 13:54:32 N/A [error] _w_fm_log_mod      Inside _w_fm_log_nodeapi me..
Feb 05 13:22:44 N/A [notice] _w_fm_log_mod      Inside _w_fm_log_nodeapi me..
Feb 05 14:22:34 N/A [error] _w_fm_log_mod      Inside _w_fm_log_nodeapi  me..
Feb 07 13:22:44 N/A [debug] _w_fm_log_mod      Inside _w_fm_log_nodeapi me..

From this i need the lines which having errors if i will give date Jan 08 and Feb 05, It have to show the the range of errors between the date range.
But

Code:
$cat my.log |sed -n '/Jan 08/,/Feb 05/p' |sed -n '/error/p' 

Jan 08 05:58:36 N/A [error] _M_Channel            getIntByNode(): getting _chan from
Jan 08 05:58:36 N/A [error] _M_LocalDirectory     isDirectory(): directory '/opt/_d_/
Feb 03 14:32:02 N/A [error] _w_fm_log_mod      Inside _w_fm_log_nodeapi me..

In sed for range of pattern search /pattern1/,/pattern2/ here i am searching till the first occurrence of pattern2 , But how could i do till the last occurrence of the pattern2.
and i have to count the numbers of error as the mentioned date

So the desired result be

Code:
Jan 08 05:58:36 N/A [error] _M_Channel            getIntByNode(): getting _chan from
Jan 08 05:58:36 N/A [error] _M_LocalDirectory     isDirectory(): directory '/opt/_d_/
Feb 03 14:32:02 N/A [error] _w_fm_log_mod      Inside _w_fm_log_nodeapi me..
Feb 05 13:54:32 N/A [error] _w_fm_log_mod      Inside _w_fm_log_nodeapi me..
Feb 05 14:22:34 N/A [error] _w_fm_log_mod      Inside _w_fm_log_nodeapi me..

Result:
Jan 08 : 2 error
Feb 03 : 1 error
Feb 05 : 2 error

Thank & Regards
Posix
---------------------------
I love to work in sed and awk.....
# 2  
Old 02-24-2010
Hi,

I do not have a solution per say but why dont you just increase your search?
Code:
sed -n '/Jan 08/,/Feb 06/p'

Ah ha!
"Increasing" the search will not work. Sorry.

Last edited by ni2; 02-24-2010 at 04:17 AM.. Reason: Removing repetition.
# 3  
Old 02-24-2010
Try:

Code:
awk '/Jan 08/ { f=1; }
                /Feb 05/ {c=1; }
                c&&!/Feb 05/ { c=0;f=0; }
                (f||c) && /rror/ { Arr[$1" "$2]++ }
                END { for (i in Arr) print i" "Arr[i]" error"; }' file

# 4  
Old 02-25-2010
Nice post by Dj

Could it is possible that if we give pattern2 which is not available in file then it's going for whole file search & result is coming in that format.

If i give Feb 04 rather Feb 05 it should not display data after the Feb 04 date.
# 5  
Old 02-25-2010
Bug

While using variables i am getting a problem in the 2nd date replacement of the 2nd variables in code
Code:
$cat test.sh
#!/bin/sh
echo -e " Please provide the log file"
read file

echo -e " Plese give start date\n"
read st_dt

echo -e " Please give End date \n"
read en_dt

awk "/$st_dt/" ' { f=1; }
               "/$en_dt/" {c=1; }
                c&&! "/$en_dt/" { c=0;f=0; }
                (f||c) && /error/ { Arr[$1" "$2]++ }
                END { for (i in Arr) print i" "Arr[i]" error"; }'"$file"

What is the best we can do for it?
Also got confused that how i'll do multiple variable declaration in awk.
# 6  
Old 02-25-2010
Hello Posix,you can use the following script to get your desired output.

The following script will give the result as you defined.

Code:
cat my.log |sed  -n '/Jan 08/,/Feb 05/p'| grep 'error' > temp  #getting the content from Jan 08 to Feb 05
cat my.log | sed -n '/Feb 05/,/Feb 05/p' |grep 'error' >> temp #appending the Feb 05 content to the temp file
i=0 #initializing the iterator variable
cat temp #printing the matched content
check=0 #initializing the variable to 0 to check the number of occurrences of errors for particular day
while read line
do
  current=`echo $line | cut -d ' ' -f 1,2` #getting the date alone from the matched lines
  for ((l=0;l<$i;l++))
  do
    if [[ ${k[$l]} == $current ]] #checking whether the content is already available or not to avoid duplicates
    then
      check=1
    fi
  done
  if [[ $check -ne 1 ]]
  then
    k[$i]=$current #adding the value to an array
    let i=i+1;
  fi
  check=0
done <temp 

ind=1
echo -e "\nResult:"
for ((j=0;j<$i;j++)) #printing the final result 
do
  echo "${k[$j]} : `grep -c "${k[$j]}" temp ` error" #grep -c is used to get the count of errors for each day
done


Last edited by Franklin52; 02-25-2010 at 02:03 PM.. Reason: Please indent your code and use code tags, thank you
# 7  
Old 02-25-2010
I think you just need the date and error counts, so this will help.

Code:
awk '/\[error\]/ {a[$1 FS $2]++} END {for (i in a) print i" : "a[i] " error(s)"}' my.log

Feb 03 : 1 error(s)
Feb 05 : 2 error(s)
Jan 07 : 2 error(s)
Jan 08 : 2 error(s)

Last edited by rdcwayx; 02-25-2010 at 10:55 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extract data from log file in specified range of time

I was searching for parsing a log file and found what I need in this link http://stackoverflow.com/questions/7575267/extract-data-from-log-file-in-specified-range-of-time But the most useful answer (posted by @Kent): # this variable you could customize, important is convert to seconds. # e.g... (2 Replies)
Discussion started by: kingk110
2 Replies

2. Shell Programming and Scripting

Grep in a log file within a time range (hour)

Hi, im trying to write a grep script that returns me the last inputs added in the last hour in the log file. Literally i have nothing yet but: grep 'Line im looking for' LOGFILE.log | tail -1 this only gives me the last input, but no necessarily from the last hour. Help Please. (4 Replies)
Discussion started by: blacksteel1988
4 Replies

3. Shell Programming and Scripting

Filtering data base on range

Hi All, I have the file in following format. I need to change column 16 and column 17 for those lines which has atleast one numerical digit in column 2 , with the exception on SUPP1 Input file : S00005615|1044|0|0.00|0|0.00| |112|-30|28.1|0|0| |20130331|220.2|2|3|... (2 Replies)
Discussion started by: nua7
2 Replies

4. Shell Programming and Scripting

Display data from a range of dates

I have a data in a file called SCHED which has 5 columns: sched no, date, time, place and remarks. The image is shown below. http://dl.dropbox.com/u/54949888/Screenshot%20from%202013-01-02%2002%3A42%3A25.png Now, I want to display only the schedules which fall under a certain date range which... (2 Replies)
Discussion started by: angilulu
2 Replies

5. Shell Programming and Scripting

Search for a specific data in a file based on a date range

Hi, Currently I am working on a script to automate the process of converting the log file from binary into text format. To achieve this, partly I am depending on my application’s utility for this conversion and the rest I am relying on shell commands to search for directory, locate the file and... (5 Replies)
Discussion started by: svajhala
5 Replies

6. UNIX for Dummies Questions & Answers

Fitting a range of data

Hi, I am able to write an awk program that fits using the chi squared minimization method for each number is a data column... but I am wondering if it is possible to do that for a range of numbers at the same time. If I have a column for the observed data... and then say 10 columns for... (13 Replies)
Discussion started by: cosmologist
13 Replies

7. Shell Programming and Scripting

Extracting specific lines of data from a file and related lines of data based on a grep value range?

Hi, I have one file, say file 1, that has data like below where 19900107 is the date, 19900107 12 144 129 0.7380047 19900108 12 168 129 0.3149017 19900109 12 192 129 3.2766666E-02 ... (3 Replies)
Discussion started by: Wynner
3 Replies

8. Shell Programming and Scripting

Read Write byte range/chunk of data from specific location in file

I am new to Unix so will really appreciate if someone can guide me on this. What I want to do is: Step1: Read binary file - pick first 2 bytes, convert from hex to decimal. Read the next 3 bytes as well. 2 bytes will specify the number of bytes 'n' that I want to read and write... (1 Reply)
Discussion started by: Kbenipel
1 Replies

9. Shell Programming and Scripting

Parse a range of data

Hello, I have a file which has a range of date like: 00:00 test 00:01 test2 00:02 test3 00:03 test4 00:04 test5 00:05 test6 Using input (stdin) i would like to parse the data 00:01 to 00:04. The output file should be like this: 00:01 test2 00:02 test3 00:03 test4 00:04 test5 ... (5 Replies)
Discussion started by: BufferExploder
5 Replies

10. Shell Programming and Scripting

Log File date compare for user defined range

:confused: Hi i am a noob and need a little help to finish my shell script. I am learning as i go but hit a problem. I am search thorugh logs(*.rv) files to find entires between two user defined dates, The script so far looks for the "START" and "END" of each entry at sees if it belongs To... (0 Replies)
Discussion started by: mojo24
0 Replies
Login or Register to Ask a Question