Sponsored Content
Top Forums Shell Programming and Scripting Parsing log files, displaying logs between specific dates Post 302294475 by Rhije on Thursday 5th of March 2009 09:43:19 AM
Old 03-05-2009
Parsing log files, displaying logs between specific dates

Sorry, couldn't really think of a simple subject/title.

So, I have a log file, and the dates are displayed like so:

2009-03-05 02:49:44

So the first and second field are the date/time. I can change them into a unix timestamp easily with:

date -d "2009-03-05 02:49:44" +%s

However, trying to do that is not as easy as I thought. I wanted to get all logs between X and Y time (lets say for a few hours on Z day). To make this easily reusable, I would rather do it this way.

I have tried to use awk with the system() function, but maybe I did not use it correctly..

awk '{ system("date -d " $1,$2 " +%s"); }'

Of course, I have tried other varieties of this, but to no avail. I have thought of possibly doing it other ways, but I am not even sure what will work now.. Anyone have any suggestions?
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

reading files for specific dates

assume files are in a directory /data $ ls -ltr Just displaying Data and file name: 01/01/2004 file_3434_typea.dat 01/01/2004 file_3423_typea.dat 01/01/2004 file_3436_typea.dat 01/01/2004 file_3434_typeb.dat 01/01/2004 file_3423_typeb.dat 01/01/2004 file_3436_typeb.dat ... (3 Replies)
Discussion started by: siva_jm
3 Replies

2. Shell Programming and Scripting

Getting list of all the log files between two dates

I need to get the list of all the log files for a particular duration, say between two dates,i.e I need to get the list of all the log files between date1 and date2.The two dates are entered by the user. The format of the log file is : /path_name/graph_name_20080801180308.log. I dont... (1 Reply)
Discussion started by: avishekp
1 Replies

3. Shell Programming and Scripting

Parsing out the logs and generating report

My file will contain following(log.txt): start testcase: config loading ...... error XXXX ..... end testcase: config loading, result failed start testcase: ping check ..... error ZZZZZ ..... error AAAAA end testcase: Ping check, result failed I am expecting below output. ... (4 Replies)
Discussion started by: shellscripter
4 Replies

4. UNIX for Dummies Questions & Answers

To find files with specific dates and cp another folder.

Hi all, We have an existing script: find /u03/oraprod/perpcomn/admin/out -type f -ctime +7 \ -exec cp {} "/u08/oraprod/backup/cout" \; Which is to find all files more than 7 days and copy to another folder. However I would like to only list files with Sep 29, and cp to another folder. ... (2 Replies)
Discussion started by: *Jess*
2 Replies

5. Shell Programming and Scripting

Help parsing logs maybe with menu and variables?

I would like to parse through some logs looking for things like exception or failed (grep -i failed). Ideal would be if it were in a menu format so someone without unix ability could just choose option 1 2 or 3 etc. If I could pass the hostname to a variable also that would be awesome, so someone... (5 Replies)
Discussion started by: taekwondo
5 Replies

6. UNIX and Linux Applications

Parsing Tuxedo Logs

Right now I am parsing Tuxedo logs to calculate response times for various services. I was hoping to find a log tool that had support for Tuxedo and would generate drill down html reports. ---------- Post updated at 02:35 PM ---------- Previous update was at 02:33 PM ---------- I just wanted... (0 Replies)
Discussion started by: Lurch
0 Replies

7. Debian

Logrotate truncated my log files to 0 bytes and no logs are written

Hi, Yesterday I installed and configured logrotate on my Debian machine. I was expecting this to run at 06:25 in the morning and it actually did. All my old logs were compressed and zipped but the new logs were all with size equal to 0 bytes. Processes, while still running ok, they were not... (2 Replies)
Discussion started by: pmatsinopoulos
2 Replies

8. Shell Programming and Scripting

Redirect all logs files contents into a single log file

Hi , I have a Data cleansing process which creates different log file for each step , when the process runs it creates following log files in below order: p1_tranfrmr_log.txt p1_tranfrmr_stats.txt p2_globrtr_log.txt p2_globrtr_stats.txt p3_cusparse_log.txt p3_cusparse_stats.txt ' '... (8 Replies)
Discussion started by: sonu_pal
8 Replies

9. Shell Programming and Scripting

Search for logs traced between specific date and time from log file

HI, I want to search for a logs which are trace between specific date and time from logs file. My logs are generated like this :- Tue Jun 18 05:00:02 EEST 2013 | file_check.sh| Message:script has files to process. Thu Jun 20 05:00:02 EEST 2013 | file_check.sh| Message:script has files to... (5 Replies)
Discussion started by: ketanraut
5 Replies

10. Red Hat

Moving files with specific dates

Hi, These are the list of files in one directory in the server : # ls -lrt total 10120 -rw-r--r-- 1 root root 4484 Jul 8 2011 install.log.syslog -rw-r--r-- 1 root root 51890 Jul 8 2011 install.log -rw------- 1 root root 3140 Jul 8 2011 anaconda-ks.cfg drwxr-xr-x 2 root root... (2 Replies)
Discussion started by: anaigini45
2 Replies
GENPLOTFILES(1) 						      colplot							   GENPLOTFILES(1)

NAME
genplotfiles - utility to help automate creation of collectl plot file from raw ones SYNOPSIS
/opt/hp/colplot/genplotfiles.pl -outdir [-switches] DESCRIPTIONS
This utility is not intended to meet all plot file creation needs, but quite frankly should meet most. Features not provided with this utility are best realized by running collectl directly in playback mode (see collectl -p). It is therefore important to understand what this utility will and will not do for you before building an automation strategy around it. The algorithm used is fairly simple - genplotfiles examines each collectl raw file to see it it falls within the data ranges (if specified) and contains a matching hostname (if specified). If a plot file already exists, the timestamps are compared and the raw file skipped if its newer than the plotfile. Any raw files that pass all these tests are then passed to collectl with their timestamp wildcarded. The result is all raw files for that host/date will be processed. This means if you copy over collectl raw files multiple times over the course of the day using rsync, you can immediately genererate up-to-date plots for them using genplotfiles. Furthermore, you won`t even need any date parameters if the older plot files are alreay up to date. For an example of what a cron job that aggregates multiple collectl logs onto a single system for plotting purposes, see crontab.sh in the same directory as this script. SWITCHES
All switches are optional with the exception of -outdir: -colbin path-to-collectl This points to collectl and is only necessary if /usr/sbin is not in your path or you`ve chosen to install collectl in a non-stan- dard directory. -debug number Primarily for development, will provide intermediate processing information based on its value. See souce for details. -force Always generate plot files even in there is an existing one that is newer than the latest raw file. This will be necessary if you want to rerun this script and just change the value of -subsys. -help Print a summary of available switches. -host string Only process files that contain this string in their hostname portion of the filename. -indir directory Directory where to look for collectl logs. The default is /var/log/collectl and is therefore only necessary if your logs are in a non-standard location. -outdir directory Directory into which to write the plottable files. This field is required. -fromdate date Ignore files with a datestamp in their name less that this value. The default is 20050101 and so defaults to all files. The value of `yesterday` is also recognized to simplify scripts that call this. -thrudate date Ignore files with a datestamp in their name with a datestamp greater than this value. The default is the current day and one can also specify a value of `yesterday`. -subsys string Instead of using the default subsystems for which data was collected, generate output files based on this set. There are basically two reasons for doing this. The first is that you may want to reduce the size of the plot files, though with compression there is rarely much point in this. The second is if you`ve asked collectl to collect summary data but want to plot detail data. See the colplot FAQ for more information. AUTHOR
This program was written by Mark Seger (mjseger@gmail.com). Copyright 2005 Hewlett-Packard Development Company, L.P. SEE ALSO
LOCAL
OCTOBER 2005 GENPLOTFILES(1)
All times are GMT -4. The time now is 04:54 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy