Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Parse apache log file with three different time formats
Top Forums UNIX for Beginners Questions & Answers Parse apache log file with three different time formats Post 303038328 by stomp on Friday 30th of August 2019 02:08:03 PM
Old 08-30-2019
If the logs are very big it may be a good trick to read them backwards, because maybe the interesting part is more likely at the end of the file, so we maybe save to read tons of old lines that way:

Code:
#!/bin/sh
logfile="$3"

# reverse at the beginning to read from end to start
tac "$logfile" | awk -vstart="$1" -vend="$2" ' 

BEGIN {
        start_epoch = mktime(start)
        end_epoch   = mktime(end)
}

function monthnumber(monthname) {
        return sprintf("%02d\n",(match("JanFebMarAprMayJunJulAugSepOctNovDec",monthname)+2))/3
}

match($0,/^([0-9]+)\/([a-zA-Z]+)\/([0-9]{4}):([0-9]{2}):([0-9]{2}):([0-9]{2})/,r) { 
        current=mktime( sprintf("%s %s %s %s %s %s", r[3],monthnumber(r[2]),r[1],r[4],r[5],r[6])); }

match($0,/^[a-zA-Z]+ ([a-zA-Z]+) ([0-9]+) ([0-9]+):([0-9]+):([0-9]+) ([0-9]{4})/,r) { 
        current=mktime( sprintf("%s %s %s %s %s %s", r[6],monthnumber(r[1]),r[2],r[3],r[4],r[5])); }

match($0,/^([0-9]+)-([a-zA-Z]+) ([0-9]+):([0-9]+):([0-9]+)/,r) { 
        current=mktime( sprintf("%s %s %s %s %s %s", strftime("%Y"),monthnumber(r[2]),r[1],r[3],r[4],r[5])); } 

# we have to swap the actions here!
(current < start_epoch) { exit }
(current > end_epoch  ) { next }

1
' | tac 
# and reverse again at the end to return to chronological order

Script call stays the same.
Last edited by stomp; 08-30-2019 at 06:57 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Processing a log file based on date/time input and the date/time on the log file

Hi, I'm trying to accomplish the following and would like some suggestions or possible bash script examples that may work I have a directory that has a list of log files that's periodically dumped from a script that is crontab that are rotated 4 generations. There will be a time stamp that is... (4 Replies)
Discussion started by: primp
4 Replies

2. Shell Programming and Scripting

Setting of two time formats in one machine

Hi, Is it possible to set the two time formats in a single machine. My machine time is in EST and the logs are in PST. What would be the issue, and how to make change of this.? (5 Replies)
Discussion started by: gsiva
5 Replies

3. Shell Programming and Scripting

sed command to parse Apache config file

Hi there, am trying to parse an Apache 'server' config file. A snippet of the config file is shown below: ..... ProxyPassReverse /foo http://foo.example.com/bar ..... ..... RewriteRule ^/(.*) http://www.example.com/$1 RewriteRule /redirect https://www.example1.com/$1 ........ (7 Replies)
Discussion started by: jy2k7ca
7 Replies

4. Shell Programming and Scripting

Extracting data from a log file with date formats

Hello, I have a log file for the year, which contains lines starting with the data in the format of YYYY-MM-DD. I need to get all the lines that contain the DD being 04, how would I do this? I tried using grep "*-*04" but it didn't work. Any quick one liners I should know about? Thank you. (2 Replies)
Discussion started by: cpickering
2 Replies

5. Shell Programming and Scripting

Check/Parse log file's lines using time difference/timestamp

I was looking at this script which outputs the two lines which differs less than one sec. #!/usr/bin/perl -w use strict; use warnings; use Time::Local; use constant SEC_MILIC => 1000; my $file='infile'; ## Open for reading argument file. open my $fh, "<", $file or die "Cannot... (1 Reply)
Discussion started by: cele_82
1 Replies

6. Shell Programming and Scripting

Using awk to parse a file with mixed formats in columns

Greetings I have a file formatted like this: rhino grey weight=1003;height=231;class=heaviest;histology=9,0,0,8 bird white weight=23;height=88;class=light;histology=7,5,1,0,0 turtle green weight=40;height=9;class=light;histology=6,0,2,0... (2 Replies)
Discussion started by: Twinklefingers
2 Replies

7. Shell Programming and Scripting

Parse A Log File

Hello All, Below is the excerpt from my Informatica log file which has 4 blocks of lines (starting with WRITER_1_*_1). Like these my log file will have multiple blocks of same pattern. WRITER_1_*_1> WRT_8161 TARGET BASED COMMIT POINT Thu May 08 09:33:21 2014... (13 Replies)
Discussion started by: Ariean
13 Replies

8. Shell Programming and Scripting

Shell Script | Parse log file after a given date and time stamp

I am developing one script which will take log file name, output file name, date, hour and minute as an argument and based on these inputs, the script will scan and capture all the error(s) that have been triggered from a given time. Example: script should capture all the error after 13:50 on Jan... (2 Replies)
Discussion started by: ROMA3
2 Replies

9. Shell Programming and Scripting

Comparing different time formats

I am trying to do a comparison of files based on their last modified date. I am pulling the first file from a webapp folder using curl. curl --silent -I http://localhost:8023/conf/log4j2.xml | grep Last Last-Modified: Tue, 22 Mar 2016 22:02:18 GMT The second file is on local disk. stat... (2 Replies)
Discussion started by: Junaid Subhani
2 Replies

10. Shell Programming and Scripting

Perl to parse a variety of formats

The below perl script parses a variety of formats. If I use the numeric text file as input the script works correctly. However using the alpha text file as input there is a black output file. The portion in bold splits the field to parse f or NC_000023.10:g.153297761C>A into a variable $common but... (3 Replies)
Discussion started by: cmccabe
3 Replies

LEARN ABOUT CENTOS

pmdaapache

PMDAAPACHE(1)						      General Commands Manual						     PMDAAPACHE(1)

NAME

       pmdaapache - Apache2 web server performance metrics domain agent (PMDA)

SYNOPSIS

       $PCP_PMDAS_DIR/apache/pmdaapache [-d domain] [-l logfile] [-U username] [-S server] [-P port] [-L location]

DESCRIPTION

       pmdaapache is a Performance Metrics Domain Agent (PMDA) which extracts performance metrics describing the state of an Apache web server.

       The  apache  PMDA  exports metrics that measure the request rate, cumulative request sizes, uptime and various connection states for active
       clients.

       This information is obtained by performing a HTTP request to the server status URL, which must be enabled in the  httpd.conf  configuration
       file.

	    ExtendedStatus on
	    <Location /server-status>
	    SetHandler server-status
	    Order deny,allow
	    Deny from all
	    Allow from localhost
	    </Location>

       A brief description of the pmdaapache command line options follows:

       -d   It	is  absolutely crucial that the performance metrics domain number specified here is unique and consistent.  That is, domain should
	    be different for every PMDA on the one host, and the same domain number should be used for the same PMDA on all hosts.

       -l   Location of the log file.  By default, a log file named apache.log is written in the current directory of pmcd(1) when  pmdaapache	is
	    started,  i.e.   $PCP_LOG_DIR/pmcd	.   If	the log file cannot be created or is not writable, output is written to the standard error
	    instead.

       -S   Query the Apache status information from the named server rather than the local host.

       -P   Query the Apache status information from the given port rather than the default (80).

       -L   Specify an alternative location for finding the server-status page.

       -U   User account under which to run the agent.	The default is the unprivileged "pcp" account in current versions of  PCP,  but  in  older
	    versions the superuser account ("root") was used by default.

INSTALLATION

       If you want access to the names, help text and values for the apache performance metrics, do the following as root:

	    # cd $PCP_PMDAS_DIR/apache
	    # ./Install

       If you want to undo the installation, do the following as root:

	    # cd $PCP_PMDAS_DIR/apache
	    # ./Remove

       pmdaapache  is  launched by pmcd(1) and should never be executed directly.  The Install and Remove scripts notify pmcd(1) when the agent is
       installed or removed.

FILES

       $PCP_PMCDCONF_PATH
		 command line options used to launch pmdaapache
       $PCP_PMDAS_DIR/apache/help
		 default help text file for the apache metrics
       $PCP_PMDAS_DIR/apache/Install
		 installation script for the pmdaapache agent
       $PCP_PMDAS_DIR/apache/Remove
		 undo installation script for the pmdaapache agent
       $PCP_LOG_DIR/pmcd/apache.log
		 default log file for error messages and other information from pmdaapache

PCP ENVIRONMENT

       Environment variables with the prefix PCP_ are used to parameterize the file and directory names used by PCP.  On  each	installation,  the
       file  /etc/pcp.conf contains the local values for these variables.  The $PCP_CONF variable may be used to specify an alternative configura-
       tion file, as described in pcp.conf(5).

SEE ALSO

       PCPIntro(1), httpd(8), pmcd(1), pcp.conf(5) and pcp.env(5).

Performance Co-Pilot							PCP							     PMDAAPACHE(1)
All times are GMT -4. The time now is 07:20 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy