Sponsored Content
Top Forums Shell Programming and Scripting Grep string from logs of last 1 hour on files of 2 different servers and calculate count Post 302450573 by poweroflinux on Friday 3rd of September 2010 06:11:42 AM
Old 09-03-2010
Question Grep string from logs of last 1 hour on files of 2 different servers and calculate count

Hi,
I am trying to grep a particular string from the files of 2 different servers without copying and calculate the total count of its occurence on both files.
File structure is same on both servers and for reference as follows:
Code:
27-Aug-2010 10:04:30,601|919122874903|phtunes_app|1282243292627|NotifySmsReception|DMGenerateLogInterceptor - ExternalTransactionID:SDP-DM-26713018, TransactionStatus:Requested
27-Aug-2010 10:05:30,601|919122874903|phtunes_app|1282243292627|NotifySmsReception|MaskingUnMaskingInterceptor - msisdn before masking 9122874903
27-Aug-2010 16:33:30,627|919122874903|phtunes_app|1282243292627|NotifySmsReception|MaskingUnMaskingInterceptor - msisdn after masking BJ#13340708
27-Aug-2010 16:34:30,637|BJ#13340708|phtunes_app|1282243292627|NotifySmsReception|CP URL:http://172.30.24.52/unitech_sms/unitechsms.php, AppInstanceIdhtunes_app, Keyword:busnews, SID:tel:BJ%2313340708, TransactionID:SDP-DM-26713018



---------- Post updated at 03:11 PM ---------- Previous update was at 03:09 PM ----------

Please consider this example for grep string from logs of last 15 minutes too.

---------- Post updated at 03:41 PM ---------- Previous update was at 03:11 PM ----------

Find the code below which i have made but getting error.
The code checks the last 15 minutes logs and search the string and get the total count.Can anybody correct the following code or provide me the optimized way:
Code:
bin/bash

to=`date +"%d-%b-%Y %T"`
echo $to
let from_in_seconds=`date +%s`-900
from=`date -d @$from_in_seconds +"%d-%b-%Y %T"`
echo $from

shortcodes=( "56882" "58585" "58888" "57575" "57677" );
for shortcode in ${shortcodes[@]}
do
    count=0
    sh_count=`awk '$0>=from && $0<=to' from="$from" to="$to" /opt/bea/ocsg_4.1/logs/customlogs/App_OP.log | grep "ShortCode=tel:${shortcode}" | wc -l`
    count=`expr $count + $sh_count`
    sh_count2=`ssh -n smehan@10.0.0.1 "awk '$0>=from && $0<=to' from=\"$from\" to=\"$to\" /opt/bea/ocsg_4.1/logs/customlogs/App_OP.log | grep \"ShortCode=tel:${shortcode}\" | wc -l"`
    count=`expr $count + $sh_count2`
    echo "${shortcode} : "$count
done

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

how to calculate busy hour

Dear All, please tell me any script which can able to calculate the bus hours by claculating the volume for example hour volume 1 100 2 200 3 300 4 7000 5 50 6 80 7 77 8 77 9 165 10 888 11 99 12 89 13 33 14 676 15 878 16 90 17 56 (14 Replies)
Discussion started by: shary
14 Replies

2. Shell Programming and Scripting

Count files every hour

Hi, I have a directory which uploads files every minute.I want to check the number of files uploaded for every one hour. I want to run a cron every hour to chk this but iam hanged like how to write the script. Like the script should count for one hr ie from 00:00 to 01:00 hrs then next hr... (3 Replies)
Discussion started by: nessj
3 Replies

3. Shell Programming and Scripting

Parse file from remote server to calculate count of string existence in that file

Hi I need to parse the file of same name which exist on different servers and calculate the count of string existed in both files. Say a file abc.log exist on 2 servers. I want to search for string "test" on both files and calculate the total count of search string's existence. For... (6 Replies)
Discussion started by: poweroflinux
6 Replies

4. UNIX for Dummies Questions & Answers

Rotate logs every 1 hour

Hello All, I am learning unix and basically I want to rotate one of my application logs every 1 hour. I need to rotate that file every one hour. I looked in the forums and googled.. but couldn;t get proper information. Requesting you all to kindly guide me. Our application is running on... (4 Replies)
Discussion started by: arunpvp
4 Replies

5. Shell Programming and Scripting

script to grep latest outofmemory string from the logs

I have requirement to prepare script which will grep for latest outofmemory message from the logs. I have used following command to grep the string from the logs,this script is not effective when logs are not getting updated as it will grep for old message. f=catalina.out var=`tail -10 $f |... (17 Replies)
Discussion started by: coolguyamy
17 Replies

6. Shell Programming and Scripting

Calculate total of log by hour

Hi, Just wondering, is there anyway I can get the total of logs generated by hours ? Let say I have these logs, Sep 23 04:48:43 hsbcufs: NOTICE: realloccg /: file system full Sep 23 04:48:47 hsbcufs: NOTICE: alloc: /: file system full Sep 23 04:48:51 hsbcufs: NOTICE: realloccg /: file... (14 Replies)
Discussion started by: dehetoxic
14 Replies

7. Shell Programming and Scripting

Finding/Grep on files with date and hour in the file name

Hi, I have a folder structure as follows, DATA -> 2012-01-01 -> 00 -> ABC_2012-01-03_00.txt -> 01 -> ABC_2012-01-03_01.txt -> 02 -> ABC_2012-01-03_02.txt ... -> 23 -> ABC_2012-01-03_02.txt -> 2012-01-02 -> 2012-01-03 So the dir DATA contains the above hierarchy, User input Start and... (6 Replies)
Discussion started by: mihirvora16
6 Replies

8. UNIX for Advanced & Expert Users

Finding/Grep on files with date and hour in the file name

Hi, I have a folder structure as follows, DATA -> 2012-01-01 -> 00 -> ABC_2012-01-03_00.txt -> 01 -> ABC_2012-01-03_01.txt -> 02 -> ABC_2012-01-03_02.txt ... -> 23 -> ABC_2012-01-03_02.txt -> 2012-01-02 ... (1 Reply)
Discussion started by: mihirvora16
1 Replies

9. Shell Programming and Scripting

Search string or words in logs without using Grep

I'm in need of some kind of script that will search for a string in each logfile in a directory but we don't want to use GREP. GREP seems to use up to much of our memory causing the server to use up a lot of swap space. Our log files are bigger than 500M on a daily basis. We lately started... (8 Replies)
Discussion started by: senormarquez
8 Replies

10. Shell Programming and Scripting

Grep a string and count following lines starting with another string

I have a large dataset with following structure; C 0001 Carbon D SAR001 methane D SAR002 ethane D SAR003 propane D SAR004 butane D SAR005 pentane C 0002 Hydrogen C 0003 Nitrogen C 0004 Oxygen D SAR011 ozone D SAR012 super oxide C 0005 Sulphur D SAR013... (3 Replies)
Discussion started by: Syeda Sumayya
3 Replies
PREG_MATCH(3)								 1							     PREG_MATCH(3)

preg_match - Perform a regular expression match

SYNOPSIS
int preg_match (string $pattern, string $subject, [array &$matches], [int $flags], [int $offset]) DESCRIPTION
Searches $subject for a match to the regular expression given in $pattern. PARAMETERS
o $pattern - The pattern to search for, as a string. o $subject - The input string. o $matches - If $matches is provided, then it is filled with the results of search. $matches[0] will contain the text that matched the full pattern, $matches[1] will have the text that matched the first captured parenthesized subpattern, and so on. o $flags -$flags can be the following flag: o PREG_OFFSET_CAPTURE - If this flag is passed, for every occurring match the appendant string offset will also be returned. Note that this changes the value of $matches into an array where every element is an array consisting of the matched string at offset 0 and its string offset into $subject at offset 1. o $offset - Normally, the search starts from the beginning of the subject string. The optional parameter $offset can be used to specify the alternate place from which to start the search (in bytes). Note Using $offset is not equivalent to passing substr($subject, $offset) to preg_match(3) in place of the subject string, because $pattern can contain assertions such as ^, $ or (?<=x). Compare: <?php $subject = "abcdef"; $pattern = '/^def/'; preg_match($pattern, $subject, $matches, PREG_OFFSET_CAPTURE, 3); print_r($matches); ?> The above example will output: Array ( ) while this example <?php $subject = "abcdef"; $pattern = '/^def/'; preg_match($pattern, substr($subject,3), $matches, PREG_OFFSET_CAPTURE); print_r($matches); ?> will produce Array ( [0] => Array ( [0] => def [1] => 0 ) ) RETURN VALUES
preg_match(3) returns 1 if the $pattern matches given $subject, 0 if it does not, or FALSE if an error occurred. Warning This function may return Boolean FALSE, but may also return a non-Boolean value which evaluates to FALSE. Please read the section on Booleans for more information. Use the === operator for testing the return value of this function. CHANGELOG
+--------+---------------------------------------------------+ |Version | | | | | | | Description | | | | +--------+---------------------------------------------------+ | 5.3.6 | | | | | | | Returns FALSE if $offset is higher than $subject | | | length. | | | | | 5.2.2 | | | | | | | Named subpatterns now accept the syntax | | | (?<name>) and (?'name') as well as (?P<name>). | | | Previous versions accepted only (?P<name>). | | | | +--------+---------------------------------------------------+ EXAMPLES
Example #1 Find the string of text "php" <?php // The "i" after the pattern delimiter indicates a case-insensitive search if (preg_match("/php/i", "PHP is the web scripting language of choice.")) { echo "A match was found."; } else { echo "A match was not found."; } ?> Example #2 Find the word "web" <?php /* The  in the pattern indicates a word boundary, so only the distinct * word "web" is matched, and not a word partial like "webbing" or "cobweb" */ if (preg_match("/web/i", "PHP is the web scripting language of choice.")) { echo "A match was found."; } else { echo "A match was not found."; } if (preg_match("/web/i", "PHP is the website scripting language of choice.")) { echo "A match was found."; } else { echo "A match was not found."; } ?> Example #3 Getting the domain name out of a URL <?php // get host name from URL preg_match('@^(?:http://)?([^/]+)@i', "http://www.php.net/index.html", $matches); $host = $matches[1]; // get last two segments of host name preg_match('/[^.]+.[^.]+$/', $host, $matches); echo "domain name is: {$matches[0]} "; ?> The above example will output: domain name is: php.net Example #4 Using named subpattern <?php $str = 'foobar: 2008'; preg_match('/(?P<name>w+): (?P<digit>d+)/', $str, $matches); /* This also works in PHP 5.2.2 (PCRE 7.0) and later, however * the above form is recommended for backwards compatibility */ // preg_match('/(?<name>w+): (?<digit>d+)/', $str, $matches); print_r($matches); ?> The above example will output: Array ( [0] => foobar: 2008 [name] => foobar [1] => foobar [digit] => 2008 [2] => 2008 ) NOTES
Tip Do not use preg_match(3) if you only want to check if one string is contained in another string. Use strpos(3) or strstr(3) instead as they will be faster. SEE ALSO
PCRE Patterns, preg_quote(3), preg_match_all(3), preg_replace(3), preg_split(3), preg_last_error(3). PHP Documentation Group PREG_MATCH(3)
All times are GMT -4. The time now is 02:30 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy