Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Grep string from logs of last 1 hour on files of 2 different servers and calculate count

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Grep string from logs of last 1 hour on files of 2 different servers and calculate count
# 1  
Old 09-03-2010
Question Grep string from logs of last 1 hour on files of 2 different servers and calculate count
Hi,
I am trying to grep a particular string from the files of 2 different servers without copying and calculate the total count of its occurence on both files.
File structure is same on both servers and for reference as follows:
Code:
27-Aug-2010 10:04:30,601|919122874903|phtunes_app|1282243292627|NotifySmsReception|DMGenerateLogInterceptor - ExternalTransactionID:SDP-DM-26713018, TransactionStatus:Requested
27-Aug-2010 10:05:30,601|919122874903|phtunes_app|1282243292627|NotifySmsReception|MaskingUnMaskingInterceptor - msisdn before masking 9122874903
27-Aug-2010 16:33:30,627|919122874903|phtunes_app|1282243292627|NotifySmsReception|MaskingUnMaskingInterceptor - msisdn after masking BJ#13340708
27-Aug-2010 16:34:30,637|BJ#13340708|phtunes_app|1282243292627|NotifySmsReception|CP URL:http://172.30.24.52/unitech_sms/unitechsms.php, AppInstanceIdhtunes_app, Keyword:busnews, SID:tel:BJ%2313340708, TransactionID:SDP-DM-26713018



---------- Post updated at 03:11 PM ---------- Previous update was at 03:09 PM ----------

Please consider this example for grep string from logs of last 15 minutes too.

---------- Post updated at 03:41 PM ---------- Previous update was at 03:11 PM ----------

Find the code below which i have made but getting error.
The code checks the last 15 minutes logs and search the string and get the total count.Can anybody correct the following code or provide me the optimized way:
Code:
bin/bash

to=`date +"%d-%b-%Y %T"`
echo $to
let from_in_seconds=`date +%s`-900
from=`date -d @$from_in_seconds +"%d-%b-%Y %T"`
echo $from

shortcodes=( "56882" "58585" "58888" "57575" "57677" );
for shortcode in ${shortcodes[@]}
do
    count=0
    sh_count=`awk '$0>=from && $0<=to' from="$from" to="$to" /opt/bea/ocsg_4.1/logs/customlogs/App_OP.log | grep "ShortCode=tel:${shortcode}" | wc -l`
    count=`expr $count + $sh_count`
    sh_count2=`ssh -n smehan@10.0.0.1 "awk '$0>=from && $0<=to' from=\"$from\" to=\"$to\" /opt/bea/ocsg_4.1/logs/customlogs/App_OP.log | grep \"ShortCode=tel:${shortcode}\" | wc -l"`
    count=`expr $count + $sh_count2`
    echo "${shortcode} : "$count
done

# 2  
Old 09-03-2010
post the errors. also please confirm that the first line of your scripts is
Code:
#!/bin/bash

# 3  
Old 09-05-2010
Question Grep string from logs of last 1 hour on files of 2 different servers and calculate count
Hi Frank,

Yes the first line is #!/bin/bash in code.
Error i am getting is grepstring.sh is not found on line where i am calling command on server 2.


Quote:
Originally Posted by frank_rizzo
post the errors. also please confirm that the first line of your scripts is
Code:
#!/bin/bash

# 4  
Old 09-05-2010
add these lines to the top of the script(but after the first line) and then post the entire output. the command your running remotely is kind of complex to troubleshoot because of the variables and quoting. can you put the logic in a script on the remote side and then just execute the script?

Code:
set -x
set -u

another suggestion - use
Code:
grep -c

instead of
Code:
|wc -l

you could easily remove the grep and place use the logic in awk to streamline even further.
# 5  
Old 09-06-2010
Question Grep string from logs of last 1 hour on files of 2 different servers and calculate count
Hi frank,

Check my output below and suggest the changes that have to be done or more optimized way of doing that:
Code:
+ set -u
++ date '+%d-%b-%Y %T'
+ to='06-Sep-2010 10:46:22'
+ echo 06-Sep-2010 10:46:22
06-Sep-2010 10:46:22
++ date +%s
+ let from_in_seconds=1283750182-900
++ date -d @1283749282 '+%d-%b-%Y %T'
+ from='06-Sep-2010 10:31:22'
+ echo 06-Sep-2010 10:31:22
06-Sep-2010 10:31:22
+ shortcodes=("56882")
+ for shortcode in '${shortcodes[@]}'
+ count=0
++ awk '$0>=from && $0<=to' 'from=06-Sep-2010 10:31:22' 'to=06-Sep-2010 10:46:22' /opt/bea/ocsg_4.1/logs/customlogs/App_OP.log
++ grep -c ShortCode=tel:56882
+ sh_count=0
++ expr 0 + 0
+ count=0
++ ssh -n smehan@172.30.16.225
+ grep -c ShortCode=tel:56882
Pseudo-terminal will not be allocated because stdin is not a terminal.
+ sh_count2=awk
+ '$0>=from && $0<=to' 'from=06-Sep-2010 10:31:22' 'to=06-Sep-2010 10:46:22' /opt/bea/ocsg_4.1/logs/customlogs/App_OP.log
./test_mo_count2: line 17: $0>=from && $0<=to: command not found
0
./test_mo_count2: line 18: sh_count2: unbound variable
+ count=
+ echo '56882 : '
56882 :

Quote:
Originally Posted by frank_rizzo
add these lines to the top of the script(but after the first line) and then post the entire output. the command your running remotely is kind of complex to troubleshoot because of the variables and quoting. can you put the logic in a script on the remote side and then just execute the script?

Code:
set -x
set -u

another suggestion - use
Code:
grep -c

instead of
Code:
|wc -l

you could easily remove the grep and place use the logic in awk to streamline even further.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Grep a string and count following lines starting with another string

I have a large dataset with following structure; C 0001 Carbon D SAR001 methane D SAR002 ethane D SAR003 propane D SAR004 butane D SAR005 pentane C 0002 Hydrogen C 0003 Nitrogen C 0004 Oxygen D SAR011 ozone D SAR012 super oxide C 0005 Sulphur D SAR013... (3 Replies)
Discussion started by: Syeda Sumayya
3 Replies

2. Shell Programming and Scripting

Search string or words in logs without using Grep

I'm in need of some kind of script that will search for a string in each logfile in a directory but we don't want to use GREP. GREP seems to use up to much of our memory causing the server to use up a lot of swap space. Our log files are bigger than 500M on a daily basis. We lately started... (8 Replies)
Discussion started by: senormarquez
8 Replies

3. UNIX for Advanced & Expert Users

Finding/Grep on files with date and hour in the file name

Hi, I have a folder structure as follows, DATA -> 2012-01-01 -> 00 -> ABC_2012-01-03_00.txt -> 01 -> ABC_2012-01-03_01.txt -> 02 -> ABC_2012-01-03_02.txt ... -> 23 -> ABC_2012-01-03_02.txt -> 2012-01-02 ... (1 Reply)
Discussion started by: mihirvora16
1 Replies

4. Shell Programming and Scripting

Finding/Grep on files with date and hour in the file name

Hi, I have a folder structure as follows, DATA -> 2012-01-01 -> 00 -> ABC_2012-01-03_00.txt -> 01 -> ABC_2012-01-03_01.txt -> 02 -> ABC_2012-01-03_02.txt ... -> 23 -> ABC_2012-01-03_02.txt -> 2012-01-02 -> 2012-01-03 So the dir DATA contains the above hierarchy, User input Start and... (6 Replies)
Discussion started by: mihirvora16
6 Replies

5. Shell Programming and Scripting

Calculate total of log by hour

Hi, Just wondering, is there anyway I can get the total of logs generated by hours ? Let say I have these logs, Sep 23 04:48:43 hsbcufs: NOTICE: realloccg /: file system full Sep 23 04:48:47 hsbcufs: NOTICE: alloc: /: file system full Sep 23 04:48:51 hsbcufs: NOTICE: realloccg /: file... (14 Replies)
Discussion started by: dehetoxic
14 Replies

6. Shell Programming and Scripting

script to grep latest outofmemory string from the logs

I have requirement to prepare script which will grep for latest outofmemory message from the logs. I have used following command to grep the string from the logs,this script is not effective when logs are not getting updated as it will grep for old message. f=catalina.out var=`tail -10 $f |... (17 Replies)
Discussion started by: coolguyamy
17 Replies

7. UNIX for Dummies Questions & Answers

Rotate logs every 1 hour

Hello All, I am learning unix and basically I want to rotate one of my application logs every 1 hour. I need to rotate that file every one hour. I looked in the forums and googled.. but couldn;t get proper information. Requesting you all to kindly guide me. Our application is running on... (4 Replies)
Discussion started by: arunpvp
4 Replies

8. Shell Programming and Scripting

Parse file from remote server to calculate count of string existence in that file

Hi I need to parse the file of same name which exist on different servers and calculate the count of string existed in both files. Say a file abc.log exist on 2 servers. I want to search for string "test" on both files and calculate the total count of search string's existence. For... (6 Replies)
Discussion started by: poweroflinux
6 Replies

9. Shell Programming and Scripting

Count files every hour

Hi, I have a directory which uploads files every minute.I want to check the number of files uploaded for every one hour. I want to run a cron every hour to chk this but iam hanged like how to write the script. Like the script should count for one hr ie from 00:00 to 01:00 hrs then next hr... (3 Replies)
Discussion started by: nessj
3 Replies

10. Shell Programming and Scripting

how to calculate busy hour

Dear All, please tell me any script which can able to calculate the bus hours by claculating the volume for example hour volume 1 100 2 200 3 300 4 7000 5 50 6 80 7 77 8 77 9 165 10 888 11 99 12 89 13 33 14 676 15 878 16 90 17 56 (14 Replies)
Discussion started by: shary
14 Replies
Login or Register to Ask a Question