Mapping with series from master file and calculate count


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Mapping with series from master file and calculate count
# 8  
Old 09-08-2010
You should get the result directly now.

Code:
Ydate=`date -d'-1 days' +'%Y-%m-%d'`
shortcodes=( "56882" "58585" "58888" "57575" "57677" );
for shortcode in ${shortcodes[@]}
do
        echo "--------------------------";
        echo $shortcode;
#        echo "series count";
#        grep "ShortCode=tel:${shortcode}" /opt/bea/ocsg_4.1/logs/customlogs/App_OP.log.$Ydate | awk -F"|" '{ arr[substr($2,1,4)]++ } END { for( no in arr) { print no , arr[no] } }'
        awk -v s=$shortcode -F"|" '$0~"ShortCode=tel:" s {print substr($2,1,4)}' /opt/bea/ocsg_4.1/logs/customlogs/App_OP.log.$Ydate > tmp.log
        awk 'NR==FNR{a[$1]=$2;next}
               { if (a[$1]=="North") North++
                 if (a[$1]=="South") South++}
               END {  print "North", North; print "South", South}
              ' master.txt tmp.log
done

If in Solaris, replace awk to nawk or /usr/xpg4/bin/awk
# 9  
Old 09-08-2010
Bug Mapping with series from master file and calculate count

Hi rdcwayx,

Thanks for the help, its working fine now.
I want your help for one more thing:
I want to check for the given string that is "ShortCode=tel:" in last 15 minutes of running logs.
Can I do that???

Quote:
Originally Posted by rdcwayx
You should get the result directly now.

Code:
Ydate=`date -d'-1 days' +'%Y-%m-%d'`
shortcodes=( "56882" "58585" "58888" "57575" "57677" );
for shortcode in ${shortcodes[@]}
do
        echo "--------------------------";
        echo $shortcode;
#        echo "series count";
#        grep "ShortCode=tel:${shortcode}" /opt/bea/ocsg_4.1/logs/customlogs/App_OP.log.$Ydate | awk -F"|" '{ arr[substr($2,1,4)]++ } END { for( no in arr) { print no , arr[no] } }'
        awk -v s=$shortcode -F"|" '$0~"ShortCode=tel:" s {print substr($2,1,4)}' /opt/bea/ocsg_4.1/logs/customlogs/App_OP.log.$Ydate > tmp.log
        awk 'NR==FNR{a[$1]=$2;next}
               { if (a[$1]=="North") North++
                 if (a[$1]=="South") South++}
               END {  print "North", North; print "South", South}
              ' master.txt tmp.log
done

If in Solaris, replace awk to nawk or /usr/xpg4/bin/awk
# 10  
Old 09-08-2010
from "App_OP.log.$Ydate ", I think your log file is automatically sperated (or we call log rotated) per day.

Code:
time=`date -d '-15 minutes' +%H%M%S`

awk -v t=$time -v s=$shortcode -F"|" 'substr($1,13,8)>t&&$0~"ShortCode=tel:" s {print substr($2,1,4)}' /opt/bea/ocsg_4.1/logs/customlogs/App_OP.log.$TODAY

# 11  
Old 09-08-2010
MySQL Mapping with series from master file and calculate count

Hi rdcwayx,

I got the way from searching the last 15 minutes logs check the code below and tell me if you have more optimized way:
Code:
#!/bin/bash
to=`date +"%d-%b-%Y %T"`
echo $to
let from_in_seconds=`date +%s`-90000
from=`date -d @$from_in_seconds +"%d-%b-%Y %T"`
echo $from
shortcodes=( "52211" );
for shortcode in ${shortcodes[@]}
do
    count=0
    sh_count=`awk '$0>=from && $0<=to && $0~"ShortCode=tel:" s' from="$from" to="$to" s="$shortcode" App_OP.log | wc -l`
    count=`expr $count + $sh_count`
    echo "${shortcode} : "$count
done

Quote:
Originally Posted by rdcwayx
from "App_OP.log.$Ydate ", I think your log file is automatically sperated (or we call log rotated) per day.

Code:
time=`date -d '-15 minutes' +%H%M%S`
 
awk -v t=$time -v s=$shortcode -F"|" 'substr($1,13,8)>t&&$0~"ShortCode=tel:" s {print substr($2,1,4)}' /opt/bea/ocsg_4.1/logs/customlogs/App_OP.log.$TODAY

Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Replacing 12 columns of one file by second file based on mapping in third file

i have a real data prod file with 80+ fields containing 1k -2k records. i have to extract say 12 columns out of this which are sensitive fields along with one primary key say SEQ_ID (like DOB,account no, name, SEQ_ID, govtid etc) in a lookup file. i have to replace these sensitive fields in... (11 Replies)
Discussion started by: megh12
11 Replies

2. UNIX Desktop Questions & Answers

How can I replicate master master and master master MySQL databse replication and HA?

I have an application desigend in PHP and MySQl running on apache web server that I is running on a Amazon EC2 server Centos. I want to implement the master-master and master slave replication and high availability disaster recovery on this application database. For this I have created two... (0 Replies)
Discussion started by: Palak Sharma
0 Replies

3. Shell Programming and Scripting

Search and replace with mapping from a mapper file in a target file

Hello, I have a special problem. I have a file in 8 bit and would like to convert the whole database to 16Bit unicode. The mapping file has the following structure: The mapper is provided as a zip file The target file to be converted contains data in English and 8 bit Urdu mapping, a... (4 Replies)
Discussion started by: gimley
4 Replies

4. Shell Programming and Scripting

Creating unique mapping from multiple mapping

Hello, I do not know if this is the right title to use. I have a large dictionary database which has the following structure: where a b c d e are in English and p q r s t are in a target language., the two separated by the delimiter =. What I am looking for is a perl script which will take... (5 Replies)
Discussion started by: gimley
5 Replies

5. Shell Programming and Scripting

Calculate age of a file | calculate time difference

Hello, I'm trying to create a shell script (#!/bin/sh) which should tell me the age of a file in minutes... I have a process, which delivers me all 15 minutes a new file and I want to have a monitoring script, which sends me an email, if the present file is older than 20 minutes. To do... (10 Replies)
Discussion started by: worm
10 Replies

6. Shell Programming and Scripting

Grep string from logs of last 1 hour on files of 2 different servers and calculate count

Hi, I am trying to grep a particular string from the files of 2 different servers without copying and calculate the total count of its occurence on both files. File structure is same on both servers and for reference as follows: 27-Aug-2010... (4 Replies)
Discussion started by: poweroflinux
4 Replies

7. Shell Programming and Scripting

Parse file from remote server to calculate count of string existence in that file

Hi I need to parse the file of same name which exist on different servers and calculate the count of string existed in both files. Say a file abc.log exist on 2 servers. I want to search for string "test" on both files and calculate the total count of search string's existence. For... (6 Replies)
Discussion started by: poweroflinux
6 Replies

8. Programming

Mapping all file at once or by page

This might be a silly question but I was wondering if, in case of huge files (2-3GB), it is more efficient to map the whole file at once, or to map it page by page. The file has to be processed sequentially from the start to the end. Thanks. (12 Replies)
Discussion started by: emitrax
12 Replies

9. Shell Programming and Scripting

calculate the average of time series data using AWK

Hi, I have two time series data (below) merged into a file. t1 and t2 are in unit of second I want to calculate the average of V1 every second and count how many times "1" in V2 is occur within a second Input File: t1 V1 t2 V2 10.000000... (5 Replies)
Discussion started by: nica
5 Replies
Login or Register to Ask a Question