Parse file from remote server to calculate count of string existence in that file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Parse file from remote server to calculate count of string existence in that file
# 1  
Old 08-30-2010
Question Parse file from remote server to calculate count of string existence in that file

Hi

I need to parse the file of same name which exist on different servers and calculate the count of string existed in both files.

Say a file abc.log exist on 2 servers.
I want to search for string "test" on both files and calculate the total count of search string's existence.

For example if file abc.log on server 1 has string "test" 2 times
and file abc.log on server 2 has string "test" 4 times.

then the output will be
StringName : Count
example
test : 6 times

Note : I have created the password less connectivity using ssh-keygen.

Please help.

Thanks in advance.
# 2  
Old 08-30-2010
You can do something like:

Create a file with a list of the hosts you want to query:
Code:
$> vi hostlist
host1
host2
host3
..

Then take some code:
Code:
PATTERN=mypattern
while read LINE; do
     ssh -n $LINE "grep -c $PATTERN /path/to/file"
done < hostlist |\
awk -v pattern="$PATTERN" '
     {sum+=$1}
     END{
          print "Pattern:",pattern
          print "Sum:",sum
     }
'

# 3  
Old 08-30-2010
Can we do it without using ssh?
# 4  
Old 08-30-2010
You can do that with any remote access tool that is suitable. ssh might be the easiest and still secure.
# 5  
Old 08-30-2010
Actually, I am willing to write a shell script that can fetch all the text in between certain tags (like <span class="someclass">Fetch This Text</span>) from an HTML file located at different web-server.
# 6  
Old 08-30-2010
Maybe you open up a new thread for your request where you can go into details so we don't hijack this thread.
# 7  
Old 08-31-2010
Question Parse file from remote server to calculate count of string existence in that file

Hi zaxxon,

Thanks for the script but
I want to use awk command with ssh in the script mentioned and want to search from the logs of last 15 minute only of the file which includes the data of whole day.

What i have done till now is as follows and executes the code on same server itself which is working fine:
Code:
#!/bin/bash

to=`date +"%d-%b-%Y %T"`
echo $to
let from_in_seconds=`date +%s`-900
from=`date -d @$from_in_seconds +"%d-%b-%Y %T"`
echo $from

shortcodes=( "56882" "58585" "58888" "57575" "57677" );
for shortcode in ${shortcodes[@]}
do
    count=0
    sh_count=`awk '$0>=from && $0<=to' from="$from" to="$to" App_OP.log | grep "ShortCode=tel:${shortcode}" | wc -l`
    count=`expr $count + $sh_count`   
    echo "${shortcode} : "$count
done

In the above code i want to execute the above sh_count command on other server too from the same script itself and get count from the output and add to the total count to get final output.

Please help for the above situation
Thanks in advance.

Moderator's Comments:
Mod Comment Use code tags please.



Quote:
Originally Posted by zaxxon
You can do something like:

Create a file with a list of the hosts you want to query:
Code:
$> vi hostlist
host1
host2
host3
..

Then take some code:
Code:
PATTERN=mypattern
while read LINE; do
     ssh -n $LINE "grep -c $PATTERN /path/to/file"
done < hostlist |\
awk -v pattern="$PATTERN" '
     {sum+=$1}
     END{
          print "Pattern:",pattern
          print "Sum:",sum
     }
'


Last edited by zaxxon; 08-31-2010 at 05:14 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Is it possible to create 10GB file in Linux and transferring the file to other remote server

Hi folks, Is it possible to create 10GB file in linux and transferring the file to other remote server? Regards, J (3 Replies)
Discussion started by: scriptscript
3 Replies

2. Shell Programming and Scripting

Check file for string existence before appending it with string

I want to append file with a string but before doing that i want to check if this string already exist in that file.I tried with grep on Solaris 10 but unsuccessful.Man pages from grep seems to suggest if the string is found command status will be 0 and if not 1.But i am not finding it.May be i... (2 Replies)
Discussion started by: sahil_shine
2 Replies

3. Shell Programming and Scripting

Parse large file on line count (random lines)

I have a file that needs to be parsed into multiple files every time there line contains a number 1. the problem i face is the lines are random and the file size is random. an example is that on line 4, 65, 187, 202 & 209 are number 1's so there has to be file breaks between all those to create 4... (6 Replies)
Discussion started by: darbs121
6 Replies

4. Shell Programming and Scripting

Polling for existence of file on remote host

I am polling a file on remote host. I have this code that works, but can't explain why it works. while user@remote.no-exist.com 'ls /user/app1/.done'` ] do echo Sleeping for 5 secs sleep 5; done This code works in the way that when the .done file exists on the remote host, the script... (1 Reply)
Discussion started by: starlatch
1 Replies

5. Shell Programming and Scripting

Existence of a string in a file

Hi, I want to know whether a string or variable is exists in a perticular file or not. I want to use IF command in cshell. I am not sure how to use it. can any one help me.. (2 Replies)
Discussion started by: arup1980
2 Replies

6. Shell Programming and Scripting

Mapping with series from master file and calculate count

Hi All, My shell script is calculating the count of each shortcode series wise whose sample output is as follows: -------------------------- 56882 9124 1 9172 1 9173 4 8923 6 9175 1 9058 2 7398 2 -------------------------- 58585 series count 9124 1 8858 17 9061 21 9125 21 (10 Replies)
Discussion started by: poweroflinux
10 Replies

7. Shell Programming and Scripting

Grep string from logs of last 1 hour on files of 2 different servers and calculate count

Hi, I am trying to grep a particular string from the files of 2 different servers without copying and calculate the total count of its occurence on both files. File structure is same on both servers and for reference as follows: 27-Aug-2010... (4 Replies)
Discussion started by: poweroflinux
4 Replies

8. Shell Programming and Scripting

check for file existence on remote machine using sftp

Hi all, I am a beginner to shell script.Can any one please help me on the below requirement I need to check whether the file (called 3Com_Files_Delivered.txt) exists on the remote mechaine or not? if so i need to copy all the files from there to my local mechaine.Especially i am... (7 Replies)
Discussion started by: narasimha123
7 Replies

9. Shell Programming and Scripting

Calculate the time difference between a local file and a remote file.

I m stuck with a issue. I need to calculate the time difference between two files.. one on the local machine and one on the remote machine using a script. Can any one suggest the way this can be achevied Thanks, manohar (1 Reply)
Discussion started by: meetmano143
1 Replies

10. Shell Programming and Scripting

Parse String in XML file

Hello All, I am new to this and I need to parse an XML file. Here's the XML Input File: <Report version="1.2"> <summary fatals="0" testcases="1" expected_fails="0" unexpected_passes="0" warnings="9" tests="21" errors="0" fails="1" passes="20" /> <testresult... (4 Replies)
Discussion started by: racbern
4 Replies
Login or Register to Ask a Question