Help Me - How to grep in multiple servers


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Help Me - How to grep in multiple servers
# 1  
Old 06-23-2009
Help Me - How to grep in multiple servers

Hi All,

I need help , Regarding the keyword search in multiple servers at a time . we are desiging a search website . we have a multiple servers and each of the server have 3 instances having Unix compressed files.Our requirement was we need to search the particular key word for eg. we need to search Prasad in all the servers .is there any way to search the keyword in all servers and all the instances at a time .

exact requirement we have the servers located in Mumbai , Bangalore , Chennai , Delhi and hyderabad each of the location like Mumbai have 3 instances with file name as voter.gz . i want to search the keyword Prasad in all servers in instances having voter.gz file . we need to display the servers and instances having name Prasad . we have the username and password for the connecting server. and proxy password also

Could any body please help me to reduce the time complexity of searching the files .

With Regards,
Prasad .
# 2  
Old 06-23-2009
Is the keyword and fileset always the same? You could set up a cron-ob and send the results to a central server.
# 3  
Old 06-23-2009
If your're designing a _real_ search website I humbly think that you have to revise your whole strategy because searching that way is completely inefficient and resource wasteful.

After saying that I can suggest a few possibilities working with what you have.

1) If you want to search from a workstation through all the servers through SSH, firstly I suggest you configure a simple PKI infrastructure (private/public key). With that, you avoid typing the remote server password each time. In the client you need to have running a ssh-agent to login in behaf of your own user.

After thar you could use something like this:

for server in [ mumbai bangalore chennai delhi hyderabad other-hostnames ]; do echo $server; ssh $server zcat /path/to/voter.gz | grep -l "keyword"; done

You have to rewrite the above one-liner to your exact needs.


A quick work around for a better solution should be a script that parses that compressed files extracting all keywords and populating a MySQL (or any other) database. You're searches will be a lot faster and consistence because you have all the info in a single repository.

Ragards,
Leandro.
# 4  
Old 06-23-2009
Thanks Leandro, i already done with SSH setup with private and public key set up. i need the exact command to grep simultaneously in aall the servers

Quote:
Originally Posted by l_vbosch
If your're designing a _real_ search website I humbly think that you have to revise your whole strategy because searching that way is completely inefficient and resource wasteful.

After saying that I can suggest a few possibilities working with what you have.

1) If you want to search from a workstation through all the servers through SSH, firstly I suggest you configure a simple PKI infrastructure (private/public key). With that, you avoid typing the remote server password each time. In the client you need to have running a ssh-agent to login in behaf of your own user.

After thar you could use something like this:

for server in [ mumbai bangalore chennai delhi hyderabad other-hostnames ]; do echo $server; ssh $server zcat /path/to/voter.gz | grep -l "keyword"; done

You have to rewrite the above one-liner to your exact needs.


A quick work around for a better solution should be a script that parses that compressed files extracting all keywords and populating a MySQL (or any other) database. You're searches will be a lot faster and consistence because you have all the info in a single repository.

Ragards,
Leandro.


---------- Post updated at 09:21 PM ---------- Previous update was at 09:11 PM ----------

Thanks a lot Leandro .Actually these are the production servers it having the complete data, currently we don't have the contingency servers . We need to search ts data with out effecting any production environment.
Could you please suggest/help me , the command which is provide to us , by using this command can we create shell script to enter the servers and path in user console ?

With Regards,
Prasad G.
# 5  
Old 06-24-2009
You can try:
Code:
echo server1 server2 server3 ... | 
xargs -P 10 -I % ssh user@% grep "'regex'" remote-file-name

# 6  
Old 06-24-2009
Otheus: a plain grep won't give any results if the file is compressed, that's why I suggested the use of zcat.
# 7  
Old 06-24-2009
oh right. well, given that he's got .gz file, he should expect to find "zgrep" working just fine.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to apply the update statement in multiple servers on multiple dbs at a time .?

Hi , Can any please help the below requirement on all multiple servers and multiple dbs. update configuration set value='yes' ;1) the above statement apply on 31 Databases at a time on different Ip address eg : 10.104.1.12 (unix ip address ) the above ip box contains 4 db's eg : db... (2 Replies)
Discussion started by: venkat918
2 Replies

2. UNIX for Beginners Questions & Answers

ssh multiple servers

Hi folks. I'm pretty new to unix, while I'm learning a lot I'm finding bash scripting quite confusing. Im sure it's not really, my head just hasn't clicked with it. Anyway, I need a script to loop the ip addresses stored in a file and run a "pgrep <process>" and return the pid or some... (2 Replies)
Discussion started by: MuntyScrunt
2 Replies

3. UNIX for Advanced & Expert Users

Help with credentials when using Grep across multiple servers

Hello all, I need some help with a script I have been working on. I was wanting to know if it is possible to add authentication to it for each server it runs across? The credentials are all the same on each server. This is what I am using so far and it seems to work. I am trying to avoid... (3 Replies)
Discussion started by: Smorgen
3 Replies

4. Shell Programming and Scripting

Grep from multiple patterns multiple file multiple output

Hi, I want to grep multiple patterns from multiple files and save to multiple outputs. As of now its outputting all to the same file when I use this command. Input : 108 files to check for 390 patterns to check for. output I need to 108 files with the searched patterns. Xargs -I {} grep... (3 Replies)
Discussion started by: Diya123
3 Replies

5. Shell Programming and Scripting

Checking running process status using "grep" on multiple servers in load sharing system.

Suppose i have 3 different servers say x,y and z. Im running some process say ABC and 40 instances for the same is being created. In load sharing suppose on server x, 20 instances are running server y, 10 instances are running server z, 10 instances are running. While checking the... (1 Reply)
Discussion started by: ankitknit
1 Replies

6. Shell Programming and Scripting

Deploying Cronjob to multiple servers

I figure that this is a shell scripting issue so I'll post here. I'm attempting to deploy a cronjob to a variety of servers, some 200-odd, of all walks of life. Mostly AIX, but some Solaris, some HP-UX, some Linux (of varying distributions), etc. I initially thought about utilizing a mass... (2 Replies)
Discussion started by: Magus Zeal
2 Replies

7. UNIX for Advanced & Expert Users

grep for a line in a file on multiple unix servers

Hi, I need to grep(look) for a line in filename named /tmp/Test on all 90 unix servers. Without logging to each servers manually and grep'ing for the line, is there a way to pull this info in output file thru' the ksh/sh/bash script. Can someone pl. help to create this script for me? Ex.... (3 Replies)
Discussion started by: Mike1234
3 Replies

8. Shell Programming and Scripting

rsh to change multiple ip in multiple servers?

good day. i jsut wanted to know what is the best script or the best way changing a lot of Ip's in all servers. Do you have any idea? im using awk to change IP,what if, you have lots of servers. You need to change it one by one? It will take time to change it manually. (2 Replies)
Discussion started by: kenshinhimura
2 Replies

9. Shell Programming and Scripting

login into multiple servers thru script...

I need to login into multiple servers thru a script run couple commands and run find command as root. I only have ssh access to the servers as a user than I can "su" to root. If you have a similar script please post it. Also if you can suggest commands that I should consider please let me know. ... (1 Reply)
Discussion started by: avcert1998
1 Replies

10. Shell Programming and Scripting

ftp to multiple servers

Hi folks. I am writing a ksh ftp script. The problem is, I need to transfer the files to several different servers. Is there a way to close a connection and move on to the next in one script or do I need to write a separate script for each one? Thanks, kristy (2 Replies)
Discussion started by: kristy
2 Replies
Login or Register to Ask a Question