Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Shell script to search all entries from 1 file to all other separated files.

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Shell script to search all entries from 1 file to all other separated files.
# 1  
Old 08-26-2013
Shell script to search all entries from 1 file to all other separated files.
Hi,

I am trying to create a shell script in unix platform, hence will need info on how to start and any ideas from you guys. million thx.

Objective: Shell script to search all entries from 1 file(a.out) to all other files and extract the search output to 1 file (c.out).

Situation,

1) a.out file contains item that i need to search.
Code:
more a.out
mailadm@tm.net.my
tm_network@tm.net.my
tmm_cic@tm.net.my
webmaster@tm.net.my
tmcops@tm.net.my
dnsteam@tm.net.my
tm_quake@tm.net.my
tm_billing@tm.net.my
peering@tm.net.my
tm_dco@tm.net.my

2) list of other files ( need to search all content from a.out file)
Code:
ls -al|grep mail.log
-rw-r--r--   1 mailsrv  mail     10431447 May  3  2011 mail.log.01052011.gz
-rw-r--r--   1 mailsrv  mail     24449972 Jun  3  2011 mail.log.01062011.gz
-rw-r--r--   1 mailsrv  mail     23294154 Jul  3  2011 mail.log.01072011.gz
-rw-r--r--   1 mailsrv  mail     9886937 Jul  3  2012 mail.log.01072012.gz
-rw-r--r--   1 mailsrv  mail     19758797 Aug  3  2011 mail.log.01082011.gz
-rw-r--r--   1 mailsrv  mail     21122727 Aug  3 00:31 mail.log.01082013.gz
-rw-r--r--   1 mailsrv  mail     12600862 Sep  3  2011 mail.log.01092011.gz
-rw-r--r--   1 mailsrv  mail     21982491 Nov  3  2011 mail.log.01112011.gz
-rw-r--r--   1 mailsrv  mail     13511183 May  4  2011 mail.log.02052011.gz
-rw-r--r--   1 mailsrv  mail     12473063 Jul  4  2011 mail.log.02072011.gz
-rw-r--r--   1 mailsrv  mail     19321867 Jul  4  2012 mail.log.02072012.gz
-rw-r--r--   1 mailsrv  mail     22605484 Aug  4  2011 mail.log.02082011.gz
-rw-r--r--   1 mailsrv  mail     18831478 Aug  4 00:32 mail.log.02082013.gz
-rw-r--r--   1 mailsrv  mail     15772747 Sep  4  2011 mail.log.02092011.gz
-rw-r--r--   1 mailsrv  mail     22903663 May  5  2011 mail.log.03052011.gz
.
.
.
-rw-r--r--   1 mailsrv  mail     17611225 Aug 31  2011 mail.log.29082011.gz
-rw-r--r--   1 mailsrv  mail     17393355 Feb  1  2012 mail.log.30012012.gz
-rw-r--r--   1 mailsrv  mail     25865466 Jul  2  2011 mail.log.30062011.gz
-rw-r--r--   1 mailsrv  mail     13508155 Aug  1  2011 mail.log.30072011.gz
-rw-r--r--   1 mailsrv  mail     19499004 Aug  1 00:31 mail.log.30072013.gz
-rw-r--r--   1 mailsrv  mail     12118182 Sep  1  2011 mail.log.30082011.gz
-rw-r--r--   1 mailsrv  mail     9555015 Aug  2  2011 mail.log.31072011.gz
-rw-r--r--   1 mailsrv  mail     20240410 Aug  2 00:32 mail.log.31072013.gz
-rw-r--r--   1 mailsrv  mail     12667014 Sep  2  2011 mail.log.31082011.gz
-rw-r--r--   1 mailsrv  mail     189438124 Nov  2  2012 mail.log.31102012
-rw-------   1 mailsrv  mail     76248835 Aug 26 12:43 mail.log_current


output should be something like this,
file should be name as c.out
Code:
more c.out
26-Aug-2013 09:03:11.10 19c0.2c507b.9406946 tcp_fromvirus reprocess    E 90 prvs=3950219457=mohamadzarifi@tm.net.my rfc822;mailadm@tm.net.my mailadm@tm.net.my <01af01cea1f8$069b1070$13d13150$@tm.net.my> mailsrv pp-inbound.tm.net.my ([192.168.10.91])
26-Aug-2013 09:03:11.99 68ff.1.28 reprocess                 D 90 prvs=3950219457=mohamadzarifi@tm.net.my rfc822;mailadm@tm.net.my mailadm@tm.net.my <01af01cea1f8$069b1070$13d13150$@tm.net.my> mailsrv
26-Aug-2013 09:32:34.81 19c0.2c61ff.9421218 tcp_fromvirus reprocess    E 36 noradila@tm.com.my rfc822;mailadm@tm.net.my mailadm@tm.net.my <58AF33FACEC1B7409319333B74FA8E2B05F413C5@SMSGVS32.tm.my> mailsrv pp-inbound.tm.net.my ([192.168.10.85])
26-Aug-2013 09:32:35.48 68ff.1.112 reprocess                 D 37 noradila@tm.com.my rfc822;mailadm@tm.net.my mailadm@tm.net.my <58AF33FACEC1B7409319333B74FA8E2B05F413C5@SMSGVS32.tm.my> mailsrv
26-Aug-2013 11:04:51.87 19c0.2c9fab.9473076 tcp_fromvirus reprocess    E 40 nurhanani@tm.com.my rfc822;mailadm@tm.net.my mailadm@tm.net.my <02F70B247ED17F48B8889E578A3F066008300942@SMSGVS33.tm.my> mailsrv pp-inbound.tm.net.my ([192.168.10.91])
26-Aug-2013 11:04:52.42 68ff.1.508 reprocess                 D 40 nurhanani@tm.com.my rfc822;mailadm@tm.net.my mailadm@tm.net.my <02F70B247ED17F48B8889E578A3F066008300942@SMSGVS33.tm.my> mailsrv
26-Aug-2013 11:30:50.55 19c0.2cb274.9488313 tcp_fromvirus reprocess    E 6 prvs=3950b1d694=tmcops@tm.net.my rfc822;mailadm@tm.net.my mailadm@tm.net.my <8D26E305AED74660975E208051D72F2F@B12562PCTM01> mailsrv pp-inbound.tm.net.my ([192.168.10.84])
26-Aug-2013 11:30:51.11 6db5.1.54 reprocess                 D 6 prvs=3950b1d694=tmcops@tm.net.my rfc822;mailadm@tm.net.my mailadm@tm.net.my <8D26E305AED74660975E208051D72F2F@B12562PCTM01> mailsrv
26-Aug-2013 11:35:00.89 19c0.2cb636.9491404 tcp_fromvirus reprocess    E 19 mdkamarulfadzli@vads.com rfc822;mailadm@tm.net.my mailadm@tm.net.my <D0E896F4E4B10E40B4375570AE94878108B57C6C@SMSGVS21.tm.my> mailsrv pp-inbound.tm.net.my ([192.168.10.89])
26-Aug-2013 11:35:01.20 6db5.1.60 reprocess                 D 20 mdkamarulfadzli@vads.com rfc822;mailadm@tm.net.my mailadm@tm.net.my <D0E896F4E4B10E40B4375570AE94878108B57C6C@SMSGVS21.tm.my> mailsrv
26-Aug-2013 11:48:12.15 19c0.2cbf0e.9498526 tcp_fromvirus reprocess    E 6 nurhanani@tm.com.my rfc822;mailadm@tm.net.my mailadm@tm.net.my <02F70B247ED17F48B8889E578A3F066008300996@SMSGVS33.tm.my> mailsrv pp-inbound.tm.net.my ([192.168.10.85])
26-Aug-2013 11:48:12.53 6db5.1.97 reprocess                 D 6 nurhanani@tm.com.my rfc822;mailadm@tm.net.my mailadm@tm.net.my <02F70B247ED17F48B8889E578A3F066008300996@SMSGVS33.tm.my> mailsrv
26-Aug-2013 12:36:03.61 19c0.2ce165.9528027 tcp_fromvirus reprocess    E 992 prvs=3950303a3a=adlinothman@tm.net.my rfc822;mailadm@tm.net.my mailadm@tm.net.my <521ABDDD.9070701@tm.net.my> mailsrv pp-inbound.tm.net.my ([192.168.10.89])
26-Aug-2013 12:36:05.60 6db5.1.246 reprocess                 D 992 prvs=3950303a3a=adlinothman@tm.net.my rfc822;mailadm@tm.net.my mailadm@tm.net.my <521ABDDD.9070701@tm.net.my> mailsrv

note: original structure of other files that need to be search is similar to the output at c.out
Last edited by Mr_47; 08-26-2013 at 02:40 AM..
# 2  
Old 08-26-2013
Have you tried something like:
Code:
gzcat mail.log.*.gz | cat - mail.log*[!g][!z] | grep -f a.out > c.out

This User Gave Thanks to Scrutinizer For This Post:
Mr_47
# 3  
Old 08-26-2013
What about zgrep?

(untested):
Code:
zgrep -f a.out maillog.*.gz > c.out

# 4  
Old 08-26-2013
yup similar idea but cant use grep -f
Code:
grep: illegal option -- f
Usage: grep -hblcnsviw pattern file . . .

# 5  
Old 08-26-2013
Try /usr/xpg4/bin/grep instead
This User Gave Thanks to Scrutinizer For This Post:
Mr_47
# 6  
Old 08-26-2013
tested to remove -f, its running but result seems not right
Code:
gzcat mail.log.*.gz | cat - mail.log*[!g][!z] | grep a.out > c.out

cat: cannot open mail.log*[!g][!z]
[root] ls
a.out                 c.out                 mail.log.02072012.gz  mail.log.10092008.gz  mail.log.16082013.gz  mail.log_current.gz
[root|mx1.tm.net.my:/jes/SUNWmsgsr/log/zz] more c.out
02-Jul-2012 00:34:43.13 3e31.905d6e.32070899 tcp_fromvirus tcp_lmtpcs   E 25 madabout@vps.madaboutdeal.sg rfc822;Innoipoh@tm.net.my @mstore2.secure.tmnet.lmtp:innoi
poh@lmtpcs-daemon <E1SlN6n-00039U-QG@vps.madaboutdeal.sg> mailsrv pp-inbound.tm.net.my ([192.168.10.91])
02-Jul-2012 00:34:43.54 4c25.75.387 tcp_lmtpcs                D 25 madabout@vps.madaboutdeal.sg rfc822;Innoipoh@tm.net.my @mstore2.secure.tmnet.lmtp:innoipoh@lmtpcs
-daemon <E1SlN6n-00039U-QG@vps.madaboutdeal.sg> mailsrv mstore2.secure.tmnet dns;mstore2.secure.tmnet (TCP|192.168.10.100|42012|192.168.10.150|225) (mail1.secure.tm
net -- Server LMTP [Sun ONE Messaging Server 6.2-8.04 [built Feb 28 2007]]) smtp;250 2.1.5 innoipoh@lmtpcs-daemon and options OK.
02-Jul-2012 02:13:15.61 3e31.907710.32093863 tcp_fromvirus tcp_lmtpcs   E 10 elviracovaultrp93@yahoo.com rfc822;acoutech@tm.net.my @mstore2.secure.tmnet.lmtp:acoute
ch@lmtpcs-daemon <1341166392.79347.androidMobile@web140501.mail.bf1.yahoo.com> mailsrv pp-inbound.tm.net.my ([192.168.10.89])
02-Jul-2012 02:13:15.67 4f50.93.483 tcp_lmtpcs                R 10 elviracovaultrp93@yahoo.com rfc822;acoutech@tm.net.my @mstore2.secure.tmnet.lmtp:acoutech@lmtpcs-
daemon <1341166392.79347.androidMobile@web140501.mail.bf1.yahoo.com> mailsrv mstore2.secure.tmnet dns;mstore2.secure.tmnet (TCP|192.168.10.100|49067|192.168.10.150|
225) (mail1.secure.tmnet -- Server LMTP [Sun ONE Messaging Server 6.2-8.04 [built Feb 28 2007]]) lmtp;550 5.2.2 Delivery failed: Over quota
02-Jul-2012 05:37:04.63 3e31.90971b.32122534 tcp_fromvirus tcp_lmtpcs   E 57 pabounce@gsol.globalsources.com rfc822;acoutech@tm.net.my @mstore2.secure.tmnet.lmtp:ac
outech@lmtpcs-daemon <1146025520.1341178620109.JavaMail.web@turquoise1.globalsources.com> mailsrv pp-inbound.tm.net.my ([192.168.10.90])
02-Jul-2012 05:37:04.72 55e7.0d2.677 tcp_lmtpcs                R 57 pabounce@gsol.globalsources.com rfc822;acoutech@tm.net.my @mstore2.secure.tmnet.lmtp:acoutech@lm
tpcs-daemon <1146025520.1341178620109.JavaMail.web@turquoise1.globalsources.com> mailsrv mstore2.secure.tmnet dns;mstore2.secure.tmnet (TCP|192.168.10.100|58432|192
.168.10.150|225) (mail1.secure.tmnet -- Server LMTP [Sun ONE Messaging Server 6.2-8.04 [built Feb 28 2007]]) lmtp;550 5.2.2 Delivery failed: Over quota
02-Jul-2012 06:00:30.54 3e31.909a28.32125129 tcp_fromvirus tcp_lmtpcs   E 15 email@mail-greatoutdoors.com rfc822;rkslim@streamyx.com @mstore1.secure.tmnet.lmtp:rksl
im%streamyx.com@lmtpcs-daemon <GreatOutdoors_com.6436prxv9z2.fp5f@mail-greatoutdoors.com> mailsrv pp-inbound.tm.net.my ([192.168.10.89])
02-Jul-2012 06:00:30.59 56aa.0b0.562 tcp_lmtpcs                D 15 email@mail-greatoutdoors.com rfc822;rkslim@streamyx.com @mstore1.secure.tmnet.lmtp:rkslim%stream
yx.com@lmtpcs-daemon <GreatOutdoors_com.6436prxv9z2.fp5f@mail-greatoutdoors.com> mailsrv mstore1.secure.tmnet dns;mstore1.secure.tmnet (TCP|192.168.10.100|59372|192
.168.10.146|225) (mail2.secure.tmnet -- Server LMTP [Sun ONE Messaging Server 6.2-8.04 [built Feb 28 2007]]) smtp;250 2.1.5 rkslim%streamyx.com@lmtpcs-daemon and op
tions OK.

---------- Post updated at 03:59 PM ---------- Previous update was at 03:58 PM ----------

Quote:
Originally Posted by MR.bean
What about zgrep?

(untested):
Code:
zgrep -f a.out maillog.*.gz > c.out


Code:
 zgrep -f a.out maillog.*.gz > c.out
ksh: zgrep:  not found

# 7  
Old 08-26-2013
You need -f to do this. /usr/xpg4/bin/grep has a -f option.

In your example there were also:
Code:
-rw-r--r--   1 mailsrv  mail     189438124 Nov  2  2012 mail.log.31102012
-rw-------   1 mailsrv  mail     76248835 Aug 26 12:43 mail.log_current

unzipped files that also needed to be searched, hence the cat construct...

If they are not there, you could simply try:
Code:
gzcat mail.log.*.gz | /usr/xpg4/bin/grep -f a.out > c.out

This User Gave Thanks to Scrutinizer For This Post:
Mr_47
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to Search a string in a file in shell script?

I have a text file which is generated when the batch job is run. This batch may take few mins to run. When completed, the last line of the text file would be process completed. I need a shell script which will wait for this file till the process completed is printed in it, once found, it would move... (2 Replies)
Discussion started by: Lalat
2 Replies

2. Shell Programming and Scripting

Shell script to search all files for every string in another file

Hello All I have a pattern.txt file in source directory ((/project/source/) in linux server and data looks like: 123abc17 234cdf19 235ifg20 I have multiple log files in log directory (/project/log/) in linux server and data for one log file looks like: <?xml version="1.0" processid... (11 Replies)
Discussion started by: pred55
11 Replies

3. Shell Programming and Scripting

Need to build Shell Script to search content of a text file into a folder consist several files

Have to read one file say sourcefile containing several words and having another folder containing several files. Now read the first word of Sourcefile & search it into the folder consisting sevral files, and create another file with result. We hhave to pick the filename of the file in which... (3 Replies)
Discussion started by: mukesh.baranwal
3 Replies

4. Shell Programming and Scripting

Reading comma separated variable into other variables in shell script

Hi, In shell script, I have a variable var = xyz, inn, day, night, calif ....n and I would like to read them in to var1 = xzy, var2 = inn, var3= day, var4 = night....var. probably in a loop. I would like to read the variables until end of the line. Comma is the delimiter and there's no comma at... (3 Replies)
Discussion started by: suryaemlinux
3 Replies

5. Shell Programming and Scripting

shell script to search and copy files

Hello Im new to this forums, I would like some help regarding a script that I need in order to copy some files. Heres the scenario: I need to search several files which have a particular code inside, lets say "test" all of them on different directories. I need to copy all of them on a new... (4 Replies)
Discussion started by: c.watson
4 Replies

6. Shell Programming and Scripting

To search a file for a specific word in a file using shell script

Hi All, I have a sql output file has below. I want to get the values 200000040 and 1055.49 .Can anyone help me to write a shell script to get this. ACCOUNT_NO ------------------------------------------------------------ BILL_NO ... (8 Replies)
Discussion started by: girish.raos
8 Replies

7. Shell Programming and Scripting

reverse ':' separated numbers in a shell script

I want to reverse a the following: 00:11:22:33:44:55 I currently use something like below to pass it as is. But now I want the same script to reverse the above and pass it to ethtool. // psuedo code i=0 skip=0 for m in $@ do if then skip=1 ... (1 Reply)
Discussion started by: bhanu.nani
1 Replies

8. UNIX for Dummies Questions & Answers

Shell script to search for text in a file and copy file

Compete noob question.... I need a script to search through a directory and find files containing text string abcde1234 for example and then copy that file with that text string to another directory help please :eek: (9 Replies)
Discussion started by: imeadows
9 Replies

9. Shell Programming and Scripting

Unix shell script to parse the contents of comma-separated file

Dear All, I have a comma-separated file. 1. The first line of the file(header) should have 4 commas(5 fields). 2. The last line of the file should have 1 comma(2 fields). Pls help me in checking this condition in a shell script. And the number of lines between the first line and last... (11 Replies)
Discussion started by: KrishnaSaran
11 Replies

10. Shell Programming and Scripting

search for the contents in many file and print that file using shell script

hello have a file1 H87I Y788O T347U J23U and file2 J23U U887Y I99U T556U file3 I99O J99T F557J file4 N99I T666U R55Y file5 H87I T347U file6 H77U R556Y E44T file7 Y788O K98U H8I May be using script we can use file1 to search for all the files and have the output H87I file5... (3 Replies)
Discussion started by: cdfd123
3 Replies
Login or Register to Ask a Question