Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

UNIX shell script to search a string in a file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting UNIX shell script to search a string in a file
# 8  
Old 06-04-2014
Dear pilnet101,

I'd worry that your code will be extremely heavy processing when confronted with a large input file. You will spawn processes for every record. My way will not and should run faster. Of course, a single awk to process all would be better still, but I do not have that knowledge.



Robin
Last edited by rbatte1; 06-04-2014 at 09:03 AM.. Reason: Grammar
This User Gave Thanks to rbatte1 For This Post:
pilnet101
# 9  
Old 06-04-2014
Thanks for the feedback rbatte1.

I do agree, parameter expansion is much more efficient. I have updated my last comment to advise OP accordingly.
This User Gave Thanks to pilnet101 For This Post:
rbatte1
# 10  
Old 06-04-2014
heyyy

---------- Post updated at 07:26 AM ---------- Previous update was at 07:22 AM ----------

Thank you very much pilnet
script is working as expected.
but I want keep all access and error logs their respective cname folders
could you please suggest to how to achieve ?

Regards,
Reddy.
Image
# 11  
Old 06-04-2014
What determines the cname? Can you supply your expected output and the required file names from this or a slightly larger sample?



Robin
# 12  
Old 06-05-2014
Line 6: :: invalid character in expression - +http://www.google.com/bot.html)"
Hi,
Thanks for script but not working below log
Code:
66.249.75.49 - - [15/May/2014:00:12:01 +0000] "GET http://abc.def.com/80DF9D/plantronics/us/support/software-downloads/download.jsp HTTP/1.1" 200 3956 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"
127.0.0.1 - - [15/May/2014:00:12:02 +0000] "GET http://abc.def.com/80DF9D/plantronics/images/icons/forms/icon-field-required.gif HTTP/1.1" 200 486 "-" "Serf/1.1.0 mod_pagespeed/1.6.29.7-6576" "-"
127.0.0.1 - - [15/May/2014:00:12:01 +0000] "GET http://abc.def.com/80DF9D/plantronics/css/global/forms.css HTTP/1.1" 200 8183 "-" "Serf/1.1.0 mod_pagespeed/1.6.29.7-6576" "-"
127.0.0.1 - - [15/May/2014:00:12:01 +0000] "GET http://abc.def.com/80DF9D/plantronics/js/validation/validationFormDownloads.js HTTP/1.1" 200 1126 "-" "Serf/1.1.0 mod_pagespeed/1.6.29.7-6576" "-"
66.249.75.49 - - [15/May/2014:00:12:01 +0000] "GET http://abc.def.com/80DF9D/plantronics/us/support/software-downloads/download.jsp HTTP/1.1" 200 3561 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"

Please help me on this

Script below

Code:
#!/bin/ksh
while read -r line
do
cname=$(echo ${line} | awk -F"[ ]" '{print $10}')
scode=$(echo ${line} | awk -F"[ ]" '{print $(NF-1)}')
[[ ( ${scode} -ge 200 ) && ( ${scode} -le 399 ) ]] && {
echo ${line} >> ${cname}_access.log
}
[[ ( ${scode} -ge 400 ) && ( ${scode} -le 599 ) ]] && {
echo ${line} >> ${cname}_error.log
}
done < /home/vizion/Desktop/adn_DF9D_20140515_0001.log

getting Error "
Code:
line 6: :: invalid character in expression - +http://www.google.com/bot.html)

"
Last edited by Scrutinizer; 06-05-2014 at 05:17 AM.. Reason: CODE tags
# 13  
Old 06-05-2014
Whether this is your expected output ?

Code:
$ cat log.awk
     {
	  split($7,X,/\/|\/\//)
          cname=X[2]
	  scode=$9
	  file = scode>=200 && scode<=399 ?  cname"_access.log": \
	         scode>=400 && scode<=599 ?  cname"_error.log" : \
	  	 cname"_other.log"
	  if(!(file in f))
	       {
		  print >file
		  f[file]
	       } 
	   else{
		  print >>file
               }
          close(file)
     }

Code:
$ cat file1
127.0.0.1 - - [03/Jun/2014:11:50:15 +0530] "GET /common/support/resources/faqs.jsp?category=All&vfurl=%2FknowledgeSearch&t=All&c=All&k=.Audio\x99+310 HTTP/1.0" 404 1048
127.0.0.1 - - [03/Jun/2014:11:50:41 +0530] "GET /configurator/ HTTP/1.0" 404 988
127.0.0.1 - - [03/Jun/2014:11:51:06 +0530] "GET /common/support/resources/faqs.jsp?category=All&vfurl=%2FknowledgeSearch&t=All&c=All&k=.Audio+655+DSP HTTP/1.0" 404 1048
127.0.0.1 - - [03/Jun/2014:11:51:11 +0530] "GET /uk/solutions/home-based-worker/work-shifting.pdf HTTP/1.0" 404 1093
127.0.0.1 - - [03/Jun/2014:11:51:25 +0530] "GET /common/support/resources/faqs.jsp?category=All&vfurl=%2FknowledgeSearch&t=All&c=All&k=ML18 HTTP/1.0" 404 1048
127.0.0.1 - - [03/Jun/2014:11:52:06 +0530] "GET /pl/solutions/successful-collaboration/ HTTP/1.0" 404 1063

Code:
$ awk -f log.awk file1

Code:
$ ls *.log -1
common_error.log
configurator_error.log
pl_error.log
uk_error.log

Code:
$ for i in *.log; do printf "%s\n\n" $i; cat $i; printf "\n";  done
common_error.log

127.0.0.1 - - [03/Jun/2014:11:50:15 +0530] "GET /common/support/resources/faqs.jsp?category=All&vfurl=%2FknowledgeSearch&t=All&c=All&k=.Audio\x99+310 HTTP/1.0" 404 1048
127.0.0.1 - - [03/Jun/2014:11:51:06 +0530] "GET /common/support/resources/faqs.jsp?category=All&vfurl=%2FknowledgeSearch&t=All&c=All&k=.Audio+655+DSP HTTP/1.0" 404 1048
127.0.0.1 - - [03/Jun/2014:11:51:25 +0530] "GET /common/support/resources/faqs.jsp?category=All&vfurl=%2FknowledgeSearch&t=All&c=All&k=ML18 HTTP/1.0" 404 1048

configurator_error.log

127.0.0.1 - - [03/Jun/2014:11:50:41 +0530] "GET /configurator/ HTTP/1.0" 404 988

pl_error.log

127.0.0.1 - - [03/Jun/2014:11:52:06 +0530] "GET /pl/solutions/successful-collaboration/ HTTP/1.0" 404 1063

uk_error.log

127.0.0.1 - - [03/Jun/2014:11:51:11 +0530] "GET /uk/solutions/home-based-worker/work-shifting.pdf HTTP/1.0" 404 1093

Code:
$ cat file2
66.249.75.49 - - [15/May/2014:00:12:01 +0000] "GET http://abc.def.com/80DF9D/plantronics/us/support/software-downloads/download.jsp HTTP/1.1" 200 3956 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"
127.0.0.1 - - [15/May/2014:00:12:02 +0000] "GET http://abc.def.com/80DF9D/plantronics/images/icons/forms/icon-field-required.gif HTTP/1.1" 200 486 "-" "Serf/1.1.0 mod_pagespeed/1.6.29.7-6576" "-"
127.0.0.1 - - [15/May/2014:00:12:01 +0000] "GET http://abc.def.com/80DF9D/plantronics/css/global/forms.css HTTP/1.1" 200 8183 "-" "Serf/1.1.0 mod_pagespeed/1.6.29.7-6576" "-"
127.0.0.1 - - [15/May/2014:00:12:01 +0000] "GET http://abc.def.com/80DF9D/plantronics/js/validation/validationFormDownloads.js HTTP/1.1" 200 1126 "-" "Serf/1.1.0 mod_pagespeed/1.6.29.7-6576" "-"
66.249.75.49 - - [15/May/2014:00:12:01 +0000] "GET http://abc.def.com/80DF9D/plantronics/us/support/software-downloads/download.jsp HTTP/1.1" 200 3561 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"

Code:
$ awk -f log.awk file2

Code:
$ ls *.log -1
abc.def.com_access.log

Code:
$ for i in *.log; do printf "%s\n\n" $i; cat $i; printf "\n";  done
abc.def.com_access.log

66.249.75.49 - - [15/May/2014:00:12:01 +0000] "GET http://abc.def.com/80DF9D/plantronics/us/support/software-downloads/download.jsp HTTP/1.1" 200 3956 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"
127.0.0.1 - - [15/May/2014:00:12:02 +0000] "GET http://abc.def.com/80DF9D/plantronics/images/icons/forms/icon-field-required.gif HTTP/1.1" 200 486 "-" "Serf/1.1.0 mod_pagespeed/1.6.29.7-6576" "-"
127.0.0.1 - - [15/May/2014:00:12:01 +0000] "GET http://abc.def.com/80DF9D/plantronics/css/global/forms.css HTTP/1.1" 200 8183 "-" "Serf/1.1.0 mod_pagespeed/1.6.29.7-6576" "-"
127.0.0.1 - - [15/May/2014:00:12:01 +0000] "GET http://abc.def.com/80DF9D/plantronics/js/validation/validationFormDownloads.js HTTP/1.1" 200 1126 "-" "Serf/1.1.0 mod_pagespeed/1.6.29.7-6576" "-"
66.249.75.49 - - [15/May/2014:00:12:01 +0000] "GET http://abc.def.com/80DF9D/plantronics/us/support/software-downloads/download.jsp HTTP/1.1" 200 3561 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-"

Last edited by Akshay Hegde; 06-05-2014 at 02:58 PM..
# 14  
Old 06-06-2014
Now am getting which is expecting using below script

Code:
 #!/bin/ksh
while read -r line
do
        cname=$(echo ${line} | awk '{split($7,c,"/"); print c[3]}')
        scode=$(echo ${line} | awk -F"[ ]" '{print $9}')
        [[ ( ${scode} -ge 200 ) && ( ${scode} -le 399 ) ]] && {
                echo ${line} >> ${cname}_access.log
                }
        [[ ( ${scode} -ge 400 ) && ( ${scode} -le 599 ) ]] && {
                echo ${line} >> ${cname}_error.log
                }
done < /home/vizion/Desktop/adn_DF9D_20140515_0001.log


Thanks to all of who are helped in this
Last edited by Don Cragun; 06-06-2014 at 07:16 AM.. Reason: Add CODE tags.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Search partial string in a file and replace the string - UNIX

I have the below string which i need to compare with a file and replace this string in the file which matches closely. Can anyone help me on this. string(Scenario 1)- user::r--,user::ourfrd:r-- String(Scenario 2)- user::r-- File **** # file: /local/Desktop/myfile # owner: me # group:... (6 Replies)
Discussion started by: sarathy_a35
6 Replies

2. Shell Programming and Scripting

How to Search a string in a file in shell script?

I have a text file which is generated when the batch job is run. This batch may take few mins to run. When completed, the last line of the text file would be process completed. I need a shell script which will wait for this file till the process completed is printed in it, once found, it would move... (2 Replies)
Discussion started by: Lalat
2 Replies

3. Shell Programming and Scripting

Linux shell script, search by an input string

So, there is a large file where I have to conduct several search using bash shell scripting. The file is like this: TITLE and AUTHOR ETEXT NO. Aspects of plant life; with special reference to the British flora, 56900 by Robert Lloyd... (1 Reply)
Discussion started by: Philia
1 Replies

4. Shell Programming and Scripting

Shell script to search all files for every string in another file

Hello All I have a pattern.txt file in source directory ((/project/source/) in linux server and data looks like: 123abc17 234cdf19 235ifg20 I have multiple log files in log directory (/project/log/) in linux server and data for one log file looks like: <?xml version="1.0" processid... (11 Replies)
Discussion started by: pred55
11 Replies

5. Shell Programming and Scripting

UNIX Scripting help to input string and search a file to find

Hi Don, this is not homework question. I work for a Credit card company and my development goal this year is to learn Unix. I would love if others can help me get started, thanks. Hi everyone I am new to Unix and need help writing a script that can ask user for an input, then search that input... (2 Replies)
Discussion started by: 12ic11
2 Replies

6. Shell Programming and Scripting

UNIX Scripting help to input string and search a file to find

Hi everyone, I am new to Unix and need help writing a script that can ask user for an input, then search that input within a file I know will have to use the read and grep commands, anyone can give me somewhere to start would help Task: Write a script to display which volume pool a given... (1 Reply)
Discussion started by: 12ic11
1 Replies

7. UNIX for Dummies Questions & Answers

UNIX Scripting help to input string and search a file to find

Hi everyone, I am new to Unix and need help writing a script that can ask user for an input, then search that input within a file I know will have to use the read and grep commands, anyone can give me somewhere to start would help Task: Write a script to display... (1 Reply)
Discussion started by: 12ic11
1 Replies

8. UNIX for Dummies Questions & Answers

Unix search a string in the text file

File name : Sample.txt Actually i would like to read <schema>Oracle<schema> string from input file and return only once database as my output. Please advise me. Moved to appropriate forum. (1 Reply)
Discussion started by: balajikalai
1 Replies

9. Shell Programming and Scripting

Shell Script to Search for a particular String and copy the timestamp to a variable

Hi, We Perfrom Loads to the database through a Perl script which generates a statistics file. I need to read the statistics. the Statistics file looks something like below: Process Beginning - 08-26-2010-23.41.47 DB2 CONNECTION SUCCESSFUL! Ready to process and load file: FILENAME # of... (2 Replies)
Discussion started by: Praveenkulkarni
2 Replies

10. Shell Programming and Scripting

shell script to search a string and delete the content

Hi, I've a shell script e.g. #!/bin/bash echo "Enter the next hop id" read nhid echo "enter the IP address" read IP echo "enter the interface name" read name echo "enter the enable/disable state" read state exit 0 now from this script i want to search strings in another (.cam) ... (6 Replies)
Discussion started by: vic_mnnit
6 Replies
Login or Register to Ask a Question