UNIX shell script to search a string in a file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting UNIX shell script to search a string in a file
# 1  
Old 06-04-2014
RedHat UNIX shell script to search a string in a file

Hi folks,

I am new for shell script, I hope somebody to help me to write shell script
My requirement is below steps
1. I have apache access.log i.e located in /var/log/httpd/
Ex.
Code:
127.0.0.1 - - [03/Jun/2014:11:50:15 +0530] "GET /common/support/resources/faqs.jsp?category=All&vfurl=%2FknowledgeSearch&t=All&c=All&k=.Audio\x99+310 HTTP/1.0" 404 1048
127.0.0.1 - - [03/Jun/2014:11:50:41 +0530] "GET /configurator/ HTTP/1.0" 404 988
127.0.0.1 - - [03/Jun/2014:11:51:06 +0530] "GET /common/support/resources/faqs.jsp?category=All&vfurl=%2FknowledgeSearch&t=All&c=All&k=.Audio+655+DSP HTTP/1.0" 404 1048
127.0.0.1 - - [03/Jun/2014:11:51:11 +0530] "GET /uk/solutions/home-based-worker/work-shifting.pdf HTTP/1.0" 404 1093
127.0.0.1 - - [03/Jun/2014:11:51:25 +0530] "GET /common/support/resources/faqs.jsp?category=All&vfurl=%2FknowledgeSearch&t=All&c=All&k=ML18 HTTP/1.0" 404 1048
127.0.0.1 - - [03/Jun/2014:11:52:06 +0530] "GET /pl/solutions/successful-collaboration/ HTTP/1.0" 404 1063

2. Need to be first column is afte GET i.e we treated as name Ex: /common and /pl ...etc
if column match also needs to check Http status code i.e 404 or 200

for example match common and 404 needs to be push to one file we call it error.log
if not above condition remaining all goes to another file

some thing to /pl/, /UK/, /configurarator/ all goes to different files with respective of their cname folders.

really appreciate if help me.

Regards


Moderator's Comments:
Mod Comment Please use code tags next time for your code and data. Thanks

Last edited by vbe; 06-04-2014 at 08:07 AM..
# 2  
Old 06-04-2014
couldn't understand the requirement clearly
Can you please be specific about what needs to go where and what is 'cname'?
# 3  
Old 06-04-2014
Hi Srini
Thanks for your prompt response

Generally we called Cname after GET from access logs
ex : GET /common/

Simply say when common(ex:/common/) and status code(ex. 404) match needs to be send to one file remaining wills goes to another file.

Let me know if any more details.

Regards,
Reddy.
# 4  
Old 06-04-2014
So you are going to end up with numerous error files in the form of:
{cname}_{statuscode}.err

Such as:
common_404.err

Is that what you mean?

---------- Post updated at 05:56 AM ---------- Previous update was at 05:44 AM ----------

If that is what you want, you can try the below code. It will read each line from your log file and insert it into its own error log file accordingly. It will build these error log files in the same directory as where you run the script.

Code:
#!/bin/ksh
while read -r line
do
echo ${line} >> $(echo ${line} | awk -F"[/ ]" '{print $10"_"$(NF-1)".err"}') 
done < logfile 

# 5  
Old 06-04-2014
Hi ,

Thanks for you it seem to be working, but
1) I need to send status code 200 to 399 to all send to access log
2) I need to send status code 400 to 599 send to error log

Please help me on this

Regards,
Reddy
# 6  
Old 06-04-2014
So if the input file is considered as space separated, do you want each message sent to a file based on column 8 (they are all 404 in your example)

You could use less processing with:-
Code:
#!/bin/ksh
while read a b c d e f g h rest
do
   echo "$a $b $c $d $e $f $g $h $rest" >> /path/to/file-$h
done < logfile

If you want to base it on the first bit of column 7 (the pl, uk or common bit) you need another tweak to split up to column:-
Code:
#!/bin/ksh
while read a b c d e f g h rest
do
   ref="${g#/}"          # Trim off leading /
   ref="${ref%%/*}"      # Trim off everything after the first /
   echo "$a $b $c $d $e $f $g $h $rest" >> /path/to/file-$ref
done < logfile

Of course, you could use both if you with, so your output file will become /path/to/file-$ref-$h or whatever.

If you have codes in column 8 that are errors for one file and others for just loggings, you could:-
Code:
#!/bin/ksh
while read a b c d e f g h rest
do
   case $h in
      2??|3??) type=valid ;;
      4??|5??) type=error ;;
      *)       type=other ;;
   esac   
   echo "$a $b $c $d $e $f $g $h $rest" >> /path/to/file-$type
done < logfile

.... or again some combination with other suggestions to build your output name.


I'm not sure I've understood the question, but these are a few options for what I think you are after.

If I've got it wrong, can you post some input and the expected out with the relevant file names you want to generate and I will have another go.




Robin

Last edited by rbatte1; 06-04-2014 at 08:35 AM.. Reason: Just tidying the formatting of comments in the code.
# 7  
Old 06-04-2014
Code:
#!/bin/ksh
while read -r line
do
        cname=$(echo ${line} | awk -F"[/ ]" '{print $10}')
        scode=$(echo ${line} | awk -F"[/ ]" '{print $(NF-1)}')
        [[ ( ${scode} -ge 200 ) && ( ${scode} -le 399 ) ]] && {
                echo ${line} >> ${cname}_access.log
                }
        [[ ( ${scode} -ge 400 ) && ( ${scode} -le 599 ) ]] && { 
                echo ${line} >> ${cname}_error.log
                }
done < logfile

Like that?

PLEASE BE AWARE:
As rbatte1 has stated, this code may be become quite performance heavy when encountered with large files.

Last edited by pilnet101; 06-04-2014 at 09:08 AM..
This User Gave Thanks to pilnet101 For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Search partial string in a file and replace the string - UNIX

I have the below string which i need to compare with a file and replace this string in the file which matches closely. Can anyone help me on this. string(Scenario 1)- user::r--,user::ourfrd:r-- String(Scenario 2)- user::r-- File **** # file: /local/Desktop/myfile # owner: me # group:... (6 Replies)
Discussion started by: sarathy_a35
6 Replies

2. Shell Programming and Scripting

How to Search a string in a file in shell script?

I have a text file which is generated when the batch job is run. This batch may take few mins to run. When completed, the last line of the text file would be process completed. I need a shell script which will wait for this file till the process completed is printed in it, once found, it would move... (2 Replies)
Discussion started by: Lalat
2 Replies

3. Shell Programming and Scripting

Linux shell script, search by an input string

So, there is a large file where I have to conduct several search using bash shell scripting. The file is like this: TITLE and AUTHOR ETEXT NO. Aspects of plant life; with special reference to the British flora, 56900 by Robert Lloyd... (1 Reply)
Discussion started by: Philia
1 Replies

4. Shell Programming and Scripting

Shell script to search all files for every string in another file

Hello All I have a pattern.txt file in source directory ((/project/source/) in linux server and data looks like: 123abc17 234cdf19 235ifg20 I have multiple log files in log directory (/project/log/) in linux server and data for one log file looks like: <?xml version="1.0" processid... (11 Replies)
Discussion started by: pred55
11 Replies

5. Shell Programming and Scripting

UNIX Scripting help to input string and search a file to find

Hi Don, this is not homework question. I work for a Credit card company and my development goal this year is to learn Unix. I would love if others can help me get started, thanks. Hi everyone I am new to Unix and need help writing a script that can ask user for an input, then search that input... (2 Replies)
Discussion started by: 12ic11
2 Replies

6. Shell Programming and Scripting

UNIX Scripting help to input string and search a file to find

Hi everyone, I am new to Unix and need help writing a script that can ask user for an input, then search that input within a file I know will have to use the read and grep commands, anyone can give me somewhere to start would help Task: Write a script to display which volume pool a given... (1 Reply)
Discussion started by: 12ic11
1 Replies

7. UNIX for Dummies Questions & Answers

UNIX Scripting help to input string and search a file to find

Hi everyone, I am new to Unix and need help writing a script that can ask user for an input, then search that input within a file I know will have to use the read and grep commands, anyone can give me somewhere to start would help Task: Write a script to display... (1 Reply)
Discussion started by: 12ic11
1 Replies

8. UNIX for Dummies Questions & Answers

Unix search a string in the text file

File name : Sample.txt Actually i would like to read <schema>Oracle<schema> string from input file and return only once database as my output. Please advise me. Moved to appropriate forum. (1 Reply)
Discussion started by: balajikalai
1 Replies

9. Shell Programming and Scripting

Shell Script to Search for a particular String and copy the timestamp to a variable

Hi, We Perfrom Loads to the database through a Perl script which generates a statistics file. I need to read the statistics. the Statistics file looks something like below: Process Beginning - 08-26-2010-23.41.47 DB2 CONNECTION SUCCESSFUL! Ready to process and load file: FILENAME # of... (2 Replies)
Discussion started by: Praveenkulkarni
2 Replies

10. Shell Programming and Scripting

shell script to search a string and delete the content

Hi, I've a shell script e.g. #!/bin/bash echo "Enter the next hop id" read nhid echo "enter the IP address" read IP echo "enter the interface name" read name echo "enter the enable/disable state" read state exit 0 now from this script i want to search strings in another (.cam) ... (6 Replies)
Discussion started by: vic_mnnit
6 Replies
Login or Register to Ask a Question