Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

How to tail sed and awk in one line?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to tail sed and awk in one line?
# 8  
Old 10-13-2016
Hello Don,
Thanks for your help. I understood the point .
Here is the final status:

Code:
#!/bin/bash
while true; do
tail -f /var/log/apache2/access.log | awk '/myword/' /var/log/apache2/access.log | awk '{print $1}' | awk '!a[$0]++' > ip | while \ 
inotifywait -e close_write ip; do ./ban.sh; done
done

I also have a script to block ip and it kicks just in second.

Kind regards
Boris
# 9  
Old 10-14-2016
Hello Boris,
I still don't quite think you get it. Without seeing an example of what is in your log file, I'm just guessing, but try this instead of what you have now:
Code:
#!/bin/bash
while true
do	awk '/myword/ && !a[$1]++ {print $1}' /var/log/apache2/access.log > ip
	./ban.sh
done

The tail in your pipeline isn't doing anything but wasting system resources. I think the awk above does the same thing as the three awk commands in your pipeline. I don't have an inotifywait utility on my system, but it looks like it is waiting for file opened by the awk command to be closed. If that is what it is doing, it makes MUCH more sense to just wait for the awk command to complete instead of sticking more stuff in a pipeline that doesn't belong in a pipeline. Your awk command output is being redirected to a file, so there is nothing for your while loop to read so your while loop should not be in the pipeline.

If the intent is to call your script every time a line of data is written to the file ip by your awk script, that would be something more like:
Code:
#!/bin/bash
while true
do	awk '/myword/ && !a[$1]++ {print $1}' /var/log/apache2/access.log |
	  while IFS= read -r ip
	  do	printf '%s\n' "$ip" >> ip
		./ban.sh
	  done
done

Hope this helps,
Don

---
Note, the missing pipe symbol noted in post #14 in this thread has now been fixed above to avoid confusing anyone else reading this thread.
Last edited by Don Cragun; 10-14-2016 at 06:57 PM.. Reason: Add missing pipe symbol and Note.
# 10  
Old 10-14-2016
Hello Don,

Here is the sample log :

Code:
11.22.33.44 - - [13/Oct/2016:21:51:06 -0400] "GET /mydrive/admin/load.php?&action=get_current&JsHttpRequest=1-xml HTTP/1.1" 200 510 "http://vps_ip:44056/mydrive/?myword" "Mozilla/5.0"

Below code is not printing ip file
Code:
#!/bin/bash
while true
do	awk '/myword/ && !a[$1]++ {print $1}' /var/log/apache2/access.log
	while IFS= read -r ip
	do	printf '%s\n' "$ip" >> ip
		./ban.sh
	done
done

Final working status:
Code:
#!/bin/bash
while true 
do awk '/myword/' /var/log/apache2/access.log | awk '{print $1}' | awk '!a[$0]++' > ip | while inotifywait -e close_write ip; do ./ban.sh; done
	PID=`ps -eaf | grep syncapp | grep -v grep | awk '{print $2}'`
if [[ "" !=  "$PID" ]]; then
  echo "killing $PID"
  kill -9 $PID
fi

done

Terminal Output:
Code:
root@root:~# ./grep3.sh
Setting up watches.
Watches established.
ip CLOSE_WRITE,CLOSE
Setting up watches.
Watches established.

Could you please let me know if I could make it shorter or make system less busy?

Thanks in advance
Boris
Moderator's Comments:
Mod Comment Please use CODE tags when displaying sample input, output, and code segments; not QUOTE tags.
Last edited by Don Cragun; 10-14-2016 at 04:03 PM.. Reason: Change QUOTE tags to CODE tags so we can see line breaks in sample input data.
# 11  
Old 10-14-2016
Hello baris35,
My repeated attempts to explain how pipelines work have clearly failed. You are still stringing together things in a pipeline that are completely unrelated to each other.

You say you now have final working code. It contains what appears to be an infinite loop inside an infinite loop. It contains a ps, grep, awk, awk pipeline and an if statement in the outer loop that will never be executed because the inner loop will never terminate. It runs the command ./ban.sh inside the inner loop, but we have no idea what that does, nor why you would want to run it repeatedly. You have given us no indication of why you need an infinite loop waiting for a file to be opened and closed repeatedly then that file that will only be opened and closed once by the preceding code.

You say that the code I suggested doesn't print the list of IP addresses it found. This is true; it only writes them to a file (the file named ip) just like your earlier code did.

If you're only going to show us code that doesn't work and then complain that our suggestions to help you improve it don't work, there is nothing we can do. If you're willing to write a clear description of what your code is trying to do, explain what your current code is doing correctly, and explain what your current code is failing to do; then we might be able to help you correct your code so it will do what you want it to do.
This User Gave Thanks to Don Cragun For This Post:
baris35
# 12  
Old 10-14-2016
Hello Don,
I mean that it's not creating a file named ip

Thanks for your time and suggestions

Kind regards
Boris
# 13  
Old 10-14-2016
Quote:
Originally Posted by baris35
.
.
.
Below code is not printing ip file
Code:
#!/bin/bash
while true
do	awk '/myword/ && !a[$1]++ {print $1}' /var/log/apache2/access.log
	while IFS= read -r ip
	do	printf '%s\n' "$ip" >> ip
		./ban.sh
	done
done

Of course not. At least, not immediately. ip is not written to by the awk command. The awk prints to screen, the while loop reads from stdin --> you're supposed to enter the IPs found manually (which I doubt is what you really want).

Quote:
Final working status:
Code:
#!/bin/bash
while true 
do awk '/myword/' /var/log/apache2/access.log | awk '{print $1}' | awk '!a[$0]++' > ip | while inotifywait -e close_write ip; do ./ban.sh; done
	PID=`ps -eaf | grep syncapp | grep -v grep | awk '{print $2}'`
if [[ "" !=  "$PID" ]]; then
  echo "killing $PID"
  kill -9 $PID
fi

done

As has been pointed out before, the three awks can be concentrated into one as done in the example before. Redirecting stdout into the file ip PLUS piping it into something (here: while loop) doesn't work - use either. (Ususally, redirection takes precedence)
A while loop checking a closed file is somewhat pointless - either it exists or it does not.
The ban.sh is unknown, and so is grep3.sh, so there is not a snowball's chance in hell for to judge

Quote:
Terminal Output:
Code:
root@root:~# ./grep3.sh
Setting up watches.
Watches established.
ip CLOSE_WRITE,CLOSE
Setting up watches.
Watches established.

.
.
.
Wouldn't it make sense you depict the entire situation (file/directory structure, input samples, scripts used, final action/output desired) so you can get useful help?
These 2 Users Gave Thanks to RudiC For This Post:
baris35 Don Cragun
# 14  
Old 10-14-2016
Quote:
Originally Posted by baris35
Hello Don,
I mean that it's not creating a file named ip

Thanks for your time and suggestions

Kind regards
Boris
Ouch. Yes, the script I suggested:
Code:
#!/bin/bash
while true
do	awk '/myword/ && !a[$1]++ {print $1}' /var/log/apache2/access.log
	while IFS= read -r ip
	do	printf '%s\n' "$ip" >> ip
		./ban.sh
	done
done

was missing a pipe symbol (as pointed out by RudiC). I intended to write:
Code:
#!/bin/bash
while true
do	awk '/myword/ && !a[$1]++ {print $1}' /var/log/apache2/access.log |
	  while IFS= read -r ip
	  do	printf '%s\n' "$ip" >> ip
		./ban.sh
	  done
done

But, as I said before, I have no idea if this is what you want to do since you still have not specified what you are trying to do! And, we have no idea what ./ban.sh does, whether you want to run it each time an IP address is added to the file ip, or if you just want to run it once each time awk processes your apache2 log file.
This User Gave Thanks to Don Cragun For This Post:
baris35
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Multiple line search, replace second line, using awk or sed

All, I appreciate any help you can offer here as this is well beyond my grasp of awk/sed... I have an input file similar to: &LOG &LOG Part: "@DB/TC10000021855/--F" &LOG &LOG &LOG Part: "@DB/TC10000021852/--F" &LOG Cloning_Action: RETAIN &LOG Part: "@DB/TCCP000010713/--A" &LOG &LOG... (5 Replies)
Discussion started by: KarmaPoliceT2
5 Replies

2. Shell Programming and Scripting

sed and awk giving error ./sample.sh: line 13: sed: command not found

Hi, I am running a script sample.sh in bash environment .In the script i am using sed and awk commands which when executed individually from terminal they are getting executed normally but when i give these sed and awk commands in the script it is giving the below errors :- ./sample.sh: line... (12 Replies)
Discussion started by: satishmallidi
12 Replies

3. UNIX for Dummies Questions & Answers

What should be precedence of using awk, sed, head and tail in UNIX?

Hi All, I am new to unix. In this forum some days back, I have read something like below: 1) Do not use perl if awk can do your work. 2) Do not use awk if sed can do your work. . . . I do not re-collect the whole thing. I think it is good to know the precedence of using these... (2 Replies)
Discussion started by: Prathmesh
2 Replies

4. Shell Programming and Scripting

sed or awk to replace a value in a certain line.

I have an input like following. *DEFINE_CURVE_TITLE Force for tool binder $# lcid sidr sfa sfo offa offo dattyp 3 0 1 .000000 125.00000 0.000 0.000 0 $# a1 ... (5 Replies)
Discussion started by: hamnsan
5 Replies

5. UNIX for Advanced & Expert Users

sed one liner simialr to tail command

Can anyone explain the below sed oneliner? sed -e ':a' -e '$q;N;11,$D;ba' It works same as tail command. I just want to know how it works. Thanks (1 Reply)
Discussion started by: pandeesh
1 Replies

6. Shell Programming and Scripting

awk;sed appending line to previous line....

I know this has been asked before but I just can't parse the syntax as explained. I have a set of files that has user information spread out over two lines that I wish to merge into one: User1NameLast User1NameFirst User1Address E-Mail:User1email User2NameLast User2NameFirst User2Address... (11 Replies)
Discussion started by: walkerwheeler
11 Replies

7. Solaris

Tail command in one line

HI i have to copy the last 5000 lines form a log file and copy the same in the same file .overwriting the same log file. ex: tail -5000 testfile1 > testfile2 cat testfile2 mv tesftfile2 testfile1 will produce the correct result.but i want to have this done in one line???? (4 Replies)
Discussion started by: saurabh84g
4 Replies

8. Shell Programming and Scripting

Read logline line by line with awk/sed

Hello, I have a logfile which is in this format: 1211667249500#3265 1211667266687#2875 1211667270781#1828 Is there a way to read the logfile line by line every time I execute the code and put the two numbers in the line in two separate variables? Something like: 1211667249500#3265... (7 Replies)
Discussion started by: dejavu88
7 Replies

9. Shell Programming and Scripting

Head and Tail in One Line

I am new to UNIX......I have one file which contains thousnads of records with header and tailer. Header Record 1 Record 2 .... .... Last Record Trailer I want to concatenate Header and Trailer in the first line....now the output should look like this: Header: Header value, Trailer:... (2 Replies)
Discussion started by: 33junaid
2 Replies

10. UNIX for Dummies Questions & Answers

how to sed with tail

hi, I am searching error and exception in my log and >> to report file, my code is : sed -n '//p;//p' $ARIBA_LOG_DIR/MyLog.txt >> $LOG_ERR_REP I need to report avove 5 line, that line and bellow 5 line.. what change is required in my code? (1 Reply)
Discussion started by: redlotus72
1 Replies
Login or Register to Ask a Question