How to tail sed and awk in one line?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to tail sed and awk in one line?
# 1  
Old 10-13-2016
How to tail sed and awk in one line?

Hello,
I am trying to create an iptables script with tail ,sed and awk.

1st Request: Search keyword "secret" in access.log file
2nd Request: Get first column matching lines (ip address)
3rd Request: Save it to a file

This is what I did so far:

grep.sh
Code:
#!/bin/bash
while true;
do
tail -f /var/log/apache2/access.log | awk '{print $1}' 
#tail -f /var/log/apache2/access.log | grep --line-buffered secret | awk '/secret/' | awk '{print $1}'     #not working as expected
#tail -f /var/log/apache2/access.log | grep --line-buffered debug | awk '$1' > ip.txt                          #gives all lines but not column1
#echo "iptables -A INPUT -s "$1" -j DROP" >> black
#echo "$A" >> black
#chmod 755 black
#./black
done

It prints out all ip address but not searching for the keyword "secret".
When I clear shebang symbol existing at the beginning of the following lines, it does not print matching output.

Is it possible to assign the pipe output to a variable so that I can use that value in following steps for iptables command?

Thanks
Boris

Last edited by baris35; 10-13-2016 at 12:58 AM..
# 2  
Old 10-13-2016
Quote:
It prints out all ip address but not searching for the keyword "secret".
Code:
tail -f /var/log/apache2/access.log | awk '/secret/{print $1}'

# 3  
Old 10-13-2016
Hello baris35,

If you want to continuously read the lines and search for a string and put that's matched line's first field to an output_file then following may help you too in same.
Code:
tail -f /var/log/apache2/access.log | awk '/TEST1/{print $1 >> "Output_file"}'

When you are done with tailing the logs(You could come out of above command) then you could check the file named Output_file for matched logs.

NOTE: In above command you could change search string from TEST1to your required text.

Thanks,
R. Singh

Last edited by RavinderSingh13; 10-13-2016 at 06:59 AM.. Reason: Added more clear comment "come out of that command" so that I could justify my statement "when you are done with tailing log"
This User Gave Thanks to RavinderSingh13 For This Post:
# 4  
Old 10-13-2016
I'm confused. People seem to be ignoring the tail -f flag in the pipelines in this code. Note that the command:
Code:
tail -f /var/log/apache2/access.log | awk '{print $1}'

will run forever. No command following this in your script will ever be executed.

If you want to took at the current contents of the log (instead of looking at the current and future contents of the log), use:
Code:
awk '{print $1}' /var/log/apache2/access.log

With tail -f, you don't need (or want) a while true loop, but you need to combine all of the tail and awk pipelines into a single pipeline (not in a loop).
# 5  
Old 10-13-2016
Regarding the while true loop, are you concerned that the log file might be rotated/aged/re-written/whatever? If so, does your version of tail have the -F or --retry flags? These will keep looking for the file if it becomes inaccessible, so you keep reading the new log files that are created by normal ageing processes.

The alternate would be that you would have to have some way to terminate your tail and then restart it, all within the while true loop.


Can you put this code into context as to why you have written it like this so far. I'm sure we can help you to find a solution.



Kind regards,
Robin
# 6  
Old 10-13-2016
Hello,
Thanks for your answers. In comply with your suggestions, I changed the script as shown below:

Code:
#!/bin/bash
#while true;                                                               ----removed according to your suggestions
#do                                                                           ----removed according to your suggestions
prefix="/sbin/iptables -A INPUT -s "
tail -f /var/log/apache2/access.log | awk '/debug/{print $1 >> "ip2"}' | awk '!a[$0]++' ip2 >> ip | echo "$prefix$ip" > fail2ban_ip | sed -e 's/$/ \
-p tcp -dports 80 -j DROP/' -i fail2ban_ip > block | chmod 755 block | ./block

It did not give an output. Then I tested below from command line, I see that it's giving just an empty file. I checked access.log and I saw searched word in latest lines.

tail -f /var/log/apache2/access.log | awk '/debug/{print $1 >> "ip2"}'

Output file ip2 is empty.
I will keep searching a solution for this

Thanks
Boris

Last edited by baris35; 10-13-2016 at 03:24 PM..
# 7  
Old 10-13-2016
You don't seem to understand the concept of a pipeline. In a pipeline, the standard output from the first command in the pipeline becomes the standard input to the second command in the pipeline. If there are more than two commands in the pipeline, the output from every command in the pipeline but the last become the standards input for the next command in the pipeline. And all stages in the pipeline are free to run asynchronously as long as each command reading from its standard input has data to read and each process writing data to the next element of the pipeline is still there to read from the pipe. But in your pipeline:
Code:
tail -f /var/log/apache2/access.log |
  awk '/debug/{print $1 >> "ip2"}' |
  awk '!a[$0]++' ip2 >> ip |
  echo "$prefix$ip" > fail2ban_ip |
  sed -e 's/$/ -p tcp -dports 80 -j DROP/' -i fail2ban_ip > block |
  chmod 755 block |
  ./block

The first awk command writes nothing to standard output. The second awk does not read anything from its standard input. (It reads from a file named ip2 which might or might not exist when that awk tries to open it since it is being created asynchronously by the previous awk. If the file does exist, there is a good chance that it will be processed by the 2nd awk before the 1st awk adds anything to it.) And, the second awk creates a file named ip, but does not write anything to the next stage of the pipeline, but that may be OK because echo never reads from its standard input. Note also that the variable ip which is being passed to echo is not defined by this script. And, since the output from echo is directed to a file, again nothing is written to the standard input of the next stage of the pipeline. But, again, that may be OK because sed when given a filename operand does not read from its standard input. And, sed when invoked with a -i option either gives you a syntax error for an unrecognized option or updates its input file in place. In either of these cases sed writes nothing to its standard output, so the file created by the redirection of standard output from sed will be an empty file. Then we have the next stage of the pipeline (chmod 755 block) which neither reads from standard input nor writes to standard output. And, finally, we have the last stage of the pipeline (./block) which either executes an empty file (if the chmod completed before this stage starts) or fails because it is not executable (if the file hadn't been created yet, or had been created but had not yet been made executable).

I don't know what you are trying to do with this pipeline, but there are so many different concurrent operations going on here with nothing to guarantee that they happen in the proper sequence that the actual behavior will vary considerably depending on what shell you use to run this script, what operating system you use to run this script, the number of other processes that are running on the system when you run this script, and the phase of the moon.

I don't have an apache2 access log file to look at, so I have no idea what is in some of the data you seem to be trying to extract from records in that file. And, you haven't really given us an explanation of what you are trying to do with this script. Without a much better description of what you're trying to do and a sample log file to use as a test case, I don't see how I can help you.
This User Gave Thanks to Don Cragun For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Multiple line search, replace second line, using awk or sed

All, I appreciate any help you can offer here as this is well beyond my grasp of awk/sed... I have an input file similar to: &LOG &LOG Part: "@DB/TC10000021855/--F" &LOG &LOG &LOG Part: "@DB/TC10000021852/--F" &LOG Cloning_Action: RETAIN &LOG Part: "@DB/TCCP000010713/--A" &LOG &LOG... (5 Replies)
Discussion started by: KarmaPoliceT2
5 Replies

2. Shell Programming and Scripting

sed and awk giving error ./sample.sh: line 13: sed: command not found

Hi, I am running a script sample.sh in bash environment .In the script i am using sed and awk commands which when executed individually from terminal they are getting executed normally but when i give these sed and awk commands in the script it is giving the below errors :- ./sample.sh: line... (12 Replies)
Discussion started by: satishmallidi
12 Replies

3. UNIX for Dummies Questions & Answers

What should be precedence of using awk, sed, head and tail in UNIX?

Hi All, I am new to unix. In this forum some days back, I have read something like below: 1) Do not use perl if awk can do your work. 2) Do not use awk if sed can do your work. . . . I do not re-collect the whole thing. I think it is good to know the precedence of using these... (2 Replies)
Discussion started by: Prathmesh
2 Replies

4. Shell Programming and Scripting

sed or awk to replace a value in a certain line.

I have an input like following. *DEFINE_CURVE_TITLE Force for tool binder $# lcid sidr sfa sfo offa offo dattyp 3 0 1 .000000 125.00000 0.000 0.000 0 $# a1 ... (5 Replies)
Discussion started by: hamnsan
5 Replies

5. UNIX for Advanced & Expert Users

sed one liner simialr to tail command

Can anyone explain the below sed oneliner? sed -e ':a' -e '$q;N;11,$D;ba' It works same as tail command. I just want to know how it works. Thanks (1 Reply)
Discussion started by: pandeesh
1 Replies

6. Shell Programming and Scripting

awk;sed appending line to previous line....

I know this has been asked before but I just can't parse the syntax as explained. I have a set of files that has user information spread out over two lines that I wish to merge into one: User1NameLast User1NameFirst User1Address E-Mail:User1email User2NameLast User2NameFirst User2Address... (11 Replies)
Discussion started by: walkerwheeler
11 Replies

7. Solaris

Tail command in one line

HI i have to copy the last 5000 lines form a log file and copy the same in the same file .overwriting the same log file. ex: tail -5000 testfile1 > testfile2 cat testfile2 mv tesftfile2 testfile1 will produce the correct result.but i want to have this done in one line???? (4 Replies)
Discussion started by: saurabh84g
4 Replies

8. Shell Programming and Scripting

Read logline line by line with awk/sed

Hello, I have a logfile which is in this format: 1211667249500#3265 1211667266687#2875 1211667270781#1828 Is there a way to read the logfile line by line every time I execute the code and put the two numbers in the line in two separate variables? Something like: 1211667249500#3265... (7 Replies)
Discussion started by: dejavu88
7 Replies

9. Shell Programming and Scripting

Head and Tail in One Line

I am new to UNIX......I have one file which contains thousnads of records with header and tailer. Header Record 1 Record 2 .... .... Last Record Trailer I want to concatenate Header and Trailer in the first line....now the output should look like this: Header: Header value, Trailer:... (2 Replies)
Discussion started by: 33junaid
2 Replies

10. UNIX for Dummies Questions & Answers

how to sed with tail

hi, I am searching error and exception in my log and >> to report file, my code is : sed -n '//p;//p' $ARIBA_LOG_DIR/MyLog.txt >> $LOG_ERR_REP I need to report avove 5 line, that line and bellow 5 line.. what change is required in my code? (1 Reply)
Discussion started by: redlotus72
1 Replies
Login or Register to Ask a Question