Passing Shell Input to AWK

Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Passing Shell Input to AWK
# 1  
Old 09-22-2009
Passing Shell Input to AWK

I am trying to search a log for a particluar pattern listing the total # of occurences in the end.

I thought using a shell script for input then calling awk to search for the paramters specified. I want the script to be usable acorss envs.

#! /usr/bin/bash
# get the variables
echo -n "1.  LogFILE ?"
read LOG
echo -n "2. SEARCH FOR ? "
read word
echo -n "3. Timestamp start = "
read time_st
echo -n "4. Timestamp end = "
read time_en
# search the file
awk  -v"1=${word}" "2=${time_st}" "3=${time_en}" "4=${log}" | wc

Source: Items in read is what I would like to search on.
Code: - - [22/Sep/2009:10:02:24 -0400] "GET /portal/framework/skins/cafe/css/body.css HTTP/1.1" 200 1304 - - [22/Sep/2009:14:06:32 -0400] "GET /portal/cafekeepalive.jsp?NONE HTTP/1.1" 200 21

"" 12 lines, 287 characters
bash-3.00$  ./
1.  LogFILE ?log.log
2. SEARCH FOR ? cafe
3. Timestamp start = 10:0*
4. Timestamp end = 14:0*
awk: syntax error near line 1
awk: bailing out near line 1

Thanks in advance for any assitance

Last edited by vgersh99; 09-22-2009 at 05:40 PM.. Reason: code tags, PLEASE!
# 2  
Old 09-22-2009
To keep the forums high quality for all users, please take the time to format your posts correctly.

First of all, use Code Tags when you post any code or data samples so others can easily read your code. You can easily do this by highlighting your code and then clicking on the # in the editing menu. (You can also type code tags [code] and [/code] by hand.)

Second, avoid adding color or different fonts and font size to your posts. Selective use of color to highlight a single word or phrase can be useful at times, but using color, in general, makes the forums harder to read, especially bright colors like red.

Third, be careful when you cut-and-paste, edit any odd characters and make sure all links are working property.

Thank You.

The UNIX and Linux Forums
# 3  
Old 09-22-2009
not sure awk is the right tool here.

you can 'grep -n' to get the line number of each $time_st and $time_en
you can use a combination of 'head' and 'tail' to output the subset of lines between $time_st and $time_en you want to search
you can use 'grep -c' to get the number of occurances of $word in that subset

# 4  
Old 09-23-2009
DV -
Thanks for the response...

Am heading down the right direction with this?

cat access.log.1253664000 | egrep "" | egrep "10:0*" | egrep "11:*" head -1


Last edited by wawa44oz; 09-23-2009 at 03:28 PM..
# 5  
Old 09-23-2009
Originally Posted by wawa44oz
DV -
Thanks for the response...

Am heading down the right direction with this?

cat access.log.1253664000 | egrep "" | egrep "10:0*" | egrep "11:*" head -1

You don't need to use cat here.

Your greps need some escaping. Right now the . means 'match any character', not 'match .'

And your greps won't return anything because first you reject everything except your start time, and after that you reject everything except your end time. You need to match start and end and everything inbetween. Since grep doesn't understand what date and time means -- or what digits mean, for that matter -- I don't think grep can do what you want all by itself. There's several things with : in it that could match anyway, so that probably won't narrow it down to what you want. I'll work on this a bit...

---------- Post updated at 12:38 PM ---------- Previous update was at 12:25 PM ----------

Here's how you'd match exactly the hours you want.

# Build a list of the hours we want, to fill into egrep
for ((N=START+1; N<=END; N++))

# This will match 14:06 but not 10:02 since it starts at 11
echo -e "[22/Sep/2009:10:02:24 -0400]\n[22/Sep/2009:14:06:32 -0400]" |
        egrep "/[0-9]+:($STR):"

You could tack another grep to match a specific hostname or what have you. It's not too sophisticated. You can't take it too much farther because grep can't understand what the dates actually mean.

This is about as complex as I'd bother making it in a shell script, since shell in general has a hard time processing date information. If you want something smart enough to just specify a beginning time and date and end time and date, I'd just use perl and process the dates wholesale to compare them.
# 6  
Old 09-23-2009
ok, so in my orig response I was thinking this:

# the number of the first line in the file with the start time and word
STARTLINE=grep -n ":$start_time:[0-5][0-9]:.*$word"|head -1|cut -f1 -d':'

# the number of the last line in the file with the end time and word
ENDLINE=grep -n ":$end_time:[0-5][0-9]:.*$word"|tail -1|cut -f1 -d':'

# the subset of lines between STARTLINE and ENDLINE
head -$ENDLINE file | tail -n +$STARTLINE > subsetfile

# the number of lines in subsetfile containing $word
grep -c $word subsetfile

regexes here will work, but could be improved.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Passing Shell Variable to awk

Hello All, May i please why my shell variable is not getting passed into awk script. #!/bin/bash -vx i="1EB07C50" /bin/awk -v ID="$i" '/ID/ {match($0,/ID/);print substr($0,RSTART,RLENGTH)}' /var/log/ScriptLogs/keys.13556.txt Thank you. (1 Reply)
Discussion started by: Ariean
1 Replies

2. UNIX for Dummies Questions & Answers

Passing Global Shell variables to awk

Hi All, Iam trying to pass global shell variables and is not working Main script is like below CYEAR=`date +"%y"` CFYEAR=`date +"%Y"` CMONTH=`date +"%m"` if then PMONTH=12 PYEAR=`expr $CYEAR - 1` PFYEAR=`expr $CFYEAR - 1` else PMONTH=`expr... (6 Replies)
Discussion started by: baanprog
6 Replies

3. Shell Programming and Scripting

Passing a variable as input to another shell

I have a shell program that calls another shell program the following code works . "10/9/12 17:51:19:783."|read c but when I am passing the the time in a variable like in the code below, the shell is not returning proper value time="10/9/12... (9 Replies)
Discussion started by: swayam123
9 Replies

4. Shell Programming and Scripting

Passing awk variables to shell

Hi. I need to parse file and assign some values to variables, right now i do like below MYHOMEDIR=`awk '/Home/ {print $NF}' output.txt` MYSHELL=`awk '/Shell/ {print $NF}' output.txt` PRGRP=`awk '/Primary/ {print $NF}' output.txt` SECGRP=`awk '/Second/ {print $NF}' output.txt` In this... (10 Replies)
Discussion started by: urello
10 Replies

5. Shell Programming and Scripting

Passing variable as an input file to AWK comand

Hi, I would like to compare 2 files using awk, which I can do by using: awk 'NR==FNR{a;next} (NR > 32 && $2 in a) {print $0}' File1 and File2. If the name of the File1 is in another file (for example, column 4 in File 3) then how can I pass this column 4 to the awk command. Thanks in... (1 Reply)
Discussion started by: ezhil01
1 Replies

6. Shell Programming and Scripting

Passing string as a input parameter for a shell script

Hi i have a shell script which needs a string as an input parameter. How to pass the string param as an input? In command line am running the script. for e.g., a="who is a buddy?" sh $a Inside the script i get this input param as $1 but only the value "who" is accepted... (12 Replies)
Discussion started by: vidhyaS
12 Replies

7. UNIX for Dummies Questions & Answers

Passing a Shell Variable to awk

Hello, I have a file with 4 columns. An arbitrary example is shown below: a Tp 10 xyz b Tq 8 abc c Tp 99 pqr d Tp 44 rst e Tr 98 efg Based on the values in col 2 and col 3, I will execute another program. I have been running this:... (5 Replies)
Discussion started by: Gussifinknottle
5 Replies

8. Shell Programming and Scripting

Passing a variable to awk while in a shell for loop

I am a newbie to awk and c programming, however am not a unix newbie. However, I do need help with a kshell script I am writing. It is almost complete, the last step is killing me. Any help would be greatly appreciated. What I am trying to do is cat a text file that has usernames. Then, using... (2 Replies)
Discussion started by: synergy_texas
2 Replies

9. Shell Programming and Scripting

Passing Value from awk to shell

I'm writting a script to calculate the total files and number of files that have been generated but not able to access the value of qwk variable in Shell. Please suggest.. Script :: cd /home/singhraa/tmp/scripts count=0 total=`wc -l hk_jobs.txt` #total number of files echo $total ... (3 Replies)
Discussion started by: raman1605
3 Replies

10. Shell Programming and Scripting

Passing shell variables to awk program..

Hello, Can we pass shell variables like $PATH etc. to a awk program part for example, awk ' { fieldValue=$PATH .... }' file (1 Reply)
Discussion started by: Vishnu
1 Replies
Login or Register to Ask a Question