Awk: passing shell variables through and extracting text


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Awk: passing shell variables through and extracting text
# 1  
Old 02-19-2018
Awk: passing shell variables through and extracting text

Hello, new to the forums and to awk. Glad to be here. Smilie


I want to pass two shell (#!/bin/sh) variables through to awk and use them. They will determine where to start and stop text extraction.

The code with the variables hard-coded in awk works fine; the same code, but with the shell variables, does not.


Here is the hard-coded (working) code:

Code:
echo "Here is a blah ... blah ... blah very nice string." | awk -F 'Here is a' '{print $2}' RS='very nice string.'

The result—and it is the result I seek—is:
Code:
blah ... blah ... blah


Here is the code with variables:
Code:
textA="Here is a"
textB="very nice string."

echo "Here is a blah ... blah ... blah very nice string." | awk  -v var1="$textA" -v var2="$textB" -F var1 '{print $2}' RS=var2

The result is that just a blank line—not nothing, but a return carriage.

Any ideas?
Moderator's Comments:
Mod Comment Please use CODE tags (not ICODE tags) when displaying full-line and multi-line sample input, output, and code segments.

Use ICODE tags when displaying sample input, output, and code segments in-line with ordinary text.

Last edited by Don Cragun; 02-19-2018 at 02:21 PM.. Reason: Change ICODE tags to CODE tags; add ICODE tags.
# 2  
Old 02-19-2018
Code:
textA="Here is a"
textB="very nice string."

echo "Here is a blah ... blah ... blah very nice string." | awk  -v var1="$textA" -v var2="$textB" '{sub(var2, ""); sub(var1, ""); print}'

This User Gave Thanks to rdrtx1 For This Post:
# 3  
Old 02-19-2018
Quote:
Originally Posted by rdrtx1
Code:
textA="Here is a"
textB="very nice string."

echo "Here is a blah ... blah ... blah very nice string." | awk  -v var1="$textA" -v var2="$textB" '{sub(var2, ""); sub(var1, ""); print}'

Thank you for such a timely response! Smilie

I don't think I was clear in my question: I wanted just the text in between the two phrases and that alone—nothing except that. If there is no match of the two phrases in one field then I want nothing to be returned, so if there is additional text before and after textA and/or textB, that text should not be there.


Okay, let me just give you guys the code I'm using and not dilly dally:

Working code to extract the local weather in my city.
Code:
#!/bin/bash

wget -q -O- "hamiltonweather.ca/traffic/" | awk -F '<h2>' '{sub(/ •/, ""); print $2}' RS='</h2>' | grep -o '[^,]*$'

Returns:
Code:
Light Rain 4.3°C

The code works beautifully as is (I added some of your code in there rdrtx to get rid of a dot! Smilie Grep clears a blank line at the end) when hard-coded, but as soon as I add the above variables it doesn't work as intended:

Unworking Code with variables added:
Code:
#!/bin/bash

textBefore='<h2>'
textAfter='</h2>'

wget -q -O- "hamiltonweather.ca/traffic/" | awk -v var1="$textBefore" -v var2="$textAfter" -F var1 '{sub(/ •/, ""); print $2}' RS=var2 | grep -o '[^,]*$'

Returns nothing at all. How to make the awk understand and use the variables as in the working code?

Last edited by Don Cragun; 02-19-2018 at 03:20 PM.. Reason: Fix tags again. QUOTE->CODE, ICODE->CODE, add ICODE.
# 4  
Old 02-19-2018
Code:
textA="Here is a"
textB="very nice string."

echo "123123123 Here is a blah ... blah ... blah very nice string. aaa aaa aaa" | awk  -v var1="$textA" -v var2="$textB" '$0 ~ var1 ".*" var2 {sub(var2 ".*", ""); sub(".*" var1, ""); print}'

This User Gave Thanks to rdrtx1 For This Post:
# 5  
Old 02-19-2018
Your sample data included in your echo statement is nothing at all like the data you are looking for in the output from wget.
Maybe the following would come closer to doing what you want with a little less bother:
Code:
#!/bin/bash
textBefore='<h2>'
textAfter='</h2>'

wget -q -O- "hamiltonweather.ca/traffic/" | awk -F "($textBefore|$textAfter)" 'NF==3{print $2}'

I can't test it on my laptop since it doesn't have a wget utility, but it should come close to doing what you want. If it doesn't, please show us the output the above wget command produces (in CODE tags), so we can see what the data you're trying to process really looks like.

Please also get into the habit of letting us know what operating system and shell you're using whenever you start a thread in this forum. The utilities (and the options they support) vary from system to system and shell features vary from shell to shell. Telling us details about your environment helps us provide you with suggestions that will work in your environment.

And, please use CODE tags (not ICODE and not QUOTE tags) for sample full-line and multi-line input, output, and code segments. Use ICODE tags when displaying partial-line sample input, output, and code segments in-line with other text.
# 6  
Old 02-19-2018
You're not too far off. Try
Code:
wget -q -O- "hamiltonweather.ca/traffic/" | awk -F "$textBefore" -v RS="$textAfter"  '{sub(/ •/, ""); print $2}'

EDIT: Or

Code:
wget -q -O- "hamiltonweather.ca/traffic/" | awk -vT1="$textBefore" -vT2="$textAfter" 'match ($0, T1 ".*" T2) {print substr ($0, RSTART+4, RLENGTH-9)}'

# 7  
Old 02-19-2018
Quote:
Originally Posted by rdrtx1
Code:
textA="Here is a"
textB="very nice string."

echo "123123123 Here is a blah ... blah ... blah very nice string. aaa aaa aaa" | awk  -v var1="$textA" -v var2="$textB" '$0 ~ var1 ".*" var2 {sub(var2 ".*", ""); sub(".*" var1, ""); print}'

Works like a charm! Smilie

Quote:
I can't test it on my laptop since it doesn't have a wget utility, but it should come close to doing what you want. If it doesn't, please show us the output the above wget command produces (in CODE tags), so we can see what the data you're trying to process really looks like.

Please also get into the habit of letting us know what operating system and shell you're using whenever you start a thread in this forum. The utilities (and the options they support) vary from system to system and shell features vary from shell to shell. Telling us details about your environment helps us provide you with suggestions that will work in your environment.

And, please use CODE tags (not ICODE and not QUOTE tags) for sample full-line and multi-line input, output, and code segments. Use ICODE tags when displaying partial-line sample input, output, and code segments in-line with other text.
Thank you. And for me it does indeed work. I found that I didn't even need to use the NF==3 code as it was my original code that necessitated its use in the first place. Smilie

I'm on Debian, using bash, mksh, and sh, all of which work with this code... It seems I used the incorrect code tags—I see them now. I thought something was a trifle amiss. Smilie

RudiC,
Code:
wget -q -O- "hamiltonweather.ca/traffic/" | awk -F "$textBefore" -v RS="$textAfter"  '{sub(/ •/, ""); print $2}'

This code worked, but for some reason it adds another line as mine had done. It's great to see some resolution on my own attempt!

Code:
wget -q -O- "hamiltonweather.ca/traffic/" | awk -vT1="$textBefore" -vT2="$textAfter" 'match ($0, T1 ".*" T2) {print substr ($0, RSTART+4, RLENGTH-9)}'

Also, works. Looks like a lot more code to get the job done though. I've have to look at man pages and find out what it all does. This is my third day using awk. Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk - passing variables in and out

Am looking to pass some Linux environment variables into AWK , can I simply use the -v option ? awk -F: -v AHOME=$HOME '{ if {rm AHOME/file.txt a=2 } }' config.txt ... (4 Replies)
Discussion started by: alldbest
4 Replies

2. Shell Programming and Scripting

Passing awk variables to bash variables

Trying to do so echo "111:222:333" |awk -F: '{system("export TESTO=" $2)}'But it doesn't work (2 Replies)
Discussion started by: urello
2 Replies

3. UNIX for Dummies Questions & Answers

Extracting a block of text from a large file using variables?

Hi UNIX Members, I've been tasked with performing the following: Extract a block of data in column form #This data changes each time, therefore automating future procedures Please Note the following: line = reading a line from a file_list that leads to the data The filename is called... (16 Replies)
Discussion started by: Klor
16 Replies

4. UNIX for Dummies Questions & Answers

Passing Global Shell variables to awk

Hi All, Iam trying to pass global shell variables and is not working Main script is like below CYEAR=`date +"%y"` CFYEAR=`date +"%Y"` CMONTH=`date +"%m"` if then PMONTH=12 PYEAR=`expr $CYEAR - 1` PFYEAR=`expr $CFYEAR - 1` else PMONTH=`expr... (6 Replies)
Discussion started by: baanprog
6 Replies

5. Shell Programming and Scripting

ksh passing to awk multiple dyanamic variables awk -v

Using ksh to call a function which has awk script embedded. It parses a long two element list file, filled with text numbers (I want column 2, beginning no sooner than line 45, that's the only known thing) . It's unknown where to start or end the data collection, dynamic variables will be used. ... (1 Reply)
Discussion started by: highnthemnts
1 Replies

6. Shell Programming and Scripting

Passing awk variables to shell

Hi. I need to parse file and assign some values to variables, right now i do like below MYHOMEDIR=`awk '/Home/ {print $NF}' output.txt` MYSHELL=`awk '/Shell/ {print $NF}' output.txt` PRGRP=`awk '/Primary/ {print $NF}' output.txt` SECGRP=`awk '/Second/ {print $NF}' output.txt` In this... (10 Replies)
Discussion started by: urello
10 Replies

7. UNIX for Dummies Questions & Answers

Passing Shell Variables to an awk command

Hello, I have two files File1 & File2. File1 76 135 136 200 250 345 .... File2 1 24 1 35 1 36 1 72 .... I want to get all the values form File2 corresponding to the range in File 1 and feed it to a program. Is the code below right? Can I pass shell variables to awk in this... (2 Replies)
Discussion started by: Gussifinknottle
2 Replies

8. Shell Programming and Scripting

Passing awk Variables

I am trying to pass the results from a variable gathered from awk, however when I echo the 'PARSE' and 'SUB', the response is blank. This is my command. awk -F= '/Unit/''{ PARSE=substr($2,1,5) ; SUB=substr($2,1,1) }' inputfile.lst Is this a kind of valid attempt or am I obligated to declare... (3 Replies)
Discussion started by: gozer13
3 Replies

9. Shell Programming and Scripting

Passing Variables to Awk

Hi I have a unix shell script with an awk statement. I would like to print some of the fields of an input file. However, I would like to print them dynamically, ie by passing the literal $1 $3 into the script to define the output. I have tried the following: variable1='$1' awk... (2 Replies)
Discussion started by: Bab00shka
2 Replies

10. Shell Programming and Scripting

Passing shell variables to awk program..

Hello, Can we pass shell variables like $PATH etc. to a awk program part for example, awk ' { fieldValue=$PATH .... }' file (1 Reply)
Discussion started by: Vishnu
1 Replies
Login or Register to Ask a Question