Visit Our UNIX and Linux User Community


Awk: passing shell variables through and extracting text


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Awk: passing shell variables through and extracting text
# 8  
Old 02-20-2018
Not sure if anyone cares, but I ran some speed tests on the code you guys helped me with:
Code:
#!/bin/sh

# Run:    time ./testtime

A='center">'
B='</p>'
C='>'
D='&deg'

count=0
while [ $count -le 10000 ]; do

	# mawk  -v v1="$A" -v v2="$B" -v v3="$C" -v v4="$D" '$0 ~ v1 ".*" v2 {sub(v2 ".*", ""); sub(".*" v1, ""); print $0; next;} $0 ~ v3 ".*" v4 {sub(v4 ".*", ""); sub(".*" v3, ""); print $0 "C"; exit}' canada.html

        # Var  Run x  bash input     CPU
	
        # awk  10000x mksh piped   @ 55%  0m23.27s real     0m14.97s user     0m09.03s system
	# awk  10000x mksh unpiped @ 51%  0m21.67s real     0m12.68s user     0m08.17s system
	# nawk 10000x mksh unpiped @ 51$  0m21.59s real     0m12.63s user     0m08.27s system
	# gawk 10000x mksh unpiped @ 51%  0m21.64s real     0m12.55s user     0m08.40s system
	# mawk 10000x bash unpiped @ 53%  0m07.83s real     0m04.54s user     0m02.81s system
	# mawk 10000x mksh unpiped @ 51%  0m07.00s real     0m04.29s user     0m02.18s system
	# mawk 10000x dash unpiped @ 52%  0m06.49s real     0m04.02s user     0m01.91s system
	# mawk 10000x sh   unpiped @ 52%  0m06.40s real     0m04.01s user     0m01.93s system


	# mawk -vT1="$A" -vT2="$B" -vT3="$C" -vT4="$D" 'match ($0, T1 ".*" T2) {print substr ($0, RSTART+146, RLENGTH-150); next;} match ($0, T3 ".*" T4) {print substr ($0, RSTART+31, RLENGTH-35) "C"; exit;}' canada.html

	# Var  Run x  bash input     CPU
	
	# awk  10000x mksh piped   @ 55%  0m21.14s real     0m12.78s user     0m09.29s system
	# awk  10000x mksh unpiped @ 51%  0m19.70s real     0m10.71s user     0m08.22s system
        # nawk 10000x mksh unpiped @ 51%  0m19.68s real     0m10.72s user     0m08.27s system
        # gawk 10000x mksh unpiped @ 51%  0m19.68s real     0m10.81s user     0m08.15s system
	# mawk 10000x bash unpiped @ 53%  0m07.70s real     0m04.59s user     0m02.71s system
	# mawk 10000x mksh unpiped @ 51%  0m07.01s real     0m04.33s user     0m02.16s system
	# mawk 10000x sh   unpiped @ 52%  0m06.34s real     0m04.00s user     0m01.86s system
	# mawk 10000x dash unpiped @ 52%  0m06.33s real     0m03.99s user     0m01.87s system

	count=$((count+1))
done
beep

RudiC's code ran fastest in these tests with rdrtx1's code close behind. Mawk was fastest overall unpiped and run in sh/dash. I would have tested the other code but all other code did not work with this site. It also seems that these two implementations worked on every site I used them in!

Also, I had the output print to screen as I have not yet looked into redirecting it. The input was saved html script from a weather website; no net was used.

Amazing the difference it can make to switch a few things around. Smilie

One more thing. Here is a weather script that anyone can use, incase someone bumps into this on the net through a search:
Code:
#!/bin/sh

# Hamilton is used in this example. Replace with your own city's link. 
URL="accuweather.com/en/ca/hamilton/l8l/weather-forecast/55490"

# Formulas:
# RSTART  = +length of A
# RLENGTH = -length of (A + B)

A="txt : '"
B="',"
C="temp:'"
D="',  realfeel:'"

# Use 'echo' as a hack to make both print outputs print on the same line
#echo $(wget -q -O- "$URL" | mawk -v v1="$A" -v v2="$B" -v v3="$C" -v v4="$D" '$0 ~ v1 ".*" v2 {sub(v2 ".*", ""); sub(".*" v1, ""); print $0; next;} $0 ~ v3 ".*" v4 {sub(v4 ".*", ""); sub(".*" v3, ""); print $0 "C"; exit;}')
echo $(wget -q -O- "$URL" | mawk -vT1="$A" -vT2="$B" -vT3="$C" -vT4="$D" 'match ($0, T1 ".*" T2) {print substr ($0, RSTART+7, RLENGTH-9); next;} match ($0, T3 ".*" T4) {print substr ($0, RSTART+6, RLENGTH-20) "C"; exit;}')

Anyways, that's all. Thank you guys!
This User Gave Thanks to bedtime For This Post:

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk - passing variables in and out

Am looking to pass some Linux environment variables into AWK , can I simply use the -v option ? awk -F: -v AHOME=$HOME '{ if {rm AHOME/file.txt a=2 } }' config.txt ... (4 Replies)
Discussion started by: alldbest
4 Replies

2. Shell Programming and Scripting

Passing awk variables to bash variables

Trying to do so echo "111:222:333" |awk -F: '{system("export TESTO=" $2)}'But it doesn't work (2 Replies)
Discussion started by: urello
2 Replies

3. UNIX for Dummies Questions & Answers

Extracting a block of text from a large file using variables?

Hi UNIX Members, I've been tasked with performing the following: Extract a block of data in column form #This data changes each time, therefore automating future procedures Please Note the following: line = reading a line from a file_list that leads to the data The filename is called... (16 Replies)
Discussion started by: Klor
16 Replies

4. UNIX for Dummies Questions & Answers

Passing Global Shell variables to awk

Hi All, Iam trying to pass global shell variables and is not working Main script is like below CYEAR=`date +"%y"` CFYEAR=`date +"%Y"` CMONTH=`date +"%m"` if then PMONTH=12 PYEAR=`expr $CYEAR - 1` PFYEAR=`expr $CFYEAR - 1` else PMONTH=`expr... (6 Replies)
Discussion started by: baanprog
6 Replies

5. Shell Programming and Scripting

ksh passing to awk multiple dyanamic variables awk -v

Using ksh to call a function which has awk script embedded. It parses a long two element list file, filled with text numbers (I want column 2, beginning no sooner than line 45, that's the only known thing) . It's unknown where to start or end the data collection, dynamic variables will be used. ... (1 Reply)
Discussion started by: highnthemnts
1 Replies

6. Shell Programming and Scripting

Passing awk variables to shell

Hi. I need to parse file and assign some values to variables, right now i do like below MYHOMEDIR=`awk '/Home/ {print $NF}' output.txt` MYSHELL=`awk '/Shell/ {print $NF}' output.txt` PRGRP=`awk '/Primary/ {print $NF}' output.txt` SECGRP=`awk '/Second/ {print $NF}' output.txt` In this... (10 Replies)
Discussion started by: urello
10 Replies

7. UNIX for Dummies Questions & Answers

Passing Shell Variables to an awk command

Hello, I have two files File1 & File2. File1 76 135 136 200 250 345 .... File2 1 24 1 35 1 36 1 72 .... I want to get all the values form File2 corresponding to the range in File 1 and feed it to a program. Is the code below right? Can I pass shell variables to awk in this... (2 Replies)
Discussion started by: Gussifinknottle
2 Replies

8. Shell Programming and Scripting

Passing awk Variables

I am trying to pass the results from a variable gathered from awk, however when I echo the 'PARSE' and 'SUB', the response is blank. This is my command. awk -F= '/Unit/''{ PARSE=substr($2,1,5) ; SUB=substr($2,1,1) }' inputfile.lst Is this a kind of valid attempt or am I obligated to declare... (3 Replies)
Discussion started by: gozer13
3 Replies

9. Shell Programming and Scripting

Passing Variables to Awk

Hi I have a unix shell script with an awk statement. I would like to print some of the fields of an input file. However, I would like to print them dynamically, ie by passing the literal $1 $3 into the script to define the output. I have tried the following: variable1='$1' awk... (2 Replies)
Discussion started by: Bab00shka
2 Replies

10. Shell Programming and Scripting

Passing shell variables to awk program..

Hello, Can we pass shell variables like $PATH etc. to a awk program part for example, awk ' { fieldValue=$PATH .... }' file (1 Reply)
Discussion started by: Vishnu
1 Replies

Featured Tech Videos