Piping through commands read as variables from file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Piping through commands read as variables from file
# 1  
Old 07-19-2009
Question Piping through commands read as variables from file

This is going to be part of a longer script with more features, but I have boiled it down to the one thing that is presently stumping me. The goal is a script which checks for updates to web pages that can be run as a cron job. The script reads (from a tab-delim file) a URL, an MD5 digest, and an optional grep statement. (The idea of supplying a grep statement or grep parameters specific to each URL is to only look at the "interesting" parts of each page, ignoring ads, reader replies to blog posts, etc.) The script retrieves each page using curl, pipes it through the appropriate grep, then pipes through MD5, comparing the resulting digest to the MD5 digest already given in the file, and if different, activates an alert for that URL and stores the new MD5 back into the file. So here's what I have...

Contents of urls.test.ini (note fields separated by tabs)
==
Code:
http://www.sinfest.net    941f0f991ea9cd4c61ceb8cbf8833f81    grep comikaze
http://freefall.purrsia.com    f8af48b832c393b02993c18c87b6a47d    grep ff[0-9][0-9]*
http://twitter.com/spoonoftheday    3adc319c52e361e80cd2d7810a17181a    grep -o entry-content[^\<]*\<

Script code under test
==
Code:
OFS=$IFS
IFS="    " #tab character in the quotes
while read pageURL oldDigest filter
do
          echo "[$pageURL][$oldDigest][$filter]"
          newDigest=`curl -f -s -m 60 "$pageURL" | "$filter" | /sbin/md5`
          echo "$newDigest"
done < urls.test.ini
IFS=$OFS

Results
==
What this gives me is:
Code:
[http://www.sinfest.net][941f0f991ea9cd4c61ceb8cbf8833f81][grep comikaze]
prompt$: line 1: grep comikaze: command not found
d41d8cd98f00b204e9800998ecf8427e

(etc., similar for the rest of the data)

From this I conclude that:
1) The variables are at least all being read in correctly, with no extraneous characters, invisibles, etc.
2) The MD5 command is executing and $newDigest being assigned
3) Since the grep command is not being understood or executed (why?) the resulting digest is the MD5 for null.

Other things I have tried:
==
0) (Specifying bash vs. sh and back again)
1) Different quotings of the back ticks and variables including
newDigest="`curl -f -s -m 60 $pageURL | $filter | /sbin/md5`"
and
newDigest=`curl -f -s -m 60 $pageURL | $filter | /sbin/md5`
2) Specifying /usr/bin/grep in the .ini file. Error message THEN is the grep statement kicked back but saying "File not found", suggesting that grep is being run in the pipe, but it is misunderstanding and choking on the arguments
3) Separating out the arguments, with ONLY the params in the .ini file:
newDigest="`curl -f -s -m 60 $pageURL | grep $filter | /sbin/md5`"

That last variation almost works. It returns the correct MD5 digest for the first two lines of the example data, but chokes on the third one, saying "-o entry-content[^\<]*\< -- no such file", the difference no doubt being the initial hyphen, and incorrectly trying to read the -o flag as part of the pattern.

Any thoughts? It seems so simple, all I want the script to do is assign a variable the result of...
$ curl -options URL | grep -FLAGS PATTERN | md5
(which all works fine done manually at a shell prompt) but with URL, FLAGS and PATTERN being read in from a separate file.

Any insights/guidance will be most appreciated, thanks in advance. --JMF

PS: If I do this in Perl, the statement
chomp ($newDigest = `curl -f -s -m 60 $url | $filter | /sbin/md5`)
works perfectly!! However, I really want to write this in pure shell script as opposed to Perl, because I think I can get the overall script cleaner and more portable, if the dang thing would only WORK.
# 2  
Old 07-20-2009
put grep regexp rule always to between " ", because before grep shell get the arguments, shell try to parse those special things like: * $ ( ) ? > < ... are very often parsed before grep.

using echo before command is nice debug, you can see howto shell parse the line
echo cmd arguments

eval is often answer in this kind of needs, something like:
cmd="some \"arg1\" \"arg2\" "
eval $cmd
eval = parse cmdline twice.

1. some \"arg1\" \"arg2\" =>
some "arg1" "arg2"
2. parse, previous line looks good = "normal" commandline
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Piping commands using xargs

Need help in piping commands using xargs I have several .tar.gz files that I need to list the folder content in a subdirectory. For example, a.tar.gz b.tar.gz c.tar.gz The following command works great for each .tar.gz file but it's a pain to run the tar command for each file. tar -tf... (11 Replies)
Discussion started by: april
11 Replies

2. Shell Programming and Scripting

Proper chaining and piping of commands

can someone please help me fix the below one liner? what i want to do is run a command, and if that command is successful, pipe the output of the command to another command. i'd prefer not to use any temp files for this. who -blahblah ; if ; then echo exit; fi | egrep username (2 Replies)
Discussion started by: SkySmart
2 Replies

3. Shell Programming and Scripting

How to read each line from input file, assign variables, and echo to output file?

I've got a file that looks like this (spaces before first entries intentional): 12345650-000005000GL140227 ANNUAL HELC FEE EN 22345650-000005000GL140227 ANNUAL HELC FEE EN 32345650-000005000GL140227 ANNUAL HELC FEE EN I want to read through the file line by line,... (6 Replies)
Discussion started by: Scottie1954
6 Replies

4. Shell Programming and Scripting

Read variables from a file and then export it.

Hi I want to read variables from one file and then set it as environment variable; The text file is test.txt which contains SPEED:1000 IP:172.26.126.11 My code is: while read line; do var1=`echo $line | awk 'BEGIN {FS=":"} { print $1 }'` echo $var1 var2=`echo $line | awk 'BEGIN {FS=":"}... (8 Replies)
Discussion started by: SSM
8 Replies

5. UNIX for Dummies Questions & Answers

Piping commands

Hi I am tryin to undertand piping command1|command2 from what i learn output of cammand 2 is an intput for command 1 right? If so . What dose next sequence do cat f1 >> f2 | grep '^' I think it takes context of f1 and Concatenate's it to f2 and then looks for ....i don't know..... (7 Replies)
Discussion started by: iliya24
7 Replies

6. UNIX for Dummies Questions & Answers

Piping multiple commands

Using the below code I want to find all .sff files and extract them. This works but it seems very cheap. Is there a safer more efficient way to go about this? #!/bin/bash G1=(/home/dirone) find ${G1} -type f -name \*.sff | xargs python /usr/local/bin/sff_extract.py (3 Replies)
Discussion started by: jrymer
3 Replies

7. Shell Programming and Scripting

[Solved] Help piping tail to read STDIN

Hello everybody, I need some help here. I have a log file that gets updated every hour approximately. I want to make some processing on each line which is added in the log file with a program written in PERL. The problem is that I don't see anything when a line is added in the log file. I... (6 Replies)
Discussion started by: Samb95
6 Replies

8. Shell Programming and Scripting

How to execute commands read from another file?

Please help with this simple example. I can not figure out how to do it. A file named “job” contains only this one line:var=5I need a script to read the job file and execute it as a command. This is my attempt (it does not work):#!/bin/sh exec < job echo "var = $var"output should read “var = 5”... (5 Replies)
Discussion started by: wolfv
5 Replies

9. Shell Programming and Scripting

Read variables and their values from file

Hi, I want to read the variables and the values from the txt file and compare these values with the ones computed by script. for ex: say var.txt contains the variable names and their values: one 1 two 2 three 3 The value of variables "one" "two" and "three" will be computed in the script... (3 Replies)
Discussion started by: bhushana
3 Replies

10. Shell Programming and Scripting

Piping output to while read

Hi. Im using cat to output the contents of a file, then piping it to my while read loop.In this loop variables get assigned values. However when i try to use the variables outside the loop their values has been reset.I understand about subshells etc. but I have no idea how to "preserve" the... (3 Replies)
Discussion started by: Ultimodiablo
3 Replies
Login or Register to Ask a Question