Piping through commands read as variables from file
This is going to be part of a longer script with more features, but I have boiled it down to the one thing that is presently stumping me. The goal is a script which checks for updates to web pages that can be run as a cron job. The script reads (from a tab-delim file) a URL, an MD5 digest, and an optional grep statement. (The idea of supplying a grep statement or grep parameters specific to each URL is to only look at the "interesting" parts of each page, ignoring ads, reader replies to blog posts, etc.) The script retrieves each page using curl, pipes it through the appropriate grep, then pipes through MD5, comparing the resulting digest to the MD5 digest already given in the file, and if different, activates an alert for that URL and stores the new MD5 back into the file. So here's what I have...
Contents of urls.test.ini (note fields separated by tabs)
==
Script code under test
==
Results
==
What this gives me is:
(etc., similar for the rest of the data)
From this I conclude that:
1) The variables are at least all being read in correctly, with no extraneous characters, invisibles, etc.
2) The MD5 command is executing and $newDigest being assigned
3) Since the grep command is not being understood or executed (why?) the resulting digest is the MD5 for null.
Other things I have tried:
==
0) (Specifying bash vs. sh and back again)
1) Different quotings of the back ticks and variables including
newDigest="`curl -f -s -m 60 $pageURL | $filter | /sbin/md5`"
and
newDigest=`curl -f -s -m 60 $pageURL | $filter | /sbin/md5`
2) Specifying /usr/bin/grep in the .ini file. Error message THEN is the grep statement kicked back but saying "File not found", suggesting that grep is being run in the pipe, but it is misunderstanding and choking on the arguments
3) Separating out the arguments, with ONLY the params in the .ini file:
newDigest="`curl -f -s -m 60 $pageURL | grep $filter | /sbin/md5`"
That last variation almost works. It returns the correct MD5 digest for the first two lines of the example data, but chokes on the third one, saying "-o entry-content[^\<]*\< -- no such file", the difference no doubt being the initial hyphen, and incorrectly trying to read the -o flag as part of the pattern.
Any thoughts? It seems so simple, all I want the script to do is assign a variable the result of...
$ curl -options URL | grep -FLAGS PATTERN | md5
(which all works fine done manually at a shell prompt) but with URL, FLAGS and PATTERN being read in from a separate file.
Any insights/guidance will be most appreciated, thanks in advance. --JMF
PS: If I do this in Perl, the statement
chomp ($newDigest = `curl -f -s -m 60 $url | $filter | /sbin/md5`)
works perfectly!! However, I really want to write this in pure shell script as opposed to Perl, because I think I can get the overall script cleaner and more portable, if the dang thing would only WORK.
put grep regexp rule always to between " ", because before grep shell get the arguments, shell try to parse those special things like: * $ ( ) ? > < ... are very often parsed before grep.
using echo before command is nice debug, you can see howto shell parse the line
echo cmd arguments
eval is often answer in this kind of needs, something like:
cmd="some \"arg1\" \"arg2\" "
eval $cmd
eval = parse cmdline twice.
1. some \"arg1\" \"arg2\" =>
some "arg1" "arg2"
2. parse, previous line looks good = "normal" commandline
Need help in piping commands using xargs
I have several .tar.gz files that I need to list the folder content in a subdirectory.
For example,
a.tar.gz
b.tar.gz
c.tar.gz
The following command works great for each .tar.gz file but it's a pain to run the tar command for each file.
tar -tf... (11 Replies)
can someone please help me fix the below one liner?
what i want to do is run a command, and if that command is successful, pipe the output of the command to another command. i'd prefer not to use any temp files for this.
who -blahblah ; if ; then echo exit; fi | egrep username (2 Replies)
I've got a file that looks like this (spaces before first entries intentional):
12345650-000005000GL140227 ANNUAL HELC FEE EN
22345650-000005000GL140227 ANNUAL HELC FEE EN
32345650-000005000GL140227 ANNUAL HELC FEE EN
I want to read through the file line by line,... (6 Replies)
Hi
I want to read variables from one file and then set it as environment variable;
The text file is test.txt which contains
SPEED:1000
IP:172.26.126.11
My code is:
while read line; do
var1=`echo $line | awk 'BEGIN {FS=":"} { print $1 }'`
echo $var1
var2=`echo $line | awk 'BEGIN {FS=":"}... (8 Replies)
Hi
I am tryin to undertand piping
command1|command2
from what i learn output of cammand 2 is an intput for command 1 right?
If so .
What dose next sequence do
cat f1 >> f2 | grep '^'
I think it takes context of f1 and Concatenate's it to f2 and then looks for ....i don't know..... (7 Replies)
Using the below code I want to find all .sff files and extract them. This works but it seems very cheap. Is there a safer more efficient way to go about this?
#!/bin/bash
G1=(/home/dirone)
find ${G1} -type f -name \*.sff | xargs python /usr/local/bin/sff_extract.py (3 Replies)
Hello everybody,
I need some help here. I have a log file that gets updated every hour approximately.
I want to make some processing on each line which is added in the log file with a program written in PERL.
The problem is that I don't see anything when a line is added in the log file.
I... (6 Replies)
Please help with this simple example. I can not figure out how to do it. A file named “job” contains only this one line:var=5I need a script to read the job file and execute it as a command. This is my attempt (it does not work):#!/bin/sh
exec < job
echo "var = $var"output should read “var = 5”... (5 Replies)
Hi,
I want to read the variables and the values from the txt file and compare these values with the ones computed by script.
for ex:
say var.txt contains the variable names and their values:
one 1
two 2
three 3
The value of variables "one" "two" and "three" will be computed in the script... (3 Replies)
Hi. Im using cat to output the contents of a file, then piping it to my while read loop.In this loop variables get assigned values. However when i try to use the variables outside the loop their values has been reset.I understand about subshells etc. but I have no idea how to "preserve" the... (3 Replies)