06-06-2013
Grep text matching problem with script which checks if web page contains text.
I wrote a Bash script which checks to see if a text string exists on a web page and then sends me an email if it does (or does not e.g. "Out of stock"). I run it from my crontab, it's quite handy from time to time and I've been using it for a few years now.
The script uses wget to download an url and then uses grep to match the text string which I lift from the original HTML in case of markups or new lines (though the latter has never actually occured).
Today I added the text to look for and it did not get matched even though it was present in the HTML. When I copied and pasted the, identically looking, text from the downloaded HTML file into the script and tried that it worked perfectly. [Problem solved in this case but I'd like to fix things properly.]
So the character encoding seems to be the problem. Or so I thought! Grep uses utf-8 but it turned out the source HTML was utf-8 as well. What a pain, so no easy fix by using iconv to convert all downloaded files to utf-8.
Anyone know what might be happening here and what I need to do to fix this?
Many thanks.
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
Is there a way to slowly scroll the output of a file instead of page or cat ?
Instead of one page at a time, I would like to slowly scroll the displayed output of the file. (12 Replies)
Discussion started by: darthur
12 Replies
2. UNIX for Dummies Questions & Answers
How do I use the grep command to take mutiple text files in a folder and make one huge text file out of them. I'm using Mac OS X and can not find a text tool that does it so I figured I'd resort to the BSD Unix CLI for a solution... there are 5,300 files that I want to write to one huge file so... (7 Replies)
Discussion started by: coppertone
7 Replies
3. Shell Programming and Scripting
Dear all,
I am calling a korn shell script(CGI script) by a web-page. This shell script do some checking in a unix file and return true or false. Now within the same script, If it returns true then I want to redirect to another web-page stored in htdocs directory.
Example: Login page sends a... (3 Replies)
Discussion started by: ravi18s
3 Replies
4. UNIX for Dummies Questions & Answers
Hi,
Say there is a web page that contains just text only - that is, even the source code is just the text itself, nothing more. An example would be "http://mynasadata.larc.nasa.gov/docs/ocean_percent.txt"
Is there a UNIX command that would allow me to download this text and store it in a... (1 Reply)
Discussion started by: Breanne
1 Replies
5. HP-UX
First, let me state that I am completely out of my realm with this.
I have a server running HPUX. I'm not even sure if this can be considered a UNIX question and for that let me apologize in advance.
I need to create a web page where a client can input 2 variables (i.e. date and phone number).... (0 Replies)
Discussion started by: grinds
0 Replies
6. Shell Programming and Scripting
Hello all,
I am in need of assistance in creating a script that will remove a specified block of text from multiple .htaccess files. (roughly 1000 files)
I am attempting to help with a project to clean up a linux server that has a series of unwanted url rewrites in place, as well as some... (4 Replies)
Discussion started by: boxx
4 Replies
7. Shell Programming and Scripting
Hi folks...
Relatively new to scripting, but really struggling with something that will no doubt be second nature to most people on here:
Trying to get an exact match on $sub, where sub is an ip address.
subnet ()
{
clear
while true
do
... (18 Replies)
Discussion started by: CiCa
18 Replies
8. Shell Programming and Scripting
All,
I require a script that grabs some text from the gitHub API and will grep (or other function) for a string a characters that starts with (") quotes followed by two letters, may contain a pipe |, and ending with ) . What i have so far is below but it's not returning anything.
... (4 Replies)
Discussion started by: ChocoTaco
4 Replies
9. Shell Programming and Scripting
Hello.
I'm trying to create a web page which the presentation is as follows:
1 °) at the top of page an image
2 °) below the text
3 °) to complete a php function that returns information.
I tried different things but none work.
Script 1:
<!DOCTYPE html>
<html>
<head>
<style>
div {
... (5 Replies)
Discussion started by: jcdole
5 Replies
10. Shell Programming and Scripting
I need help to make a script for Ubuntu to OSCam that copy the text on this website that only contains "C: ip port randomUSERNAME password" and want to exclude the text "C:" and replace the rest with the old in my test.server file.
(line 22) device = ip,port
(line 23) user =... (6 Replies)
Discussion started by: baxarn
6 Replies
aha(1) Ansi HTML Adapter aha(1)
NAME
aha - Ansi HTML Adapter
SYNOPSIS
aha [options] [-f datei]
DESCRIPTION
aha takes SGR-colored Input and prints W3C conform HTML-Code.
aha reads the Input from a file or stdin and writes HTML-Code to stdout.
OPTIONS
--help , -h , -?
A help like this
--black , -b
Black Background and white "standard color"
--pink , -p
Pink Background
--iso X , -i X
Uses ISO 8859-X instead of utf-8. X must be 1..16
--title X , -t X
Gives the html output the title
--line-fix , -l
Uses a fix for inputs using control sequences to change the cursor position like htop. It's a hot fix, it may not work with any pro-
gram like htop. (See EXAMPLE)
EXAMPLE
aha --help | aha --black --title "the awesome aha help"> aha-help.htm
Creates an HTML file with the help of aha with black background
colordiff oldfile.c newfile.c | aha > colordiff.htm
Creates an HTML file with a colorful diff-output of two files "oldfile.c" and "newfile.c" with white background
ls --color=always | aha --pink > ls.htm
Creates an HTML file with a colorful ls-output with pink background.
echo a | htop | aha --black --line-fix > htop.htm
Creates an HTML file with the output of htop. You have to use --line-fix due the other new-line-commands htop uses.
AUTHOR
Copyleft Alexander Matthes aka Ziz 2011
zizsdl@googlemail.com
SEE ALSO
http://ziz.delphigl.com/tool_aha.php
August 31, 2011 aha(1)