Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

SED to extract HTML text data, not quite right!

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting SED to extract HTML text data, not quite right!
# 1  
Old 01-30-2010
SED to extract HTML text data, not quite right!
I am attempting to extract weather data from the following website, but for the Victoria area only:

Text Forecasts - Environment Canada

I use this:

Code:
 sed -n "/Greater Victoria./,/Fraser Valley./p"

But that phrasing does not sometimes get it all and think perhaps the website has more than one linefeed, carriage return, whatever, that messes up my coding. Any ideas appreciated.

Larry
# 2  
Old 01-31-2010
Hi.

If I copy the text from the site, and modify your statement slightly:

Code:
sed -n "/Greater Victoria./,/^$/p" file1

I get this output:
Code:
Greater Victoria.
Monday..Cloudy. High 8.
Tuesday..Cloudy with 60 percent chance of showers. Low plus 3.
 High 7.
Wednesday..Cloudy with 40 percent chance of flurries or rain showers.
 Low plus 3. High 8.
Thursday..Cloudy with 60 percent chance of rain showers mixed with
 flurries. Low plus 3. High 8.
Friday..Cloudy with 30 percent chance of showers. Low plus 3. High 9.
Normals for the period..Low plus 2. High 7.

It looks exactly the same as on the website. Is it not want you wanted?
# 3  
Old 01-31-2010
Google weather api. There's a bunch of them.

For example, I looked at Yahoo! Weather RSS Feed to come up with this for Victoria weather:
Code:
curl -s "http://weather.yahooapis.com/forecastrss?w=9848&u=c"|sed "1,/Current Conditions/d"|head -n1

The text is simpler and more predictable than a regular web site, so it shouldn't be too difficult to parse out the interesting items.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extract text from html using perl or awk

I am trying to extract text after keywords fron an html file. The keywords are reportLink":, "barcodedSamples": {", "barcodedSamples": {". Both the perl and awk run but the output is just the entire index.html not the desired output. Also for the reportLink": only the text after the second / until... (5 Replies)
Discussion started by: cmccabe
5 Replies

2. Shell Programming and Scripting

Awk/sed HTML extract

I'm extracting text between table tags in HTML <th><a href="/wiki/Buick_LeSabre" title="Buick LeSabre">Buick LeSabre</a></th> using this: awk -F "</*th>" '/<\/*th>/ {print $2}' auto2 > auto3 then this (text between a href): sed -e 's/\(<*>\)//g' auto3 > auto4 How to shorten this into one... (8 Replies)
Discussion started by: p1ne
8 Replies

3. Shell Programming and Scripting

awk -- Extract data from html within multiple tags as reference

Hi, I'm trying to get some data from an html file, but the problem is before it can extract the information I have multiple patterns that need to be passed through. https://www.unix.com/shell-programming-scripting/150711-extract-data-awk-html-files.html Is a similar problem. The only... (5 Replies)
Discussion started by: counfhou
5 Replies

4. Shell Programming and Scripting

extract complex data from html table rows

I have bash, awk, and sed available on my portable device. I need to extract 10 fields from each table row from a web page that looks like this: </tr> <tr> <td>28 Apr</td> <td><a... (6 Replies)
Discussion started by: rickgtx
6 Replies

5. Shell Programming and Scripting

help with sed needed to extract content from html tags

Hi I've searched for it for few hours now and i can't seem to find anything working like i want. I've got webpage, saved in file par with form like this: <html><body><form name='sendme' action='http://example.com/' method='POST'> <textarea name='1st'>abc123def678</textarea> <textarea... (9 Replies)
Discussion started by: seb001
9 Replies

6. Shell Programming and Scripting

extract data with awk from html files

Hello everyone, I'm new to this forum and i am new as a shell scripter. my problem is to have html files in a directory and I would like to extract from these some data that lies between two different lines Here's my situation <td align="default"> oxidizability (mg / l): data_to_extract... (6 Replies)
Discussion started by: sbobotex
6 Replies

7. Shell Programming and Scripting

Extract URLs from HTML code using sed

Hello, i try to extract urls from google-search-results, but i have problem with sed filtering of html-code. what i wont is just list of urls thay apears between ........<p><a href=" and next following " in html code. here is my code, i use wget and pipelines to filtering. wget works, but... (13 Replies)
Discussion started by: L0rd
13 Replies

8. UNIX for Advanced & Expert Users

sed to extract HTML content

Hiya, I am trying to extract a news article from a web page. The sed I have written brings back a lot of Javascript code and sometimes advertisments too. Can anyone please help with this one ??? I need to fix this sed so it picks up the article ONLY (don't worry about the title or date .. i got... (2 Replies)
Discussion started by: stargazerr
2 Replies

9. UNIX for Dummies Questions & Answers

extract data from html tables

hi i need to use unix to extract data from several rows of a table coded in html. I know that rows within a table have the tags <tr> </tr> and so i thought that my first step should be to to delete all of the other html code which is not contained within these tags. i could then use this method... (8 Replies)
Discussion started by: Streetrcr
8 Replies

10. UNIX for Dummies Questions & Answers

How do I extract text only from html file without HTML tag

I have a html file called myfile. If I simply put "cat myfile.html" in UNIX, it shows all the html tags like <a href=r/26><img src="http://www>. But I want to extract only text part. Same problem happens in "type" command in MS-DOS. I know you can do it by opening it in Internet Explorer,... (4 Replies)
Discussion started by: los111
4 Replies
Login or Register to Ask a Question