Sponsored Content
Full Discussion: sed to extract HTML content
Top Forums UNIX for Advanced & Expert Users sed to extract HTML content Post 302294117 by stargazerr on Wednesday 4th of March 2009 01:29:42 PM
Old 03-04-2009
sed to extract HTML content

Hiya,

I am trying to extract a news article from a web page. The sed I have written brings back a lot of Javascript code and sometimes advertisments too. Can anyone please help with this one ??? I need to fix this sed so it picks up the article ONLY (don't worry about the title or date .. i got those using a separate sed) ..

The sed I am running is:

tr -d '\n' <03climate.html | sed -e 's/»//g' -e 's/.*nyt_text[^;]*;//' -e 's/<\/p>.*//g' -e 's/<[^>]*>//g' -e s'/[&][#]//g' -e 's/<[^>]*>//g' >> articletest

The file I am trying to extract from (03climate.html) and the result (articletest.txt) are both attached with this post .. Smilie

Thanks.
SG
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

mail: html content

hi guys, am required to prepare a report and mail it, to make it more appealing :p i wish to have content of mail in rich text format i.e html type with mailx how to specify the content type of mail body as html? Thanks in advance!!! rishi (2 Replies)
Discussion started by: RishiPahuja
2 Replies

2. UNIX for Dummies Questions & Answers

How do I extract text only from html file without HTML tag

I have a html file called myfile. If I simply put "cat myfile.html" in UNIX, it shows all the html tags like <a href=r/26><img src="http://www>. But I want to extract only text part. Same problem happens in "type" command in MS-DOS. I know you can do it by opening it in Internet Explorer,... (4 Replies)
Discussion started by: los111
4 Replies

3. Shell Programming and Scripting

sed to extract only floating point numbers from HTML

Hi All, I'm trying to extract some floating point numbers from within some HTML code like this: <TR><TD class='awrc'>Parse CPU to Parse Elapsd %:</TD><TD ALIGN='right' class='awrc'> 64.50</TD><TD class='awrc'>% Non-Parse CPU:</TD><TD ALIGN='right' class='awrc'> ... (2 Replies)
Discussion started by: pondlife
2 Replies

4. Shell Programming and Scripting

Extract URLs from HTML code using sed

Hello, i try to extract urls from google-search-results, but i have problem with sed filtering of html-code. what i wont is just list of urls thay apears between ........<p><a href=" and next following " in html code. here is my code, i use wget and pipelines to filtering. wget works, but... (13 Replies)
Discussion started by: L0rd
13 Replies

5. Shell Programming and Scripting

SED to extract HTML text data, not quite right!

I am attempting to extract weather data from the following website, but for the Victoria area only: Text Forecasts - Environment Canada I use this: sed -n "/Greater Victoria./,/Fraser Valley./p" But that phrasing does not sometimes get it all and think perhaps the website has more... (2 Replies)
Discussion started by: lagagnon
2 Replies

6. Shell Programming and Scripting

help with sed needed to extract content from html tags

Hi I've searched for it for few hours now and i can't seem to find anything working like i want. I've got webpage, saved in file par with form like this: <html><body><form name='sendme' action='http://example.com/' method='POST'> <textarea name='1st'>abc123def678</textarea> <textarea... (9 Replies)
Discussion started by: seb001
9 Replies

7. Shell Programming and Scripting

Print content between two html tags

Hi Expert, Is there any other way to print and write to a same filename the content between two html tags? Here the sample: cat file.html <div id="outline"> hello world<br> </div> <div id="container_faq"> test1<br> </div> <div class="widget_quick"> thead test<br> </div> ... (3 Replies)
Discussion started by: lxdorney
3 Replies

8. Shell Programming and Scripting

Awk/sed HTML extract

I'm extracting text between table tags in HTML <th><a href="/wiki/Buick_LeSabre" title="Buick LeSabre">Buick LeSabre</a></th> using this: awk -F "</*th>" '/<\/*th>/ {print $2}' auto2 > auto3 then this (text between a href): sed -e 's/\(<*>\)//g' auto3 > auto4 How to shorten this into one... (8 Replies)
Discussion started by: p1ne
8 Replies

9. Shell Programming and Scripting

Mailx with attachment and html content

Hi, Please see my code below i'm trying get an email send with attachment and html content in the body. Using the code below will put the encoding for attachment in the body as well SUBJECT="$(echo "XPI Monitoring "${tcnt}" transactions waiting \nContent-Type: text/html")" cat... (3 Replies)
Discussion started by: bhoshan
3 Replies

10. Shell Programming and Scripting

Convert content of file to HTML

Hi I have file like this: jack black 104 daniel nick 75 lily harm 2 albert 5 and need to convert it into the html table like this: NO.......name....family..... id 1...........jack.....black.....104 2..........daniel....nick.......75 3..........albert.................5 i mean... (5 Replies)
Discussion started by: indeed_1
5 Replies
OVERCHAN(8)						      System Manager's Manual						       OVERCHAN(8)

NAME
overchan - update the news overview database SYNOPSIS
overchan [ -D dir ] [ -c ] [ file... ] DESCRIPTION
Overchan reads article data from files or standard input if none are specified. (A single dash in the file list means to read standard input.) It uses this information to update the news overview database. Overchan is designed to be used by InterNetNews or the C News ``mkov'' packages to update the database as the articles come in. The database for each newsgroup is stored in a file named .overview in a newsgroup directory within the overview database tree. Overchan locks the database file (by locking an auxiliary file) before appending the new data. To purge data after articles have been expired, see expireover(8). By default, overchan processes its input as an INN overview stream written as a ``WO'' entry in the newsfeeds(5) file, for example: overview:*:Tc,WO:/usr/lib/news/bin/overchan This data consists of a line of text, separated into two parts by a tab. The first part is a list of all relative pathnames where the article has been written, with a single space between entries. The second part is the data to be written into the overview file, except that the initial article number is omitted. The data in the overview files should be expired by running expireover(8). This is normally done by adding the ``expireover'' flag to the news.daily(8) invocation. OPTIONS
-c To process the output of the mkov(8) program, use the ``-c'' flag. This format is described in the ``nov'' distribution. -D The ``-D'' flag can be used to specify where the databases are stored. The default directory is /var/spool/news/over.view. HISTORY
Written by Rob Robertson <rob@violet.berkeley.edu> and Rich $alz <rsalz@uunet.uu.net> for InterNetNews. This is revision 1.9, dated 1996/10/29. SEE ALSO
expireover(8), news.daily(8), newsfeeds(5), newsoverview(5), newsoverview(8). OVERCHAN(8)
All times are GMT -4. The time now is 04:09 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy