|
|
Sponsored Content
Top Forums
UNIX for Advanced & Expert Users
sed to extract HTML content
Post 302294117 by stargazerr on Wednesday 4th of March 2009 01:29:42 PM
03-04-2009
sed to extract HTML content
Hiya,
I am trying to extract a news article from a web page. The sed I have written brings back a lot of Javascript code and sometimes advertisments too. Can anyone please help with this one ??? I need to fix this sed so it picks up the article ONLY (don't worry about the title or date .. i got those using a separate sed) ..
The sed I am running is:
tr -d '\n' <03climate.html | sed -e 's/»//g' -e 's/.*nyt_text[^;]*;//' -e 's/<\/p>.*//g' -e 's/<[^>]*>//g' -e s'/[&][#]//g' -e 's/<[^>]*>//g' >> articletest
The file I am trying to extract from (03climate.html) and the result (articletest.txt) are both attached with this post ..
Thanks.
SG
I am trying to extract a news article from a web page. The sed I have written brings back a lot of Javascript code and sometimes advertisments too. Can anyone please help with this one ??? I need to fix this sed so it picks up the article ONLY (don't worry about the title or date .. i got those using a separate sed) ..
The sed I am running is:
tr -d '\n' <03climate.html | sed -e 's/»//g' -e 's/.*nyt_text[^;]*;//' -e 's/<\/p>.*//g' -e 's/<[^>]*>//g' -e s'/[&][#]//g' -e 's/<[^>]*>//g' >> articletest
The file I am trying to extract from (03climate.html) and the result (articletest.txt) are both attached with this post ..
Thanks.
SG
|
|
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
hi guys,
am required to prepare a report and mail it, to make it more appealing :p i wish to have content of mail in rich text format i.e html type
with mailx how to specify the content type of mail body as html?
Thanks in advance!!!
rishi (2 Replies)
Discussion started by: RishiPahuja
2 Replies
2. UNIX for Dummies Questions & Answers
I have a html file called myfile. If I simply put "cat myfile.html" in UNIX, it shows all the html tags like <a href=r/26><img src="http://www>. But I want to extract only text part.
Same problem happens in "type" command in MS-DOS.
I know you can do it by opening it in Internet Explorer,... (4 Replies)
Discussion started by: los111
4 Replies
3. Shell Programming and Scripting
Hi All,
I'm trying to extract some floating point numbers from within some HTML code like this:
<TR><TD class='awrc'>Parse CPU to Parse Elapsd %:</TD><TD ALIGN='right' class='awrc'> 64.50</TD><TD class='awrc'>% Non-Parse CPU:</TD><TD ALIGN='right' class='awrc'> ... (2 Replies)
Discussion started by: pondlife
2 Replies
4. Shell Programming and Scripting
Hello,
i try to extract urls from google-search-results, but i have problem with sed filtering of html-code.
what i wont is just list of urls thay apears between ........<p><a href=" and next following " in html code.
here is my code, i use wget and pipelines to filtering. wget works, but... (13 Replies)
Discussion started by: L0rd
13 Replies
5. Shell Programming and Scripting
I am attempting to extract weather data from the following website, but for the Victoria area only:
Text Forecasts - Environment Canada
I use this:
sed -n "/Greater Victoria./,/Fraser Valley./p"
But that phrasing does not sometimes get it all and think perhaps the website has more... (2 Replies)
Discussion started by: lagagnon
2 Replies
6. Shell Programming and Scripting
Hi
I've searched for it for few hours now and i can't seem to find anything working like i want. I've got webpage, saved in file par with form like this:
<html><body><form name='sendme' action='http://example.com/' method='POST'>
<textarea name='1st'>abc123def678</textarea>
<textarea... (9 Replies)
Discussion started by: seb001
9 Replies
7. Shell Programming and Scripting
Hi Expert,
Is there any other way to print and write to a same filename the content between two html tags?
Here the sample:
cat file.html
<div id="outline">
hello world<br>
</div>
<div id="container_faq">
test1<br>
</div>
<div class="widget_quick">
thead test<br>
</div>
... (3 Replies)
Discussion started by: lxdorney
3 Replies
8. Shell Programming and Scripting
I'm extracting text between table tags in HTML
<th><a href="/wiki/Buick_LeSabre" title="Buick LeSabre">Buick LeSabre</a></th>
using this:
awk -F "</*th>" '/<\/*th>/ {print $2}' auto2 > auto3
then this (text between a href):
sed -e 's/\(<*>\)//g' auto3 > auto4
How to shorten this into one... (8 Replies)
Discussion started by: p1ne
8 Replies
9. Shell Programming and Scripting
Hi, Please see my code below i'm trying get an email send with attachment and html content in the body.
Using the code below will put the encoding for attachment in the body as well
SUBJECT="$(echo "XPI Monitoring "${tcnt}" transactions waiting \nContent-Type: text/html")"
cat... (3 Replies)
Discussion started by: bhoshan
3 Replies
10. Shell Programming and Scripting
Hi
I have file like this:
jack black 104
daniel nick 75
lily harm 2
albert 5
and need to convert it into the html table like this:
NO.......name....family..... id
1...........jack.....black.....104
2..........daniel....nick.......75
3..........albert.................5
i mean... (5 Replies)
Discussion started by: indeed_1
5 Replies
LEARN ABOUT SUSE
nwbpset
NWBPSET(1) nwbpset NWBPSET(1) NAME
nwbpset - Create a bindery property or set its value SYNOPSIS
nwbpset [ -h ] [ -S server ] [ -U user name ] [ -P password | -n ] [ -C ] DESCRIPTION
nwbpset Reads a property specification from the standard input and creates and sets the corresponding property. The format is determined by the output of 'nwbpvalues -c'. nwbpset will hopefully become an important part of the bindery management suite of ncpfs, together with As another example, look at the following command line: nwbpvalues -t 1 -o supervisor -p user_defaults -c | sed '2s/.*/ME/'| sed '3s/.*/LOGIN_CONTROL/'| nwbpset With this command, the property user_defaults of the user object 'supervisor' is copied into the property login_control of the user object 'me'. nwbpvalues -t 1 -o me -p login_control -c | sed '9s/.*/ff/'| nwbpset This command disables the user object me. Feel free to contribute other examples! nwbpset looks up the file $HOME/.nwclient to find a file server, a user name and possibly a password. See nwclient(5) for more information. Please note that the access permissions of $HOME/.nwclient MUST be 600 for security reasons. OPTIONS
-h -h is used to print out a short help text. -S server server is the name of the server you want to use. -U user user is the user name to use for login. -P password password is the password to use for login. If neither -n nor -P are given, and the user has no open connection to the server, nwbpset prompts for a password. -n -n should be given if no password is required for the login. -C By default, passwords are converted to uppercase before they are sent to the server, because most servers require this. You can turn off this conversion by -C. AUTHORS
nwbpset was written by Volker Lendecke. See the Changes file of ncpfs for other contributors. nwbpset 8/7/1996 NWBPSET(1)