Sponsored Content
Operating Systems Linux Learning scrapers, webcrawlers, search engines and CURL Post 303019102 by Neo on Friday 22nd of June 2018 11:16:30 PM
Old 06-23-2018
Quote:
Originally Posted by TBotNik
  • Text only vs regular brower: which is best?
  • wget vs php fileopen vs CURL: Which is best?
  • HTML tag find/parse: Are there libraries that effectively do this?
  • HTML tag find/parse: Is REGEX the best way to parse these? Where are examples?
  • Checking for the new meta-tags of:
I think you are better off to get web page content using PHP scripts and parse the files with REGEX.

If you Google around, I am sure you can find many sample PHP scripts that do most of what you want. This is very old technology and there is no need to reinvent the wheel parsing HTML data.
 

3 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

I dont want to know any search engines

I just want to know where I can download it on this website plz (1 Reply)
Discussion started by: memattmyself
1 Replies

2. UNIX for Dummies Questions & Answers

Using cURL to save online search results

Hi, I'm attacking this from ignorance because I am not sure how to even ask the question. Here is the mission: I have a list of about 4,000 telephone numbers for past customers. I need to determine how many of these customers are still in business. Obviously, I could call all the numbers.... (0 Replies)
Discussion started by: jccbin
0 Replies

3. Shell Programming and Scripting

Checking status of engines using C-shell

I am relatively new to scripting. I am trying to develop a script that will 1. Source an executable file as an argument to the script that sets up the environment 2. Run a command "stat" that gives the status of 5 Engines running on the system 3. Check the status of the 5 Engines as either... (0 Replies)
Discussion started by: paslas
0 Replies
Strip(3pm)						User Contributed Perl Documentation						Strip(3pm)

NAME
HTML::Strip - Perl extension for stripping HTML markup from text. SYNOPSIS
use HTML::Strip; my $hs = HTML::Strip->new(); my $clean_text = $hs->parse( $raw_html ); $hs->eof; DESCRIPTION
This module simply strips HTML-like markup from text in a very quick and brutal manner. It could quite easily be used to strip XML or SGML from text as well; but removing HTML markup is a much more common problem, hence this module lives in the HTML:: namespace. It is written in XS, and thus about five times quicker than using regular expressions for the same task. It does not do any syntax checking (if you want that, use HTML::Parser), instead it merely applies the following rules: 1. Anything that looks like a tag, or group of tags will be replaced with a single space character. Tags are considered to be anything that starts with a "<" and ends with a ">"; with the caveat that a ">" character may appear in either of the following without ending the tag: Quote Quotes are considered to start with either a "'" or a """ character, and end with a matching character not preceded by an even number or escaping slashes (i.e. """ does not end the quote but "\\"" does). Comment If the tag starts with an exclamation mark, it is assumed to be a declaration or a comment. Within such tags, ">" characters do not end the tag if they appear within pairs of double dashes (e.g. "<!-- <a href="old.htm">old page</a> -->" would be stripped completely). 2. Anything the appears within so-called strip tags is stripped as well. By default, these tags are "title", "script", "style" and "applet". HTML::Strip maintains state between calls, so you can parse a document in chunks should you wish. If one chunk ends half-way through a tag, quote, comment, or whatever; it will remember this, and expect the next call to parse to start with the remains of said tag. If this is not going to be the case, be sure to call $hs->eof() between calls to $hs->parse(). METHODS new() Constructor. Can optionally take a hash of settings (with keys corresponsing to the "set_" methods below). For example, the following is a valid constructor: my $hs = HTML::Strip->new( striptags => [ 'script', 'iframe' ], emit_spaces => 0 ); parse() Takes a string as an argument, returns it stripped of HTML. eof() Resets the current state information, ready to parse a new block of HTML. clear_striptags() Clears the current set of strip tags. add_striptag() Adds the string passed as an argument to the current set of strip tags. set_striptags() Takes a reference to an array of strings, which replace the current set of strip tags. set_emit_spaces() Takes a boolean value. If set to false, HTML::Strip will not attempt any conversion of tags into spaces. Set to true by default. set_decode_entities() Takes a boolean value. If set to false, HTML::Strip will decode HTML entities. Set to true by default. LIMITATIONS Whitespace Despite only outputting one space character per group of tags, and avoiding doing so when tags are bordered by spaces or the start or end of strings, HTML::Strip can often output more than desired; such as with the following HTML: <h1> HTML::Strip </h1> <p> <em> <strong> fast, and brutal </strong> </em> </p> Which gives the following output: " HTML::Strip fast, and brutal " Thus, you may want to post-filter the output of HTML::Strip to remove excess whitespace (for example, using "tr/ / /s;"). (This has been improved since previous releases, but is still an issue) HTML Entities HTML::Strip will only attempt decoding of HTML entities if HTML::Entities is installed. EXPORT None by default. AUTHOR
Alex Bowley <kilinrax@cpan.org> SEE ALSO
perl, HTML::Parser, HTML::Entities perl v5.14.2 2011-11-15 Strip(3pm)
All times are GMT -4. The time now is 08:32 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy