Learning scrapers, webcrawlers, search engines and CURL Post: 303019243

Sponsored Content

Operating Systems Linux Learning scrapers, webcrawlers, search engines and CURL Post 303019243 by TBotNik on Monday 25th of June 2018 05:14:08 PM

06-25-2018

Registered User

Thanks Neo!

Quote:

Originally Posted by Neo

I think you are better off to get web page content using PHP scripts and parse the files with REGEX.

If you Google around, I am sure you can find many sample PHP scripts that do most of what you want. This is very old technology and there is no need to reinvent the wheel parsing HTML data.

Neo, As I stated, still struggling with the terminology and concepts, so patience, I'm total newbie at using this technology, that's why I'm asking Qs as I don't even know where to focus, right now, to accomplish this.
Cheers!
OMR/TBNK

TBotNik

View Public Profile for TBotNik

Find all posts by TBotNik

3 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

I dont want to know any search engines

I just want to know where I can download it on this website plz

2. UNIX for Dummies Questions & Answers

Using cURL to save online search results

Hi, I'm attacking this from ignorance because I am not sure how to even ask the question. Here is the mission: I have a list of about 4,000 telephone numbers for past customers. I need to determine how many of these customers are still in business. Obviously, I could call all the numbers....

3. Shell Programming and Scripting

Checking status of engines using C-shell

I am relatively new to scripting. I am trying to develop a script that will 1. Source an executable file as an argument to the script that sets up the environment 2. Run a command "stat" that gives the status of 5 Engines running on the system 3. Check the status of the 5 Engines as either...

LEARN ABOUT DEBIAN

bb-findhost.cgi

BB-FINDHOST.CGI(1)					      General Commands Manual						BB-FINDHOST.CGI(1)

NAME

       bb-findhost.cgi - Xymon CGI script to find hosts

SYNOPSIS

       bb-findhost.cgi?host=REGEX

DESCRIPTION

       bb-findhost.cgi is invoked as a CGI script via the bb-findhost.sh CGI wrapper.

       bb-findhost.cgi is passed a QUERY_STRING environment variable with the "host=REGEX" parameter. The REGEX is a Posix regular expression (see
       regex(7) ) describing the hostnames to look for.  A trailing wildcard is assumed on all hostnames - e.g. requesting the hostname "www" will
       match any host whose name begins with "www".

       It then produces a single web page, listing all of the hosts that matched any of the hostnames, with links to the Xymon webpages where they
       are located.

       The output page lists hosts in the order they appear in the bb-hosts(5) file.

       A sample web page implementing the search facility is included with bbgen, you access it via the URL /bb/help/bb-findhost.html.

OPTIONS

       --env=FILENAME
	      Loads the environment from FILENAME before executing the CGI.

FILES

       $BBHOME/web/findhost_header
	      HTML header file for the generated web page

       $BBHOME/web/findhost_footer
	      HTML footer file for the generated web page

       $BBHOME/web/findhost_form
	      Query form displayed when bb-findhost.cgi is called with no parameters.

ENVIRONMENT VARIABLES

       BBHOSTS
	      bb-findhost.cgi uses the BBHOSTS environment variable to find the bb-hosts file listing all known hosts and their page locations.

       BBHOME Used to locate the template files for the generated web pages.

SEE ALSO

       bbgen(1), bb-hosts(5), hobbitserver.cfg(5)

Xymon							    Version 4.2.3:  4 Feb 2009						BB-FINDHOST.CGI(1)