Learning scrapers, webcrawlers, search engines and CURL Post: 303019243

Sponsored Content

Operating Systems Linux Learning scrapers, webcrawlers, search engines and CURL Post 303019243 by TBotNik on Monday 25th of June 2018 05:14:08 PM

06-25-2018

Registered User

Thanks Neo!

Quote:

Originally Posted by Neo

I think you are better off to get web page content using PHP scripts and parse the files with REGEX.

If you Google around, I am sure you can find many sample PHP scripts that do most of what you want. This is very old technology and there is no need to reinvent the wheel parsing HTML data.

Neo, As I stated, still struggling with the terminology and concepts, so patience, I'm total newbie at using this technology, that's why I'm asking Qs as I don't even know where to focus, right now, to accomplish this.
Cheers!
OMR/TBNK

TBotNik

View Public Profile for TBotNik

Find all posts by TBotNik

3 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

I dont want to know any search engines

I just want to know where I can download it on this website plz

2. UNIX for Dummies Questions & Answers

Using cURL to save online search results

Hi, I'm attacking this from ignorance because I am not sure how to even ask the question. Here is the mission: I have a list of about 4,000 telephone numbers for past customers. I need to determine how many of these customers are still in business. Obviously, I could call all the numbers....

3. Shell Programming and Scripting

Checking status of engines using C-shell

I am relatively new to scripting. I am trying to develop a script that will 1. Source an executable file as an argument to the script that sets up the environment 2. Run a command "stat" that gives the status of 5 Engines running on the system 3. Check the status of the 5 Engines as either...

LEARN ABOUT DEBIAN

pinot-search

PINOT-SEARCH(1) 						   User Commands						   PINOT-SEARCH(1)

NAME

       pinot-search - Query search engines from the command-line

SYNOPSIS

       pinot-search [OPTIONS] SEARCHENGINETYPE SEARCHENGINENAME|SEARCHENGINEOPTION QUERYINPUT

DESCRIPTION

       pinot-search - Query search engines from the command-line

OPTIONS

       -d, --datefirst
	      sort by date then by relevance

       -h, --help
	      display this help and exit

       -l, --locationonly
	      only show the location of each result

       -m, --max
	      maximum number of results (default 10)

       -r, --storedquery
	      query input is the name of a stored query

       -s, --stemming
	      stemming language (in English)

       -c, --tocsv
	      file to export results in CSV format to

       -x, --toxml
	      file to export results in XML format to

       -v, --version
	      output version information and exit

       Supported search engine types are : 'opensearch' 'sherlock' 'xapian'

EXAMPLES

       pinot-search opensearch /usr/share/pinot/engines/KrustyDescription.xml "clowns"

       pinot-search --max 20 sherlock /usr/share/pinot/engines/Bozo.src "clowns"

       pinot-search googleapi mygoogleapikey "clowns"

       pinot-search xapian ~/.pinot/index "label:Clowns"

       pinot-search --stemming english xapian somehostname:12345 "clowning"

REPORTING BUGS

       Report bugs to fabrice.colin@gmail.com

       This   is   free   software.    You   may   redistribute   copies   of	it   under   the   terms   of	the  GNU  General  Public  License
       <http://www.gnu.org/licenses/old-licenses/gpl-2.0.html>.
       There is NO WARRANTY, to the extent permitted by law.

pinot-search - pinot 1.0					     June 2012							   PINOT-SEARCH(1)