Linux

RedHat, Ubuntu, SUSE, Fedora, Debian, Mandriva, Slackware, Gentoo linux, PCLinuxOS. All Linux questions here!

Learning scrapers, webcrawlers, search engines and CURL

👤 Login to reply

    #1  
Old 4 Weeks Ago
TBotNik TBotNik is offline
Registered User
 
Join Date: Jun 2018
Last Activity: 21 July 2018, 12:43 PM EDT
Posts: 9
Thanks: 0
Thanked 0 Times in 0 Posts
Learning scrapers, webcrawlers, search engines and CURL

All,

I'm trying to learn scrapers, webcrawlers, search engines and CURL. I've chosen to interrogate the
following sites:

  • Manta,
  • SuperPages,
  • Yellow Book,
  • Yellow Pages.


These show organizations/businesses by search type/category, so effective in
finding potential clients.

Since I only run Linux, I have the following questions as I consider my approaches:

  • Text only vs regular browser: which is best?
  • wget vs php fileopen vs CURL: Which is best?
  • HTML tag find/parse: Are there libraries that effectively do this?
  • HTML tag find/parse: Is REGEX the best way to parse these? Where are examples?
  • Checking for the new meta-tags of:

Code:
<meta name="category" content="">
<meta name="subcategory" content="">

Where the content is the category or subcategory name

EX:
Code:
<meta name="category" content="technology">
<meta name="subcategory" content="retail">
<meta name="subcategory" content="electronics">
<meta name="subcategory" content="computers">
<meta name="subcategory" content="laptops">
<meta name="subcategory" content="20+ inch screen">

or

Code:
<meta name="category" content="technology">
<meta name="subcategory" content="manufacturer">
<meta name="subcategory" content="electronics">
<meta name="subcategory" content="computers">
<meta name="subcategory" content="laptops">
<meta name="subcategory" content="20+ inch screen">

I'm processing these into a temp DB, but final repository for the data will be
both SugarCRM CE and PHPList.

There is differing information available on each of these at differing sites and
very confusing, as many businesses
will show on all these, but some listing have website and email and others do not
so will be updating existing records with blank fields, when the info is found on
one of the other sites.

Trying to avoid the time consuming process of calling each, however the level of
detail I need for my app I'm writing may still require the calls. I'll see where
I'm at once the intial automation is finished.

I look at all the HOWTOs and they are like GREEk to me, and so not comprehending what I'm
reading. Going to need some help with the comprehension before it all registers and
I come to a basic understanding of both terms and techniques, so sorry to bother you
with a NEWBIE level understanding of this!

Cheers!

OMR/TBNK

Last edited by TBotNik; 4 Weeks Ago at 07:18 PM..
Sponsored Links
    #2  
Old 4 Weeks Ago
Neo's Unix or Linux Image
Neo Neo is offline Forum Staff  
Administrator
 
Join Date: Sep 2000
Last Activity: 21 July 2018, 2:42 PM EDT
Location: Asia pacific region
Posts: 14,579
Thanks: 1,032
Thanked 1,438 Times in 698 Posts
Quote:
Originally Posted by TBotNik View Post
  • Text only vs regular brower: which is best?
  • wget vs php fileopen vs CURL: Which is best?
  • HTML tag find/parse: Are there libraries that effectively do this?
  • HTML tag find/parse: Is REGEX the best way to parse these? Where are examples?
  • Checking for the new meta-tags of:
I think you are better off to get web page content using PHP scripts and parse the files with REGEX.

If you Google around, I am sure you can find many sample PHP scripts that do most of what you want. This is very old technology and there is no need to reinvent the wheel parsing HTML data.
Sponsored Links
    #3  
Old 3 Weeks Ago
TBotNik TBotNik is offline
Registered User
 
Join Date: Jun 2018
Last Activity: 21 July 2018, 12:43 PM EDT
Posts: 9
Thanks: 0
Thanked 0 Times in 0 Posts
Thanks Neo!

Quote:
Originally Posted by Neo View Post
I think you are better off to get web page content using PHP scripts and parse the files with REGEX.

If you Google around, I am sure you can find many sample PHP scripts that do most of what you want. This is very old technology and there is no need to reinvent the wheel parsing HTML data.
Neo, As I stated, still struggling with the terminology and concepts, so patience, I'm total newbie at using this technology, that's why I'm asking Qs as I don't even know where to focus, right now, to accomplish this.
Cheers!
OMR/TBNK
Sponsored Links
👤 Login to reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Checking status of engines using C-shell paslas Shell Programming and Scripting 0 04-20-2012 02:39 PM
Using cURL to save online search results jccbin UNIX for Dummies Questions & Answers 0 06-02-2010 03:44 PM
I dont want to know any search engines memattmyself Shell Programming and Scripting 1 05-09-2002 11:09 AM



All times are GMT -4. The time now is 07:26 PM.

Unix & Linux Forums Content Copyrightę1993-2018. All Rights Reserved.
×
UNIX.COM Login
Username:
Password:  
Show Password





Not a Forum Member?
Forgot Password?