Learning scrapers, webcrawlers, search engines and CURL Post: 303019102

Sponsored Content

Operating Systems Linux Learning scrapers, webcrawlers, search engines and CURL Post 303019102 by Neo on Friday 22nd of June 2018 11:16:30 PM

06-23-2018

Administrator

Quote:

Originally Posted by TBotNik

Text only vs regular brower: which is best?
wget vs php fileopen vs CURL: Which is best?
HTML tag find/parse: Are there libraries that effectively do this?
HTML tag find/parse: Is REGEX the best way to parse these? Where are examples?
Checking for the new meta-tags of:

I think you are better off to get web page content using PHP scripts and parse the files with REGEX.

If you Google around, I am sure you can find many sample PHP scripts that do most of what you want. This is very old technology and there is no need to reinvent the wheel parsing HTML data.

Neo

View Public Profile for Neo

Visit Neo's homepage!

Find all posts by Neo

3 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

I dont want to know any search engines

I just want to know where I can download it on this website plz

2. UNIX for Dummies Questions & Answers

Using cURL to save online search results

Hi, I'm attacking this from ignorance because I am not sure how to even ask the question. Here is the mission: I have a list of about 4,000 telephone numbers for past customers. I need to determine how many of these customers are still in business. Obviously, I could call all the numbers....

3. Shell Programming and Scripting

Checking status of engines using C-shell

I am relatively new to scripting. I am trying to develop a script that will 1. Source an executable file as an argument to the script that sets up the environment 2. Run a command "stat" that gives the status of 5 Engines running on the system 3. Check the status of the 5 Engines as either...

LEARN ABOUT ULTRIX

group

group(5) File Formats Manual group(5)

Name
group - group file

Description
The file is an ASCII file that contains the following information for each group:

Group name
Encrypted password
Numerical group ID
Comma-separated list of all users allowed in the group

Each group name is separated from the next by a new line. The fields are separated by colons. If the password field is null, no password
is demanded.

Because of the encrypted passwords, it can and does have general read permission and can be used, for example, to map numerical group IDs
to names.

The database can be distributed in a network by a naming service, such as Yellow Pages or BIND/Hesiod. See the Guide to the Yellow Pages
Service or the Guide to the BIND/Hesiod Service for setup information.

Restrictions
The command will not change the passwords.

Increasing the number of groups that a user is in beyond 8 can affect services that use SUN RPC. The version of SUN RPC shipped with ULTRX
V4.2 supports up to 32 groups. Prior versions supported users being in up to 8 groups. Users who increase the number of groups that they
are a member of to more than 8 will not be able to NFS mount file systems from ULTRIX servers older than V4.2. If root group membership is
increased to more than 8 groups on a V4.2 system, the YP service may not work if that V4.2 system is in a mixed YP server environment where
there are both V4.2 and previous releases of YP servers. Again, this incompatibility can only occur if a user's or root's group membership
is increased to more than 8 groups.

Files
See Also
passwd(1), setgroups(2), crypt(3), getgrent(3), initgroups(3x), passwd(5)
Guide to the BIND/Hesiod Service
Guide to the Yellow Pages Service

group(5)