I have a list of 10 million page urls. I want those pages scraped and saved in the mysql database as raw html.
I own a Linux VPS server with 1GB RAM and WHM/cPanel.
I would like to scrape at least 100,000 urls in 24 hours.
So can anyone give me some sample shell scripting code? (4 Replies)
I am using an html form and a php upload script to upload files.
HTML form
<table width="500" border="0" align="center" cellpadding="0" cellspacing="1" bgcolor="#CCCCCC">
<tr>
<form action="upload_ac.php" method="post" enctype="multipart/form-data" name="form1" id="form1">
<td>
<table... (1 Reply)
Hi
I would like to convert standard online man pages from my solaris10 system into html form to publish it on my webpage.
How this can be done in Sol10 ?
thx for help. (2 Replies)
I am sure this is easy but I can't figure it out...
Here is the form.
<?php
$searchString = $_POST;
if (!isset($_POST))
?>
<html>
<head>
<title>Personal INFO</title>
</head>
<body>
<form method="post" action="search.php">
<input type="text" size="20" maxlength="20" name="search">... (1 Reply)
I have an HTML form that sends email to a large list of users one at a time by matching an email address in peoplesoft to their username. It works great, except that special characters are converted to %## format. Is there a library of these I can use to sed them back (yes this is a crappy UNIX... (1 Reply)
I wrote a script to automate user account verification against peoplesoft. Now I want to make it available to my peers via the web. It is running on Solaris.
I have the form written, but am not sure how to make it work. I think the form should call a perl cgi when submitted. The cgi should call... (7 Replies)
I am currently able to use the $QUERY_STRING variable and simply cut out what I need to be assigned as variables within the shell script. However, I've been able to use the "name" value assigned within the FORM(HTML) as a variable when I use perl. Why is it that ksh doesn't read the "name" in as... (1 Reply)
MYSQLDIFF(1p) User Contributed Perl Documentation MYSQLDIFF(1p)NAME
mysql-schema-diff - compare MySQL database schemas
SYNOPSIS
mysql-schema-diff [B<options>] B<database1> B<database2>
mysql-schema-diff --help
DESCRIPTION
mysql-schema-diff is a Perl script front-end to the CPAN <http://www.perl.com/CPAN> module MySQL::Diff
<http://search.cpan.org/search?module=MySQL::Diff> which compares the data structures (i.e. schema / table definitions) of two MySQL
<http://www.mysql.com/> databases, and returns the differences as a sequence of MySQL commands suitable for piping into mysql which will
transform the structure of the first database to be identical to that of the second (c.f. diff and patch).
Database structures can be compared whether they are files containing table definitions or existing databases, local or remote.
N.B. The program makes no attempt to compare any of the data which may be stored in the databases. It is purely for comparing the table
definitions. I have no plans to implement data comparison; it is a complex problem and I have no need of such functionality anyway.
However there is another program coldiff <http://rossbeyer.net/software/mysql_coldiff/> which does this, and is based on an older program
called datadiff which seems to have vanished off the 'net.
For PostgreSQL there are similar tools such as pgdiff <http://pgdiff.sourceforge.net/> and apgdiff <http://apgdiff.startnet.biz/>.
EXAMPLES
# compare table definitions in two files
mysql-schema-diff db1.mysql db2.mysql
# compare table definitions in a file 'db1.mysql' with a database 'db2'
mysql-schema-diff db1.mysql db2
# interactively upgrade schema of database 'db1' to be like the
# schema described in the file 'db2.mysql'
mysql-schema-diff -A db1 db2.mysql
# compare table definitions in two databases on a remote machine
mysql-schema-diff --host=remote.host.com --user=myaccount db1 db2
# compare table definitions in a local database 'foo' with a
# database 'bar' on a remote machine, when a file foo already
# exists in the current directory
mysql-schema-diff --host2=remote.host.com --password=secret db:foo bar
OPTIONS
More details to come; for now run "mysql-schema-diff --help".
INTERNALS
For both of the database structures being compared, the following happens:
o If the argument is a valid filename, the file is used to create a temporary database which "mysqldump -d" is run on to obtain the table
definitions in canonicalised form. The temporary database is then dropped. (The temporary database is named
"test_mysqldiff_temp_something" because default MySQL permissions allow anyone to create databases beginning with the prefix "test_".)
o If the argument is a database, "mysqldump -d" is run directly on it.
o Where authentication is required, the hostname, username, and password given by the corresponding options are used (type
"mysql-schema-diff --help" for more information).
o Each set of table definitions is now parsed into tables, and fields and index keys within those tables; these are compared, and the
differences outputted in the form of MySQL statements.
BUGS, DEVELOPMENT, CONTRIBUTING
See <http://software.adamspiers.org/wiki/mysqldiff>.
COPYRIGHT AND LICENSE
Copyright (c) 2000-2011 Adam Spiers. All rights reserved. This program is free software; you can redistribute it and/or modify it under the
same terms as Perl itself.
SEE ALSO
MySQL::Diff, MySQL::Diff::Database, MySQL::Diff::Table, MySQL::Diff::Utils, mysql, mysqldump, mysqlshow
AUTHOR
Adam Spiers <mysqldiff@adamspiers.org>
perl v5.14.2 2012-04-06 MYSQLDIFF(1p)