05-03-2013
Scrape 10 million pages and save the raw html data in mysql database
I have a list of 10 million page urls. I want those pages scraped and saved in the mysql database as raw html.
I own a Linux VPS server with 1GB RAM and WHM/cPanel.
I would like to scrape at least 100,000 urls in 24 hours.
So can anyone give me some sample shell scripting code?
8 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi,
Need to get help from you guys about this issue.
I need to insert data into MySql database from a text file which is located in other server.
The text file is something look like below:
Date | SubscriberNo | Call Duration
20/7/07 | 123456788 | 20
20/7/07 | 123412344 | 30
The... (4 Replies)
Discussion started by: shirleyeow
4 Replies
2. Shell Programming and Scripting
Hi,
I have gps receiver, by using gpsd data i can read gps log data to my database(my sql).
Steps:
1. telenet localhost 2947 > gps.txt (press enter)
2. r (press enter) //then i will get the data like below in gps.txt file
Trying 127.0.0.1...
Connected to localhost.... (1 Reply)
Discussion started by: gudivada213
1 Replies
3. Web Development
Hi all,
I was wondering if anyone knew a good/safe way to update a single column in a table that could contain upto 8 million rows...
simple command like:
UPDATE set blah=foo where bar=XXX;
I will be running this on tables being written to and tables that have already been created.
... (3 Replies)
Discussion started by: muay_tb
3 Replies
4. Shell Programming and Scripting
CAN I download an html file via wget and pass it to mysql and update a database field? (8 Replies)
Discussion started by: mapasainfo
8 Replies
5. Shell Programming and Scripting
Have to delete this long post. Seems nobody would spent time on it. (0 Replies)
Discussion started by: yifangt
0 Replies
6. Shell Programming and Scripting
Hi All
I would like to get cars DB from this web site link removed , and I am trying to right script to go and parse the web page and save the data for each cars into a file, type of cars, mark, model, but when I look to the source page I found out that the first car type is preselected and the... (3 Replies)
Discussion started by: molwiko
3 Replies
7. Shell Programming and Scripting
Sorry to disturb you, I would like to seek help on inserting data whenever the switch is on or off to my phpMyAdmin mySQL database from my Shell Script. I'm using Raspberry PI as my hardware and I have follow this LINK: instructables.com/id/Web-Control-of-Raspberry-Pi-GPIO/?ALLSTEPS to create my... (4 Replies)
Discussion started by: aoiregion
4 Replies
8. Programming
Hi guys, I would like to seek help on inserting data whenever the switch is on or off to my sensor mySQL database in phpMyAdmin from my control.php. I'm using Raspberry PI as my hardware and follow a few tutorials to create my own Web Control Interface, it works perfectly without insert method.... (1 Reply)
Discussion started by: aoiregion
1 Replies
LEARN ABOUT DEBIAN
phatch
PHATCH(1) General Commands Manual PHATCH(1)
NAME
Phatch - Photo Batch Processor
DESCRIPTION
Phatch is a simple photo batch processor. It handles all popular image formats and can duplicate (sub)folder hierarchies. It can also
batch resize, rotate, rename, ... and more in minutes instead of hours or days if you do it manually.
SYNOPSIS
Phatch [actionlist]
Phatch [options] [actionlist] [image folders/files/urls]
Phatch --inspect [image files/urls]
Phatch --droplet [actionlist/recent] [image files/urls]
OPTIONS
--version
Show program's version number and exit.
-h, --help
Show the command line options which are accepted by Phatch
-c, --console
Run Phatch as console program without a gui
-d, --droplet
Run Phatch as a gui droplet
--desktop
Save always on desktop
-f, --force
Ignore errors
--fonts
Initialize fonts (only for installation scripts)
-i, --interactive
Interactive
-k, --keep
Keep existing images (don't overwrite)
-l LOCALE
Specify locale language (for example en or en_GB)
-n, --inspect
Inspect metadata (requires exif & iptc plugin)
--no-save
No save action required at the end
-r, --recursive
Include all subfolders
-t, --trust
Do not check images first
--unsafe
Allow Geek action and unsafe expressions
-v, --verbose
Verbose
EXAMPLES
phatch action_list.phatch
phatch --verbose --recursive action_list.py image_file.png image_folder
phatch --inspect image_file.jpg
phatch --droplet recent
phatch -l el
AUTHOR
Stani (spe.stani.be (at) gmail.com)
User Commands February 2009 PHATCH(1)