Scrape 10 million pages and save the raw html data in mysql database Post: 302802113

8 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to insert data into MYSql database from a text file

Hi, Need to get help from you guys about this issue. I need to insert data into MySql database from a text file which is located in other server. The text file is something look like below: Date | SubscriberNo | Call Duration 20/7/07 | 123456788 | 20 20/7/07 | 123412344 | 30 The...

2. Shell Programming and Scripting

shell script to insert data from gps.txt to mysql database

Hi, I have gps receiver, by using gpsd data i can read gps log data to my database(my sql). Steps: 1. telenet localhost 2947 > gps.txt (press enter) 2. r (press enter) //then i will get the data like below in gps.txt file Trying 127.0.0.1... Connected to localhost....

3. Web Development

Mysql question: Best way to update a column containing 8 million rows

Hi all, I was wondering if anyone knew a good/safe way to update a single column in a table that could contain upto 8 million rows... simple command like: UPDATE set blah=foo where bar=XXX; I will be running this on tables being written to and tables that have already been created. ...

4. Shell Programming and Scripting

download an html file via wget and pass it to mysql and update a database

CAN I download an html file via wget and pass it to mysql and update a database field?

5. Shell Programming and Scripting

MySQL bulk retrieval of database through html form

Have to delete this long post. Seems nobody would spent time on it.

6. Shell Programming and Scripting

Script to scrape page for and save data

Hi All I would like to get cars DB from this web site link removed , and I am trying to right script to go and parse the web page and save the data for each cars into a file, type of cars, mark, model, but when I look to the source page I found out that the first car type is preselected and the...

7. Shell Programming and Scripting

Need help on Insert data to phpMyAdmin mySQL database from Shell Script

Sorry to disturb you, I would like to seek help on inserting data whenever the switch is on or off to my phpMyAdmin mySQL database from my Shell Script. I'm using Raspberry PI as my hardware and I have follow this LINK: instructables.com/id/Web-Control-of-Raspberry-Pi-GPIO/?ALLSTEPS to create my...

8. Programming

Need help on Insert data to mySQL database

Hi guys, I would like to seek help on inserting data whenever the switch is on or off to my sensor mySQL database in phpMyAdmin from my control.php. I'm using Raspberry PI as my hardware and follow a few tutorials to create my own Web Control Interface, it works perfectly without insert method....

LEARN ABOUT DEBIAN

bti-shrink-urls

BTI-SHRINK-URLS(1)						  bti-shrink-urls						BTI-SHRINK-URLS(1)

NAME

       bti-shrink-urls - convert URLs to a shorter form using a web service

SYNOPSIS

       bti [--escaped] [--help] [URL]

DESCRIPTION

       bti-shrink-urls converts URLs to a shorter form using a web service.

       Currently http://2tu.us/ (default) and http://bit.ly / http://j.mp are supported.

OPTIONS

       --escaped
	   Don't escape special characters in the URL any more, they are already percent encoded.

       --help
	   Print help text.

       URL
	   Specify the URL to be converted. If no URL is given bti-shrink-urls waits for input on stdin.

CONFIGURATION

       bti-shrink-urls is configured by setting some values in ~/.bti:

       shrink_host
	   Possible values: 2tu.us (default), bit.ly, j.mp

       shrink_bitly_login
	   API login for bit.ly, j.mp, required if shrink_host is set to bit.ly or j.mp. See
	   https://code.google.com/p/bitly-api/wiki/ApiDocumentation

       shrink_bitly_key
	   API key for bit.ly, j.mp, required if shrink_host is set to bit.ly or j.mp. See
	   https://code.google.com/p/bitly-api/wiki/ApiDocumentation

AUTHOR

       Written by Bart Trojanowski bart@jukie.net.

COPYRIGHT AND LICENSE

       Copyright (C) 2009 Bart Trojanowski bart@jukie.net.

       This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
       the Free Software Foundation version 2 of the License.

bti-shrink-urls 						    March 2009							BTI-SHRINK-URLS(1)