Sponsored Content
Top Forums Shell Programming and Scripting Script to scrape page for and save data Post 302870105 by molwiko on Friday 1st of November 2013 07:56:19 AM
Old 11-01-2013
Actually I am working on module about cars on Joomla and I would like some real data to store, but the structure of DB it's different I just want the data, I don't know why it can be any permission issue about that.
 

9 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

How to save as image from a web page

I used flot to create a graph and I would like to be able to save/export the graph as an image. In firefox on windows you can just ctl rt-click and you have a save as image feature (which I can automate with js) but...I need this to work on a linux browser. On linux in firefox I can print preview... (11 Replies)
Discussion started by: vincaStar
11 Replies

2. Shell Programming and Scripting

How to pass data from server (CGI script) to client (html page)

Hi I know how to pass data from client side (html file) to server using CGI script (POST method). I also know how to re-create the html page from server side after receiving the data (using printf). However I want to write static pages on client side (only the structure), and only to pass... (0 Replies)
Discussion started by: naamabm
0 Replies

3. Shell Programming and Scripting

Save page source, including javascript

I need to get the source code of a webpage. I have tried to use wget and curl, but it doesn't show the necessary javascript part of the source. I don't have to execute it, only to view the source. How do I do that? (1 Reply)
Discussion started by: locoroco
1 Replies

4. Shell Programming and Scripting

Get Permissions and save to data

Hi all; I have the following code which gives me kind of what I need: #!/usr/bin/perl use Fcntl ':mode'; # if ($ARGV ne "") { $filename = $ARGV; } else { print "Please specify a file!\n"; exit; } # if... (2 Replies)
Discussion started by: gvolpini
2 Replies

5. Shell Programming and Scripting

script for adding page number before page breaks

Hi, If there is an expert that can help: I have many txt files that are produced from pdftotext that include page breaks the page breaks seem to be unix style hex 0C. I want to add page numbers before each page break as in : Page XXXX Regards antman (9 Replies)
Discussion started by: antman
9 Replies

6. Shell Programming and Scripting

Open Page and save it using mozilla

HI Guys, I have one command which can open page and i want to save and exit from it. pf@home> mozilla 181.131.193.10/g/report.txt It will open one page now how can i save it. Thanks (1 Reply)
Discussion started by: pareshkp
1 Replies

7. Shell Programming and Scripting

Scrape 10 million pages and save the raw html data in mysql database

I have a list of 10 million page urls. I want those pages scraped and saved in the mysql database as raw html. I own a Linux VPS server with 1GB RAM and WHM/cPanel. I would like to scrape at least 100,000 urls in 24 hours. So can anyone give me some sample shell scripting code? (4 Replies)
Discussion started by: Viruthagiri
4 Replies

8. UNIX for Dummies Questions & Answers

Get a data and save

If I have a A.log 1 Air Flow Monitor : 34.070 Degrees C 2 Air Flow Monitor : 41.730 Degrees C 3 Air Flow Monitor : 35.340 Degrees C 4 Air Flow Monitor : 33.370 Degrees C 5 Air Flow Monitor : 36.770 Degrees C 6 Air Flow Monitor : 45.910 Degrees C 7 Air Flow Monitor ... (1 Reply)
Discussion started by: sabercats
1 Replies

9. Shell Programming and Scripting

Run sql query in shell script and output data save as delimited text

I want to run sql query in shell script and output data save as delimited text (delimited text would be comma) Code: SPOOL_FILE=/pgedw/dan.txt SQL=/pgedw/dan.sql sqlplus -s username/password@myhost:port/servicename <<EOF set head on set COLSEP , set linesize 32767 SET TRIMSPOOL ON SET... (8 Replies)
Discussion started by: Jaganjag
8 Replies
DBD::File::HowTo(3pm)					User Contributed Perl Documentation				     DBD::File::HowTo(3pm)

NAME
DBD::File::HowTo - Guide to create DBD::File based driver SYNOPSIS
perldoc DBD::File::HowTo perldoc DBI perldoc DBI::DBD perldoc DBD::File::Developers perldoc DBI::DBD::SqlEngine::Developers perldoc DBI::DBD::SqlEngine perldoc SQL::Eval perldoc DBI::DBD::SqlEngine::HowTo perldoc SQL::Statement::Embed perldoc DBD::File perldoc DBD::File::HowTo perldoc DBD::File::Developers DESCRIPTION
This document provides a step-by-step guide, how to create a new "DBD::File" based DBD. It expects that you carefully read the DBI documentation and that you're familiar with DBI::DBD and had read and understood DBD::ExampleP. This document addresses experienced developers who are really sure that they need to invest time when writing a new DBI Driver. Writing a DBI Driver is neither a weekend project nor an easy job for hobby coders after work. Expect one or two man-month of time for the first start. Those who are still reading, should be able to sing the rules of "CREATING A NEW DRIVER" in DBI::DBD. Of course, DBD::File is a DBI::DBD::SqlEngine and you surely read DBI::DBD::SqlEngine::HowTo before continuing here. CREATING DRIVER CLASSES
Do you have an entry in DBI's DBD registry? For this guide, a prefix of "foo_" is assumed. Sample Skeleton package DBD::Foo; use strict; use warnings; use vars qw(@ISA $VERSION); use base qw(DBD::File); use DBI (); $VERSION = "0.001"; package DBD::Foo::dr; use vars qw(@ISA $imp_data_size); @ISA = qw(DBD::File::dr); $imp_data_size = 0; package DBD::Foo::db; use vars qw(@ISA $imp_data_size); @ISA = qw(DBD::File::db); $imp_data_size = 0; package DBD::Foo::st; use vars qw(@ISA $imp_data_size); @ISA = qw(DBD::File::st); $imp_data_size = 0; package DBD::Foo::Statement; use vars qw(@ISA); @ISA = qw(DBD::File::Statement); package DBD::Foo::Table; use vars qw(@ISA); @ISA = qw(DBD::File::Table); 1; Tiny, eh? And all you have now is a DBD named foo which will is able to deal with temporary tables, as long as you use SQL::Statement. In DBI::SQL::Nano environments, this DBD can do nothing. Start over Based on DBI::DBD::SqlEngine::HowTo, we're now having a driver which could do basic things. Of course, it should now derive from DBD::File instead of DBI::DBD::SqlEngine, shouldn't it? DBD::File extends DBI::DBD::SqlEngine to deal with any kind of files. In principle, the only extensions required are to the table class: package DBD::Foo::Table; sub bootstrap_table_meta { my ( $self, $dbh, $meta, $table ) = @_; # initialize all $meta attributes which might be relevant for # file2table return $self->SUPER::bootstrap_table_meta($dbh, $meta, $table); } sub init_table_meta { my ( $self, $dbh, $meta, $table ) = @_; # called after $meta contains the results from file2table # initialize all missing $meta attributes $self->SUPER::init_table_meta( $dbh, $meta, $table ); } In case "DBD::File::Table::open_file" doesn't open the files as the driver needs that, override it! sub open_file { my ( $self, $meta, $attrs, $flags ) = @_; # ensure that $meta->{f_dontopen} is set $self->SUPER::open_file( $meta, $attrs, $flags ); # now do what ever needs to be done } Combined with the methods implemented using the SQL::Statement::Embed guide, the table is full working and you could try a start over. User comfort "DBD::File" since 0.39 consolidates all persistent meta data of a table into a single structure stored in "$dbh->{f_meta}". While DBD::File provides only readonly access to this structure, modifications are still allowed. Primarily DBD::File provides access via setters "get_file_meta", "set_file_meta" and "clear_file_meta". Those methods are easily accessible by the users via the "$dbh->func ()" interface provided by DBI. Well, many users don't feel comfortize when calling # don't require extension for tables cars $dbh->func ("cars", "f_ext", ".csv", "set_file_meta"); DBD::File will inject a method into your driver to increase the user comfort to allow: # don't require extension for tables cars $dbh->foo_set_meta ("cars", "f_ext", ".csv"); Better, but here and there users likes to do: # don't require extension for tables cars $dbh->{foo_tables}->{cars}->{f_ext} = ".csv"; This interface is provided when derived DBD's define following in "init_valid_attributes" (please compare carefully with the example in DBI::DBD::SqlEngine::HowTo): sub init_valid_attributes { my $dbh = $_[0]; $dbh->SUPER::init_valid_attributes (); $dbh->{foo_valid_attrs} = { foo_version => 1, # contains version of this driver foo_valid_attrs => 1, # contains the valid attributes of foo drivers foo_readonly_attrs => 1, # contains immutable attributes of foo drivers foo_bar => 1, # contains the bar attribute foo_baz => 1, # contains the baz attribute foo_manager => 1, # contains the manager of the driver instance foo_manager_type => 1, # contains the manager class of the driver instance foo_meta => 1, # contains the public interface to modify table meta attributes }; $dbh->{foo_readonly_attrs} = { foo_version => 1, # ensure no-one modifies the driver version foo_valid_attrs => 1, # do not permit to add more valid attributes ... foo_readonly_attrs => 1, # ... or make the immutable mutable foo_manager => 1, # manager is set internally only foo_meta => 1, # ensure public interface to modify table meta attributes are immutable }; $dbh->{foo_meta} = "foo_tables"; return $dbh; } This provides a tied hash in "$dbh->{foo_tables}" and a tied hash for each table's meta data in "$dbh->{foo_tables}->{$table_name}". Modifications on the table meta attributes are done using the table methods: sub get_table_meta_attr { ... } sub set_table_meta_attr { ... } Both methods can adjust the attribute name for compatibility reasons, e.g. when former versions of the DBD allowed different names to be used for the same flag: my %compat_map = ( abc => 'foo_abc', xyz => 'foo_xyz', ); __PACKAGE__->register_compat_map( \%compat_map ); If any user modification on a meta attribute needs reinitialization of the meta structure (in case of "DBD::File" these are the attributes "f_file", "f_dir", "f_ext" and "f_lockfile"), inform DBD::File by doing my %reset_on_modify = ( foo_xyz => "foo_bar", foo_abc => "foo_bar", ); __PACKAGE__->register_reset_on_modify( \%reset_on_modify ); The next access to the table meta data will force DBD::File to re-do the entire meta initialization process. Any further action which needs to be taken can handled in "table_meta_attr_changed": sub table_meta_attr_changed { my ($class, $meta, $attrib, $value) = @_; ... $class->SUPER::table_meta_attr_changed ($meta, $attrib, $value); } This is done before the new value is set in $meta, so the attribute changed handler can act depending on the old value. Testing Now you should have your own DBD::File based driver. Was easy, wasn't it? But does it work well? Prove it by writing tests and remember to use dbd_edit_mm_attribs from DBI::DBD to ensure testing even rare cases. AUTHOR
This guide is written by Jens Rehsack. DBD::File is written by Jochen Wiedmann and Jeff Zucker. The module DBD::File is currently maintained by H.Merijn Brand < h.m.brand at xs4all.nl > and Jens Rehsack < rehsack at googlemail.com > COPYRIGHT AND LICENSE
Copyright (C) 2010 by H.Merijn Brand & Jens Rehsack All rights reserved. You may freely distribute and/or modify this module under the terms of either the GNU General Public License (GPL) or the Artistic License, as specified in the Perl README file. perl v5.12.3 2010-09-16 DBD::File::HowTo(3pm)
All times are GMT -4. The time now is 01:28 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy