Sponsored Content
Top Forums Shell Programming and Scripting Script to scrape page for and save data Post 302870099 by molwiko on Friday 1st of November 2013 07:14:27 AM
Old 11-01-2013
Script to scrape page for and save data

Hi All
I would like to get cars DB from this web site link removed , and I am trying to right script to go and parse the web page and save the data for each cars into a file, type of cars, mark, model, but when I look to the source page I found out that the first car type is preselected and the car marks also, if I want the models of cars I have the change the selected mark on the dropdown and then the models is populated, I think this function is done by Javascript but I can't find it in the source code. it will be good if you help me out to right script that change car mark, save the information cars type, marks, models into a file.
Doing this manually will be a lot of work.

Thanks.

Last edited by joeyg; 11-01-2013 at 08:33 AM..
 

9 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

How to save as image from a web page

I used flot to create a graph and I would like to be able to save/export the graph as an image. In firefox on windows you can just ctl rt-click and you have a save as image feature (which I can automate with js) but...I need this to work on a linux browser. On linux in firefox I can print preview... (11 Replies)
Discussion started by: vincaStar
11 Replies

2. Shell Programming and Scripting

How to pass data from server (CGI script) to client (html page)

Hi I know how to pass data from client side (html file) to server using CGI script (POST method). I also know how to re-create the html page from server side after receiving the data (using printf). However I want to write static pages on client side (only the structure), and only to pass... (0 Replies)
Discussion started by: naamabm
0 Replies

3. Shell Programming and Scripting

Save page source, including javascript

I need to get the source code of a webpage. I have tried to use wget and curl, but it doesn't show the necessary javascript part of the source. I don't have to execute it, only to view the source. How do I do that? (1 Reply)
Discussion started by: locoroco
1 Replies

4. Shell Programming and Scripting

Get Permissions and save to data

Hi all; I have the following code which gives me kind of what I need: #!/usr/bin/perl use Fcntl ':mode'; # if ($ARGV ne "") { $filename = $ARGV; } else { print "Please specify a file!\n"; exit; } # if... (2 Replies)
Discussion started by: gvolpini
2 Replies

5. Shell Programming and Scripting

script for adding page number before page breaks

Hi, If there is an expert that can help: I have many txt files that are produced from pdftotext that include page breaks the page breaks seem to be unix style hex 0C. I want to add page numbers before each page break as in : Page XXXX Regards antman (9 Replies)
Discussion started by: antman
9 Replies

6. Shell Programming and Scripting

Open Page and save it using mozilla

HI Guys, I have one command which can open page and i want to save and exit from it. pf@home> mozilla 181.131.193.10/g/report.txt It will open one page now how can i save it. Thanks (1 Reply)
Discussion started by: pareshkp
1 Replies

7. Shell Programming and Scripting

Scrape 10 million pages and save the raw html data in mysql database

I have a list of 10 million page urls. I want those pages scraped and saved in the mysql database as raw html. I own a Linux VPS server with 1GB RAM and WHM/cPanel. I would like to scrape at least 100,000 urls in 24 hours. So can anyone give me some sample shell scripting code? (4 Replies)
Discussion started by: Viruthagiri
4 Replies

8. UNIX for Dummies Questions & Answers

Get a data and save

If I have a A.log 1 Air Flow Monitor : 34.070 Degrees C 2 Air Flow Monitor : 41.730 Degrees C 3 Air Flow Monitor : 35.340 Degrees C 4 Air Flow Monitor : 33.370 Degrees C 5 Air Flow Monitor : 36.770 Degrees C 6 Air Flow Monitor : 45.910 Degrees C 7 Air Flow Monitor ... (1 Reply)
Discussion started by: sabercats
1 Replies

9. Shell Programming and Scripting

Run sql query in shell script and output data save as delimited text

I want to run sql query in shell script and output data save as delimited text (delimited text would be comma) Code: SPOOL_FILE=/pgedw/dan.txt SQL=/pgedw/dan.sql sqlplus -s username/password@myhost:port/servicename <<EOF set head on set COLSEP , set linesize 32767 SET TRIMSPOOL ON SET... (8 Replies)
Discussion started by: Jaganjag
8 Replies
obdgpslogger(1) 					      General Commands Manual						   obdgpslogger(1)

NAME
obdgpslogger - Log OBD and GPS data to sqlite SYNOPSIS
obdgpslogger [ options ] DESCRIPTION
This is a tool to log OBDII and GPS data to an sqlite database OPTIONS
-s|--serial <serialport> Open this serial port device to connect to the elm327 device. -c|--count <count> Take this many samples at most. Leaving this option out defaults to incessant sampling. -u|--output-log <filename> Redirect stdout and stderr to this file -l|--serial-log <filename> Log all serial comms into this file. -a|--samplerate <samples-per-second> Sample at most this many times a second. The software will sleep temporarily at the end of each loop if appropriate. Keep in mind there is an upper limit to samplerate, typically capped by I/O on your serial port. Set this to zero to sample as fast as possible. BE WARNED. Values greater than ten here are forbidden for cars predating April 2002. If you think your car postdates early 2002, and you'd like to sample as fast as possible, the -o option may help -o|--enable-optimisations Enable certain elm327 optimisations. This will [usually] make sampling faster [not a noticeable amount if you're only sampling once a second], but makes it much easier to accidentally disobey the standard if you're sampling as fast as possible. -p|--capabilities Dump the commands your OBD device claims to support to stdout, then exit. -m|--daemonise Convert the program to a background daemon after successfully initialising. -B|--modifybaud [rate] Attempt to upgrade baudrate to rate. If rate isn't specified, we'll just take a few guesses and go with what works. -b|--baud <rate> Attempt to set serial port baudrate to this after launching. Passing zero tells it to try and guess, or -1 to not try changing at all. -d|--db <database> Open this database to log data to. The software will create the database if it does not exist, and create the tables it needs. -i|--log-columns <column names> Comma-separated list of column names. These are the short names listed when you use "-p" to get your car's capabilities -v|--version Print out version number and exit. -h|--help Print out help and exit. NOT OPTIONS
These options are not intended to be used by end users using the command line, they are support options for the GUI component. Use or rely on them at your own peril. -t|--spam-stdout Write all readings to stdout. The format is unlikely to change in practice, but if you choose to try to parse this yourself, I don't want to hear about it when it does change. SEE ALSO
obd2kml(1), obd2csv(1), obd2gpx(1), obdsim(1), obdgui(1), obdlogrepair(1), obdftdipty(1), dot-obdgpslogger(5) AUTHORS
Gary "Chunky Ks" Briggs <chunky@icculus.org> obdgpslogger(1)
All times are GMT -4. The time now is 02:40 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy