12-22-2016
Noob trying to improve
Hi everyone!
This is my first post here, I hope that I will not already be violating any rule! I also would like to apologize in advance as my post will definitely be a noob post... please have patience and faith
!
Now that I have set the ground rules
, my objective is trying to understand how to write a sort of script which could search and extract information out of the web and put that information into a CSV file.
To be completely transparent, I sort of already have a specific idea of what kind of information I'd like to get: The prices and details of some asset listings on the web.
I have checked the web and found out that with a while loop with a bunch of curl and grep commands could do the trick. Is there anyone who can help me in building this sort of "web-crawler".
Thanks in advance to you all!
Ardzii
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi all,
Still a newbie and learning as I go ... as you do :)
Have created this script to report on disc usage and I've just included the ChkSpace function this morning.
It's the first time I've read a file (line-by-bloody-line) and would like to know if I can improve this script ?
FYI - I... (11 Replies)
Discussion started by: Cameron
11 Replies
2. UNIX for Advanced & Expert Users
Hi ,
i'm searching for files over many Aix servers with rsh command using this request :
find /dir1 -name '*.' -exec ls {} \;
and then count them with "wc"
but i would improve this search because it's too long and replace directly find with ls command but "ls *. " doesn't work.
and... (3 Replies)
Discussion started by: Nicol
3 Replies
3. Shell Programming and Scripting
Wrote this script to find the date x days before or after today. Is there any way that this script can be speeded up or otherwise improved?
#!/usr/bin/sh
check_done() {
if
then
daysofmth=31
elif
then
if
... (11 Replies)
Discussion started by: blowtorch
11 Replies
4. UNIX for Dummies Questions & Answers
hi someone tell me which ways i can improve disk I/O and system process performance.kindly refer some commands so i can do it on my test machine.thanks, Mazhar (2 Replies)
Discussion started by: mazhar99
2 Replies
5. Shell Programming and Scripting
I have a data file of 2 gig
I need to do all these, but its taking hours, any where i can improve performance, thanks a lot
#!/usr/bin/ksh
echo TIMESTAMP="$(date +'_%y-%m-%d.%H-%M-%S')"
function showHelp {
cat << EOF >&2
syntax extreme.sh FILENAME
Specify filename to parse
EOF... (3 Replies)
Discussion started by: sirababu
3 Replies
6. IP Networking
I have a 10Gbps network link connecting two machines A and B. I want to transfer 20GB data from A to B using TCP. With default setting, I can use 50% bandwidth. How to improve the throughput? Is there any way to make throughput as close to 10Gbps as possible? thanks~ :) (3 Replies)
Discussion started by: andrewust
3 Replies
7. Shell Programming and Scripting
Hi All,
I have written a script as follows which is taking lot of time in executing/searching only 3500 records taken as input from one file in log file of 12 GB Approximately.
Working of script is read the csv file as an input having 2 arguments which are transaction_id,mobile_number and search... (6 Replies)
Discussion started by: poweroflinux
6 Replies
8. AIX
I just wrote a very small script that improves readability on system sulog. The problem with all sulog is there is lack of clarity whether the info you are looking at is the most current. So if you just need a simple soution instead of going thru the trouble of writing a script that rotate logs and... (0 Replies)
Discussion started by: sparcguy
0 Replies
9. Shell Programming and Scripting
Gents.
I have 2 different scripts for the same purpose:
raw2csv_1
Script raw2csv_1 finish the process in less that 1 minute
raw2csv_2
Script raw2csv_2 finish the process in more that 6 minutes.
Can you please check if there is any option to improve the raw2csv_2. To finish the job... (4 Replies)
Discussion started by: jiam912
4 Replies
10. Shell Programming and Scripting
Gents,
Is there the possibility to improve this script to be able to have same output information.
I did this script, but I believe there is a very short code to get same output
here my script
awk -F, '{if($10>0 && $10<=15) print $6}' tmp1 | sort -k1n | awk '{a++} END { for (n in a )... (23 Replies)
Discussion started by: jiam912
23 Replies
LEARN ABOUT DEBIAN
arch::liteweb
Arch::LiteWeb(3pm) User Contributed Perl Documentation Arch::LiteWeb(3pm)
NAME
Arch::LiteWeb - simple way to access web pages
SYNOPSIS
my $web = Arch::LiteWeb->new;
my $content = $web->get("http://some.domain:81/some/path");
die $web->error . " while processing " . $web->request_url
unless $content;
my $content_type = $web->response_headers->{content_type};
DESCRIPTION
This class provides a basic and easy to use support for the client-side HTTP. It is supplied in order to avoid dependency on LWP. If such
dependency is not a problem, consider to use LWP instead that provides much better support for HTTP and other protocols.
METHODS
The following class methods are available:
get, post, request_url, error, error_with_url, network_error, response_code, response_codestr, response_error, response_headers,
response_content.
get url [params ...]
Execute HTTP get of the given url and return the html string or undef on network/response error. Use other methods to get the details
about the error and the response.
params is key-value hash, the following keys are supported:
url_host - only used if url is none
url_port - only used if url is none(80)
url_path - only used if url is none
endl - default is "