Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Facing difficulty scraping website with curl Post 303037943 by Neo on Sunday 18th of August 2019 10:55:18 PM
Old 08-18-2019
I assume you are going to use their API?

API Reference -skyscanner developers

"Screen Scraping" is a thing of the past, really...

JSON APIs like this one are the current "in thing" Smilie
 

8 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Reverse Proxy difficulty

Hi I am trying to set up two hosts in a reverse proxy. The reverse proxy already has 8 servers running perfectly, but they are all simply mapping pure addresses, which I have registered internally and externally. The latest two I wish to add are a bit different, they are app servers, one... (1 Reply)
Discussion started by: rboekdrukker
1 Replies

2. Shell Programming and Scripting

Perl Difficulty

Hi, I am trying to upload a file to a SQL database table. The column type is IMAGE. I am looking for a solution to upload a word doc file. I tried 3 approaches. 1) my $fileToStore = "mytest.doc"; open IPFILE, "<", $name; binmode IPFILE; while (<IPFILE>) { $fileToStore .= $_; } close... (1 Reply)
Discussion started by: b.paramanatti
1 Replies

3. Shell Programming and Scripting

Loop difficulty

hi all I am new to unix and want to create a loop to repeat the loop and before that it ask me to do so.I know "while" may help but I put it in my work and getting stuk with it.any help appreciated. (13 Replies)
Discussion started by: samsami1971
13 Replies

4. UNIX and Linux Applications

Unable to connect to a website using Curl

Hello, I am using Curl command to make a function call to a web service. The issue I am facing right now is: curl h**p://www.google.com with the above command, I get a XML file as a response. Also when i execute : wget h**p://www.google.com Replace ** : tt I can see status as... (1 Reply)
Discussion started by: amitshete
1 Replies

5. Shell Programming and Scripting

difficulty with awk

hello folks, i am stuck with this awk command. i need to calculate the sum of a column of values on a flatfile and i am using the following command : awk -F"|" '{x += $10} END {print "Sum: "x}' standard_csv_file1.out that flatfile contains 180 fields and i am getting the... (5 Replies)
Discussion started by: jdsony
5 Replies

6. UNIX for Dummies Questions & Answers

Difficulty with set - command

Hi , i have a shell script with the first two lines as #! /bin/ksh set -x when i try opening the file from unix using the command 'sh filename.sh' , i am getting the below error : invalid optionline 2: set: - Pls help Use code tags for you code and data... (1 Reply)
Discussion started by: Rajankum
1 Replies

7. Shell Programming and Scripting

Scraping line - Using awk or sed

Hello, Can somone help with this command please? I have this output pattern in a file. I use a simple awk command to print each field separated by comma. For example I use this to get the first awk -F, "{ print $1 }" "ABC=abcdefg,CDF=mnqrst,GGG=hrvyess" issue: What I... (7 Replies)
Discussion started by: mnassiri
7 Replies

8. Programming

Python Web Page Scraping Urls Creating A Dictionary

I have thrown in the towel and cant figure out how to do this. I have a directory of html files that contain urls that I need to scrape (loop through) and add into a dictionary. An example of the output I would like is: bigbadwolf.htlm: https://www.blah.com, http://www.blahblah.com,... (5 Replies)
Discussion started by: metallica1973
5 Replies
libapache2-mod-perl2-2.0.7::docs::api::Apache2::URI(3pm)User Contributed Perl Documentatiolibapache2-mod-perl2-2.0.7::docs::api::Apache2::URI(3pm)

NAME
Apache2::URI - Perl API for manipulating URIs Synopsis use Apache2::URI (); $hostport = $r->construct_server(); $hostport = $r->construct_server($hostname); $hostport = $r->construct_server($hostname, $port); $hostport = $r->construct_server($hostname, $port, $pool); $url = $r->construct_url(); $url = $r->construct_url($rel_uri); $url = $r->construct_url($rel_uri, $pool); $parsed_uri = $r->parse_uri($uri); $parsed_uri = $r->parsed_uri(); $url = join '%20', qw(one two three); Apache2::URI::unescape_url($url); Description While "APR::URI" provides a generic API to dissect, adjust and put together any given URI string, "Apache2::URI" provides an API specific to Apache, by taking the information directly from the $r object. Therefore when manipulating the URI of the current HTTP request usually methods from both classes are used. API
"Apache2::URI" provides the following functions and methods: "construct_server" Construct a string made of hostname and port $hostport = $r->construct_server(); $hostport = $r->construct_server($hostname); $hostport = $r->construct_server($hostname, $port); $hostport = $r->construct_server($hostname, $port, $pool); obj: $r ( "Apache2::RequestRec object" ) The current request object opt arg1: $hostname ( string ) The hostname of the server. If that argument is not passed, "$r->get_server_name" is used. opt arg2: $port ( string ) The port the server is running on. If that argument is not passed, "$r->get_server_port" is used. opt arg3: $pool ( "APR::Pool object" ) The pool to allocate the string from. If that argument is not passed, "$r->pool" is used. ret: $hostport ( string ) The server's hostport string since: 2.0.00 Examples: o Assuming that: $r->get_server_name == "localhost"; $r->get_server_port == 8001; The code: $hostport = $r->construct_server(); returns a string: localhost:8001 o The following code sets the values explicitly: $hostport = $r->construct_server("my.example.com", 8888); and it returns a string: my.example.com:8888 "construct_url" Build a fully qualified URL from the uri and information in the request rec: $url = $r->construct_url(); $url = $r->construct_url($rel_uri); $url = $r->construct_url($rel_uri, $pool); obj: $r ( "Apache2::RequestRec object" ) The current request object opt arg1: $rel_uri ( string ) The path to the requested file (it may include a concatenation of path, query and fragment components). If that argument is not passed, "$r->uri" is used. opt arg2: $pool ( "APR::Pool object" ) The pool to allocate the URL from If that argument is not passed, "$r->pool" is used. ret: $url ( string ) A fully qualified URL since: 2.0.00 Examples: o Assuming that the request was http://localhost.localdomain:8529/test?args The code: my $url = $r->construct_url; returns the string: http://localhost.localdomain:8529/test notice that the query (args) component is not in the string. You need to append it manually if it's needed. o Assuming that the request was http://localhost.localdomain:8529/test?args The code: my $rel_uri = "/foo/bar?tar"; my $url = $r->construct_url($rel_uri); returns the string: http://localhost.localdomain:8529/foo/bar?tar "parse_uri" Break apart URI (affecting the current request's uri components) $r->parse_uri($uri); obj: $r ( "Apache2::RequestRec object" ) The current request object arg1: $uri ( string ) The uri to break apart ret: no return value warning: This method has several side-effects explained below since: 2.0.00 This method call has the following side-effects: 1. sets "$r->args" to the rest after '?' if such exists in the passed $uri, otherwise sets it to "undef". 2. sets "$r->uri" to the passed $uri without the "$r->args" part. 3. sets "$r->hostname" (if not set already) using the ("scheme://host:port") parts of the passed $uri. "parsed_uri" Get the current request's parsed uri object my $uri = $r->parsed_uri(); obj: $r ( "Apache2::RequestRec object" ) The current request object ret: $uri ( "APR::URI object" ) The parsed uri since: 2.0.00 This object is suitable for using with "APR::URI::rpath" "unescape_url" Unescape URLs Apache2::URI::unescape_url($url); obj: $url ( string ) The URL to unescape ret: no return value The argument $url is now unescaped since: 2.0.00 Example: my $url = join '%20', qw(one two three); Apache2::URI::unescape_url($url); $url now contains the string: "one two three"; See Also "APR::URI", mod_perl 2.0 documentation. Copyright mod_perl 2.0 and its core modules are copyrighted under The Apache Software License, Version 2.0. Authors The mod_perl development team and numerous contributors. perl v5.14.2 2011-02-08 libapache2-mod-perl2-2.0.7::docs::api::Apache2::URI(3pm)
All times are GMT -4. The time now is 03:28 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy