Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Facing difficulty scraping website with curl Post 303037943 by Neo on Sunday 18th of August 2019 10:55:18 PM
Old 08-18-2019
I assume you are going to use their API?

API Reference -skyscanner developers

"Screen Scraping" is a thing of the past, really...

JSON APIs like this one are the current "in thing" Smilie
 

8 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Reverse Proxy difficulty

Hi I am trying to set up two hosts in a reverse proxy. The reverse proxy already has 8 servers running perfectly, but they are all simply mapping pure addresses, which I have registered internally and externally. The latest two I wish to add are a bit different, they are app servers, one... (1 Reply)
Discussion started by: rboekdrukker
1 Replies

2. Shell Programming and Scripting

Perl Difficulty

Hi, I am trying to upload a file to a SQL database table. The column type is IMAGE. I am looking for a solution to upload a word doc file. I tried 3 approaches. 1) my $fileToStore = "mytest.doc"; open IPFILE, "<", $name; binmode IPFILE; while (<IPFILE>) { $fileToStore .= $_; } close... (1 Reply)
Discussion started by: b.paramanatti
1 Replies

3. Shell Programming and Scripting

Loop difficulty

hi all I am new to unix and want to create a loop to repeat the loop and before that it ask me to do so.I know "while" may help but I put it in my work and getting stuk with it.any help appreciated. (13 Replies)
Discussion started by: samsami1971
13 Replies

4. UNIX and Linux Applications

Unable to connect to a website using Curl

Hello, I am using Curl command to make a function call to a web service. The issue I am facing right now is: curl h**p://www.google.com with the above command, I get a XML file as a response. Also when i execute : wget h**p://www.google.com Replace ** : tt I can see status as... (1 Reply)
Discussion started by: amitshete
1 Replies

5. Shell Programming and Scripting

difficulty with awk

hello folks, i am stuck with this awk command. i need to calculate the sum of a column of values on a flatfile and i am using the following command : awk -F"|" '{x += $10} END {print "Sum: "x}' standard_csv_file1.out that flatfile contains 180 fields and i am getting the... (5 Replies)
Discussion started by: jdsony
5 Replies

6. UNIX for Dummies Questions & Answers

Difficulty with set - command

Hi , i have a shell script with the first two lines as #! /bin/ksh set -x when i try opening the file from unix using the command 'sh filename.sh' , i am getting the below error : invalid optionline 2: set: - Pls help Use code tags for you code and data... (1 Reply)
Discussion started by: Rajankum
1 Replies

7. Shell Programming and Scripting

Scraping line - Using awk or sed

Hello, Can somone help with this command please? I have this output pattern in a file. I use a simple awk command to print each field separated by comma. For example I use this to get the first awk -F, "{ print $1 }" "ABC=abcdefg,CDF=mnqrst,GGG=hrvyess" issue: What I... (7 Replies)
Discussion started by: mnassiri
7 Replies

8. Programming

Python Web Page Scraping Urls Creating A Dictionary

I have thrown in the towel and cant figure out how to do this. I have a directory of html files that contain urls that I need to scrape (loop through) and add into a dictionary. An example of the output I would like is: bigbadwolf.htlm: https://www.blah.com, http://www.blahblah.com,... (5 Replies)
Discussion started by: metallica1973
5 Replies
CURLOPT_HTTPGET(3)					     curl_easy_setopt options						CURLOPT_HTTPGET(3)

NAME
CURLOPT_HTTPGET - ask for a HTTP GET request SYNOPSIS
#include <curl/curl.h> CURLcode curl_easy_setopt(CURL *handle, CURLOPT_HTTPGET, long useget); DESCRIPTION
Pass a long. If useget is 1, this forces the HTTP request to get back to using GET. Usable if a POST, HEAD, PUT, etc has been used previ- ously using the same curl handle. When setting CURLOPT_HTTPGET(3) to 1, it will automatically set CURLOPT_NOBODY(3) to 0 and CURLOPT_UPLOAD(3) to 0. DEFAULT
0 PROTOCOLS
HTTP(S) EXAMPLE
curl = curl_easy_init(); if(curl) { curl_easy_setopt(curl, CURLOPT_URL, "http://example.com"); /* use a GET to fetch this */ curl_easy_setopt(curl, CURLOPT_HTTPGET, 1L); /* Perform the request */ curl_easy_perform(curl); } AVAILABILITY
Along with HTTP RETURN VALUE
Returns CURLE_OK if HTTP is supported, and CURLE_UNKNOWN_OPTION if not. SEE ALSO
CURLOPT_NOBODY(3), CURLOPT_UPLOAD(3), libcurl 7.54.0 February 03, 2016 CURLOPT_HTTPGET(3)
All times are GMT -4. The time now is 11:26 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy