Facing difficulty scraping website with curl Post: 303037942

8 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Reverse Proxy difficulty

Hi I am trying to set up two hosts in a reverse proxy. The reverse proxy already has 8 servers running perfectly, but they are all simply mapping pure addresses, which I have registered internally and externally. The latest two I wish to add are a bit different, they are app servers, one...

2. Shell Programming and Scripting

Perl Difficulty

Hi, I am trying to upload a file to a SQL database table. The column type is IMAGE. I am looking for a solution to upload a word doc file. I tried 3 approaches. 1) my $fileToStore = "mytest.doc"; open IPFILE, "<", $name; binmode IPFILE; while (<IPFILE>) { $fileToStore .= $_; } close...

3. Shell Programming and Scripting

Loop difficulty

hi all I am new to unix and want to create a loop to repeat the loop and before that it ask me to do so.I know "while" may help but I put it in my work and getting stuk with it.any help appreciated.

4. UNIX and Linux Applications

Unable to connect to a website using Curl

Hello, I am using Curl command to make a function call to a web service. The issue I am facing right now is: curl h**p://www.google.com with the above command, I get a XML file as a response. Also when i execute : wget h**p://www.google.com Replace ** : tt I can see status as...

5. Shell Programming and Scripting

difficulty with awk

hello folks, i am stuck with this awk command. i need to calculate the sum of a column of values on a flatfile and i am using the following command : awk -F"|" '{x += $10} END {print "Sum: "x}' standard_csv_file1.out that flatfile contains 180 fields and i am getting the...

6. UNIX for Dummies Questions & Answers

Difficulty with set - command

Hi , i have a shell script with the first two lines as #! /bin/ksh set -x when i try opening the file from unix using the command 'sh filename.sh' , i am getting the below error : invalid optionline 2: set: - Pls help Use code tags for you code and data...

7. Shell Programming and Scripting

Scraping line - Using awk or sed

Hello, Can somone help with this command please? I have this output pattern in a file. I use a simple awk command to print each field separated by comma. For example I use this to get the first awk -F, "{ print $1 }" "ABC=abcdefg,CDF=mnqrst,GGG=hrvyess" issue: What I...

8. Programming

Python Web Page Scraping Urls Creating A Dictionary

I have thrown in the towel and cant figure out how to do this. I have a directory of html files that contain urls that I need to scrape (loop through) and add into a dictionary. An example of the output I would like is: bigbadwolf.htlm: https://www.blah.com, http://www.blahblah.com,...

LEARN ABOUT MOJAVE

curlopt_timevalue

CURLOPT_TIMEVALUE(3)					     curl_easy_setopt options					      CURLOPT_TIMEVALUE(3)

NAME

       CURLOPT_TIMEVALUE - set time value for conditional

SYNOPSIS

       #include <curl/curl.h>

       CURLcode curl_easy_setopt(CURL *handle, CURLOPT_TIMEVALUE, long val);

DESCRIPTION

       Pass  a	long  val  as  parameter. This should be the time counted as seconds since 1 Jan 1970, and the time will be used in a condition as
       specified with CURLOPT_TIMECONDITION(3).

DEFAULT

       0

PROTOCOLS

       HTTP, FTP, RTSP, and FILE

EXAMPLE

       CURL *curl = curl_easy_init();
       if(curl) {
	 curl_easy_setopt(curl, CURLOPT_URL, "http://example.com");

	 /* January 1, 2020 is 1577833200 */
	 curl_easy_setopt(curl, CURLOPT_TIMEVALUE, 1577833200L);

	 /* If-Modified-Since the above time stamp */
	 curl_easy_setopt(curl, CURLOPT_TIMECONDITION, CURL_TIMECOND_IFMODSINCE);

	 /* Perform the request */
	 curl_easy_perform(curl);
       }

AVAILABILITY

       Always

RETURN VALUE

       Returns CURLE_OK

SEE ALSO

       CURLOPT_TIMECONDITION(3),

libcurl 7.54.0							  April 03, 2016					      CURLOPT_TIMEVALUE(3)