Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Facing difficulty scraping website with curl Post 303037941 by sand1234 on Sunday 18th of August 2019 07:43:46 PM
Old 08-18-2019
Facing difficulty scraping website with curl

Hi,

As an exercise, I am attempting to scrape the following website to obtain ticket price for a certain journey, so I can capture the cheapest ticket by running the script hourly.

Code:
https://www.skyscanner.com.au/transport/d/del/2019-12-14/bkkt/bkkt/2019-12-19/del?adults=2&children=0&adultsv2=2&childrenv2=&infants=0&cabinclass=economy&ref=home#/

I am not able to capture the ticket price via the curl command. Is there any better way to scrape the webpage to get the information I need?

Code:
curl -o skyscanner.txt -L https://www.skyscanner.com.au/transport/d/del/2019-12-14/bkkt/bkkt/2019-12-19/del?adults=2&children=0&adultsv2=2&childrenv2=&infants=0&cabinclass=economy&ref=home#/

Thanks.
 

8 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Reverse Proxy difficulty

Hi I am trying to set up two hosts in a reverse proxy. The reverse proxy already has 8 servers running perfectly, but they are all simply mapping pure addresses, which I have registered internally and externally. The latest two I wish to add are a bit different, they are app servers, one... (1 Reply)
Discussion started by: rboekdrukker
1 Replies

2. Shell Programming and Scripting

Perl Difficulty

Hi, I am trying to upload a file to a SQL database table. The column type is IMAGE. I am looking for a solution to upload a word doc file. I tried 3 approaches. 1) my $fileToStore = "mytest.doc"; open IPFILE, "<", $name; binmode IPFILE; while (<IPFILE>) { $fileToStore .= $_; } close... (1 Reply)
Discussion started by: b.paramanatti
1 Replies

3. Shell Programming and Scripting

Loop difficulty

hi all I am new to unix and want to create a loop to repeat the loop and before that it ask me to do so.I know "while" may help but I put it in my work and getting stuk with it.any help appreciated. (13 Replies)
Discussion started by: samsami1971
13 Replies

4. UNIX and Linux Applications

Unable to connect to a website using Curl

Hello, I am using Curl command to make a function call to a web service. The issue I am facing right now is: curl h**p://www.google.com with the above command, I get a XML file as a response. Also when i execute : wget h**p://www.google.com Replace ** : tt I can see status as... (1 Reply)
Discussion started by: amitshete
1 Replies

5. Shell Programming and Scripting

difficulty with awk

hello folks, i am stuck with this awk command. i need to calculate the sum of a column of values on a flatfile and i am using the following command : awk -F"|" '{x += $10} END {print "Sum: "x}' standard_csv_file1.out that flatfile contains 180 fields and i am getting the... (5 Replies)
Discussion started by: jdsony
5 Replies

6. UNIX for Dummies Questions & Answers

Difficulty with set - command

Hi , i have a shell script with the first two lines as #! /bin/ksh set -x when i try opening the file from unix using the command 'sh filename.sh' , i am getting the below error : invalid optionline 2: set: - Pls help Use code tags for you code and data... (1 Reply)
Discussion started by: Rajankum
1 Replies

7. Shell Programming and Scripting

Scraping line - Using awk or sed

Hello, Can somone help with this command please? I have this output pattern in a file. I use a simple awk command to print each field separated by comma. For example I use this to get the first awk -F, "{ print $1 }" "ABC=abcdefg,CDF=mnqrst,GGG=hrvyess" issue: What I... (7 Replies)
Discussion started by: mnassiri
7 Replies

8. Programming

Python Web Page Scraping Urls Creating A Dictionary

I have thrown in the towel and cant figure out how to do this. I have a directory of html files that contain urls that I need to scrape (loop through) and add into a dictionary. An example of the output I would like is: bigbadwolf.htlm: https://www.blah.com, http://www.blahblah.com,... (5 Replies)
Discussion started by: metallica1973
5 Replies
shishi_tkts_new(3)						      shishi							shishi_tkts_new(3)

NAME
shishi_tkts_new - API function SYNOPSIS
#include <shishi.h> int shishi_tkts_new(Shishi_tkts * tkts, Shishi_asn1 ticket, Shishi_asn1 enckdcreppart, Shishi_asn1 kdcrep); ARGUMENTS
Shishi_tkts * tkts ticket set handle as allocated by shishi_tkts(). Shishi_asn1 ticket input ticket variable. Shishi_asn1 enckdcreppart input ticket detail variable. Shishi_asn1 kdcrep input KDC-REP variable. DESCRIPTION
Allocate a new ticket and add it to the ticket set. Note that ticket, enckdcreppart and kdcrep are stored by reference, so you must not de-allocate them before the ticket is removed from the ticket set and de-allocated. RETURN VALUE
Returns SHISHI_OK iff successful. REPORTING BUGS
Report bugs to <bug-shishi@gnu.org>. COPYRIGHT
Copyright (C) 2002-2010 Simon Josefsson. Copying and distribution of this file, with or without modification, are permitted in any medium without royalty provided the copyright notice and this notice are preserved. SEE ALSO
The full documentation for shishi is maintained as a Texinfo manual. If the info and shishi programs are properly installed at your site, the command info shishi should give you access to the complete manual. shishi 1.0.1 shishi_tkts_new(3)
All times are GMT -4. The time now is 06:19 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy