Sponsored Content
Top Forums Shell Programming and Scripting Get data from GitHub using Curl Post 302995154 by Kochappa on Sunday 2nd of April 2017 04:13:06 AM
Old 04-02-2017
Get data from GitHub using Curl

Dear friends,

I am working on a bash script to get data from the github by hitting the API using curl and this is quite straight forward. However the authentication mechanism we have in our github servers makes this intricate.

Here is the situation :

* The github uses SSO / SAML / opensso for authentication.
* When a user tries to authenticate, it redirects the user to the SSO page and it uses the system certificate to authenticate
* After successfully authenticating, it redirects back to the github page

When I try "curl'ing" the URL, it redirects to SSO. So I passed the -L but that too didn't help.
I have also tried passing the cert and key using --cacert and --key but no luck.
Then I accessed the git using firefox, turned-on the developer tools and copied the "CURL URL" from the network tab which gave me something like below.

Code:
curl -k  "https://xyz.com/repo/Support/issues/" -H "Host: xyz.com" -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" -H "Accept-Language: en-US,en;q=0.5" --compressed -H "DNT: 1" -H "Cookie: logged_in=yes; user_session=jhdfudhfdjfugn6387384knfsf_8lKjQ7Ye5WGDQC5lySTHvuvEWlkdmfHYFklm976876fknJGF79fkkw9MKVx; dotcom_user=username-name; _gh_render=jBiMzdkMzFhMDYyYjYG"%"0AOwBGSSIPdXNlcl9sb2dpbgY7AEZJIhZmYWhlZW0tcG9ubmFtYmF0aAY7AFQ"%"3D"%"0ABAh7B0kiD3Nlc3Npb25faWQGOgZFVEkiRThhNWJhZWE2OTI5NDc2NWQ0NDkx"%"0ANWUyYTQyNTJkM2QwOWQwNDI5ZDU4NzlhZTYzMzI3N--f73aac91ec9d307fcb9c72f1b532427604c76f00; _fi_sess=GY3ZTgiLCJzcHlfcmVwbyI6IkFnaWxlRGVsaXZlcnlQbGF0Zm9AyZDA2OTIxYzE4YWZhZWE3OTIzMJ0Iiwic3B5X3JlcG9ybS9TdXBwb3eyJzZXNzaW9uX2lkIjoiMmQzYjkwYTfYXQiOjE0OTA5NDI2MTV9--77f8e92a1bac271c284f2b11f269b57a099f1992; tz=Asia"%"2FKolkata" -H "Connection: keep-alive" -H "Cache-Control: max-age=0"

This worked! and I believe its the user_session key which got me in.
Please can someone help to find how I can generate this user_session using the curl? so that I can use it and curl again to authenticate and get the data.
I have generate a Personal Access Token from the GIT and passed to curl but that too didnt help. Below are my system details.
Code:
hostname:/home/user $ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.6 (Santiago)
hostname:/home/user $ uname
Linux user 2.6.32-642.11.1.el6.x86_64 #1 SMP Wed Oct 26 10:25:23 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
hostname:/home/user $ curl -V
curl 7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Protocols: tftp ftp telnet dict ldap ldaps http file https ftps scp sftp
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz

Kind Regards,
Kochappa.

Last edited by jim mcnamara; 04-02-2017 at 01:16 PM.. Reason: extra code tags
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Posting data to a form using curl

Hello all. I have an incredible number of servers that I need to change a parameter on using a web interface. I'd like to be able to do this via curl, but I'm having some trouble. I filled out the form and hit update while snooping (tcpdump) my interface. That gave the the following as what is... (0 Replies)
Discussion started by: DeCoTwc
0 Replies

2. UNIX for Dummies Questions & Answers

GitHub documentation :: GitHub for dummies

Hi I wish to "develop" for brew. It is hosted on GitHub. What book (or online documentation) do you recommend me for GitHub and/or Git? Prefer small physical book. (1 Reply)
Discussion started by: slashdotweenie
1 Replies

3. UNIX for Dummies Questions & Answers

Read URL data from UNIX without wget,curl,lynx,w3m.

Hi Experts, Problem statement : We have an URL for which we need to read the data and get parsed inside the shell scripts. My Aix has very limited perl utility, i cant install any utility as well. Precisely, wget,cURL,Lynx,w3m and Lwp cant be used as i got these utilities only when i googled... (0 Replies)
Discussion started by: scott_cog
0 Replies

4. Shell Programming and Scripting

Read URL data from UNIX-CLI without Wget,CURL,w3m,LWP

Hi Experts, Problem statement : We have an URL for which we need to read the data and get parsed inside the shell scripts.My Aix has very limited perl utility, i cant install any utility as well. Precisely, wget,cURL,Lynx,w3m and Lwp cant be used as i got these utilities only when i googled it.... (12 Replies)
Discussion started by: scott_cog
12 Replies

5. Shell Programming and Scripting

Curl Script - Post a file (multipart/form-data)

Hi All, I am trying to post an xml file (utf-16 encoded) using curl to a REST service. The REST service is expecting 'multipart/form-data' content type. curl -k -i -H "Content-Type=multipart/form-data" -F "filename=@file.xml;type=text/xml" -X POST -u <username>:<password> <endPointURL> ... (0 Replies)
Discussion started by: Anooja G
0 Replies

6. Web Development

Data Post with curl

how to make the bash script ? http://server.com/mysql.php POST /mysql.php HTTP/1.1 Host: server.com User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:26.0) Gecko/20100101 Firefox/26.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language:... (1 Reply)
Discussion started by: iddfcr
1 Replies

7. Shell Programming and Scripting

Wget and curl to post data

i'm using this command to post data to a remote host: wget --post-data="My Data" http://<my-ip>:80 -O /dev/null -q and curl --data "My Data" http://<my-ip>:80 however, when i run the above, i see the following in my access log on the remote host: Wget: 10.10.10.10 - - "POST /... (1 Reply)
Discussion started by: SkySmart
1 Replies

8. Shell Programming and Scripting

How to post content file of data using curl?

HI All, i want to asking about my case , how to post content file of data using curl. currently if i wanna post data file , i can use this command below : curl --cacert maxaj.cer -X POST -F 'fileupload=@/data/report_3300_xxx/log/traffic_3300_20180319_1051.txt'... (0 Replies)
Discussion started by: fajar_3t3
0 Replies

9. News, Links, Events and Announcements

Microsoft buys github

Another opensource bastion goes to Microsoft. Confirmed—Microsoft Buys GitHub For $7.5 Billion It's official: Microsoft buys open-source coding community GitHub for a reported $7.5 billion | Alphr Microsoft Is Buying GitHub. Here'''s Why. | Fortune Why Microsoft is buying GitHub: It's all... (2 Replies)
Discussion started by: MadeInGermany
2 Replies
Net::GitHub::V3(3pm)					User Contributed Perl Documentation				      Net::GitHub::V3(3pm)

NAME
Net::GitHub::V3 - Github API v3 SYNOPSIS
Prefer: use Net::GitHub; my $gh = Net::GitHub->new( version => 3, login => 'fayland', pass => 'mypass', # or # access_token => $oauth_token ); Or: use Net::GitHub::V3; my $gh = Net::GitHub::V3->new( login => 'fayland', pass => 'mypass', # or # access_token => $oauth_token ); DESCRIPTION
<http://develop.github.com/> ATTRIBUTES Authentication There are two ways to authenticate through GitHub API v3: login/pass my $gh = Net::GitHub::V3->new( login => $ENV{GITHUB_USER}, pass => $ENV{GITHUB_PASS} ); access_token my $gh = Net::GitHub->new( access_token => $ENV{GITHUB_ACCESS_TOKEN} ); raw_response my $gh = Net::GitHub->new( # login/pass or access_token raw_response => 1 ); return raw HTTP::Response object raw_string my $gh = Net::GitHub->new( # login/pass or access_token raw_string => 1 ); return HTTP::Response response content as string api_throttle my $gh = Net::GitHub->new( # login/pass or access_token api_throttle => 0 ); To disable call rate limiting (e.g. if your account is whitelisted), set api_throttle to 0. RaiseError By default, error responses are propagated to the user as they are received from the API. By switching RaiseError on you can make the be turned into exceptions instead, so that you don't have to check for error response after every call. next_url, last_url, prev_url, first_url Any methods which return multiple results may be paginated. After performing a query you should check to see if there are more results. These attributes will be reset for each query. The predicates to check these attributes are "has_next_page", "has_last_page", "has_prev_page" and "has_first_page". See Github's documentation: <http://developer.github.com/v3/#pagination> The "per_page" parameter mentioned in their docs is NOT supported by this module. my @issues = $gh->issue->repos_issues; while ($gh->issue->has_next_page) { push @issues, $gh->issue->query($gh->issue->next_url); ## OR ## push @issues, $gh->issue->next_page); } METHODS query($method, $url, $data) my $data = $gh->query('/user'); $gh->query('PATCH', '/user', $data); $gh->query('DELETE', '/user/emails', [ 'myemail@somewhere.com' ]); query API directly next_page When the results have been paginated, "next_page" is sugar for the common case of iterating through all the pages in order. It simply calls "query" with the "next_url". set_default_user_repo $gh->set_default_user_repo('fayland', 'perl-net-github'); # take effects for all $gh-> $gh->repos->set_default_user_repo('fayland', 'perl-net-github'); # take effects on $gh->repos To ease the keyboard, we provided two ways to call any method which starts with :user/:repo 1. SET user/repos before call methods below $gh->set_default_user_repo('fayland', 'perl-net-github'); my @contributors = $gh->repos->contributors; 2. If it is just for once, we can pass :user, :repo before any arguments my @contributors = $repos->contributors($user, $repo); MODULES user my $user = $gh->user->show('nothingmuch'); $gh->user->update( bio => 'Just Another Perl Programmer' ); Net::GitHub::V3::Users repos my @repos = $gh->repos->list; my $rp = $gh->repos->create( { "name" => "Hello-World", "description" => "This is your first repo", "homepage" => "https://github.com" } ); Net::GitHub::V3::Repos issue my @issues = $gh->issue->issues(); my $issue = $gh->issue->issue($issue_id); Net::GitHub::V3::Issues pull_request my @pulls = $gh->pull_request->pulls(); Net::GitHub::V3::PullRequests org my @orgs = $gh->org->orgs; Net::GitHub::V3::Orgs git_data Net::GitHub::V3::GitData gist Net::GitHub::V3::Gists oauth Net::GitHub::V3::OAuth event Net::GitHub::V3::Events SEE ALSO
Any::Moose, Pithub AUTHOR &; COPYRIGHT & LICENSE Refer Net::GitHub perl v5.14.2 2012-05-03 Net::GitHub::V3(3pm)
All times are GMT -4. The time now is 12:16 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy