Get data from GitHub using Curl Post: 302995154

Sponsored Content

Top Forums Shell Programming and Scripting Get data from GitHub using Curl Post 302995154 by Kochappa on Sunday 2nd of April 2017 04:13:06 AM

04-02-2017

Registered User

Get data from GitHub using Curl

Dear friends,

I am working on a bash script to get data from the github by hitting the API using curl and this is quite straight forward. However the authentication mechanism we have in our github servers makes this intricate.

Here is the situation :

* The github uses SSO / SAML / opensso for authentication.
* When a user tries to authenticate, it redirects the user to the SSO page and it uses the system certificate to authenticate
* After successfully authenticating, it redirects back to the github page

When I try "curl'ing" the URL, it redirects to SSO. So I passed the -L but that too didn't help.
I have also tried passing the cert and key using --cacert and --key but no luck.
Then I accessed the git using firefox, turned-on the developer tools and copied the "CURL URL" from the network tab which gave me something like below.

Code:

curl -k  "https://xyz.com/repo/Support/issues/" -H "Host: xyz.com" -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" -H "Accept-Language: en-US,en;q=0.5" --compressed -H "DNT: 1" -H "Cookie: logged_in=yes; user_session=jhdfudhfdjfugn6387384knfsf_8lKjQ7Ye5WGDQC5lySTHvuvEWlkdmfHYFklm976876fknJGF79fkkw9MKVx; dotcom_user=username-name; _gh_render=jBiMzdkMzFhMDYyYjYG"%"0AOwBGSSIPdXNlcl9sb2dpbgY7AEZJIhZmYWhlZW0tcG9ubmFtYmF0aAY7AFQ"%"3D"%"0ABAh7B0kiD3Nlc3Npb25faWQGOgZFVEkiRThhNWJhZWE2OTI5NDc2NWQ0NDkx"%"0ANWUyYTQyNTJkM2QwOWQwNDI5ZDU4NzlhZTYzMzI3N--f73aac91ec9d307fcb9c72f1b532427604c76f00; _fi_sess=GY3ZTgiLCJzcHlfcmVwbyI6IkFnaWxlRGVsaXZlcnlQbGF0Zm9AyZDA2OTIxYzE4YWZhZWE3OTIzMJ0Iiwic3B5X3JlcG9ybS9TdXBwb3eyJzZXNzaW9uX2lkIjoiMmQzYjkwYTfYXQiOjE0OTA5NDI2MTV9--77f8e92a1bac271c284f2b11f269b57a099f1992; tz=Asia"%"2FKolkata" -H "Connection: keep-alive" -H "Cache-Control: max-age=0"

This worked! and I believe its the user_session key which got me in.
Please can someone help to find how I can generate this user_session using the curl? so that I can use it and curl again to authenticate and get the data.
I have generate a Personal Access Token from the GIT and passed to curl but that too didnt help. Below are my system details.

Code:

hostname:/home/user $ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.6 (Santiago)
hostname:/home/user $ uname
Linux user 2.6.32-642.11.1.el6.x86_64 #1 SMP Wed Oct 26 10:25:23 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
hostname:/home/user $ curl -V
curl 7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Protocols: tftp ftp telnet dict ldap ldaps http file https ftps scp sftp
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz

Kind Regards,
Kochappa.

Last edited by jim mcnamara; 04-02-2017 at 01:16 PM.. Reason: extra code tags

Kochappa

View Public Profile for Kochappa

Find all posts by Kochappa

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Posting data to a form using curl

Hello all. I have an incredible number of servers that I need to change a parameter on using a web interface. I'd like to be able to do this via curl, but I'm having some trouble. I filled out the form and hit update while snooping (tcpdump) my interface. That gave the the following as what is...

2. UNIX for Dummies Questions & Answers

GitHub documentation :: GitHub for dummies

Hi I wish to "develop" for brew. It is hosted on GitHub. What book (or online documentation) do you recommend me for GitHub and/or Git? Prefer small physical book.

3. UNIX for Dummies Questions & Answers

Read URL data from UNIX without wget,curl,lynx,w3m.

Hi Experts, Problem statement : We have an URL for which we need to read the data and get parsed inside the shell scripts. My Aix has very limited perl utility, i cant install any utility as well. Precisely, wget,cURL,Lynx,w3m and Lwp cant be used as i got these utilities only when i googled...

4. Shell Programming and Scripting

Read URL data from UNIX-CLI without Wget,CURL,w3m,LWP

Hi Experts, Problem statement : We have an URL for which we need to read the data and get parsed inside the shell scripts.My Aix has very limited perl utility, i cant install any utility as well. Precisely, wget,cURL,Lynx,w3m and Lwp cant be used as i got these utilities only when i googled it....

5. Shell Programming and Scripting

Curl Script - Post a file (multipart/form-data)

Hi All, I am trying to post an xml file (utf-16 encoded) using curl to a REST service. The REST service is expecting 'multipart/form-data' content type. curl -k -i -H "Content-Type=multipart/form-data" -F "filename=@file.xml;type=text/xml" -X POST -u <username>:<password> <endPointURL> ...

6. Web Development

Data Post with curl

how to make the bash script ? http://server.com/mysql.php POST /mysql.php HTTP/1.1 Host: server.com User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:26.0) Gecko/20100101 Firefox/26.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language:...

7. Shell Programming and Scripting

Wget and curl to post data

i'm using this command to post data to a remote host: wget --post-data="My Data" http://<my-ip>:80 -O /dev/null -q and curl --data "My Data" http://<my-ip>:80 however, when i run the above, i see the following in my access log on the remote host: Wget: 10.10.10.10 - - "POST /...

8. Shell Programming and Scripting

How to post content file of data using curl?

HI All, i want to asking about my case , how to post content file of data using curl. currently if i wanna post data file , i can use this command below : curl --cacert maxaj.cer -X POST -F 'fileupload=@/data/report_3300_xxx/log/traffic_3300_20180319_1051.txt'...

9. News, Links, Events and Announcements

Microsoft buys github

Another opensource bastion goes to Microsoft. Confirmed—Microsoft Buys GitHub For $7.5 Billion It's official: Microsoft buys open-source coding community GitHub for a reported $7.5 billion | Alphr Microsoft Is Buying GitHub. Here'''s Why. | Fortune Why Microsoft is buying GitHub: It's all...

LEARN ABOUT FREEBSD

xo_emit

LIBXO(3)						   BSD Library Functions Manual 						  LIBXO(3)

NAME

     xo_emit -- emit formatted output based on format string and arguments

LIBRARY

     library ``libxo''

SYNOPSIS

     #include <libxo/xo.h>

     int
     xo_emit(const char *fmt, ...);

     int
     xo_emit_h(xo_handle_t *xop, const char *fmt, ...);

     int
     xo_emit_hv(xo_handle_t *xop, const char *fmt, va_list vap);

DESCRIPTION

     The xo_emit() function emits formatted output using the description in a format string along with a set of zero or more arguments, in a style
     similar to printf(3) but using a more complex format description string, as described in xo_format(5).

     xo_emit() uses the default output handle, as described in libxo(3), where xo_emit_h() uses an explicit handle.  xo_emit_hv() accepts a
     va_list for additional flexibility.

ADDITIONAL DOCUMENTATION

     Complete documentation can be found on github:

	   http://juniper.github.io/libxo/libxo-manual.html

     libxo lives on github as:

	   https://github.com/Juniper/libxo

     The latest release of libxo is available at:

	   https://github.com/Juniper/libxo/releases

SEE ALSO

     xo_open_container(3), xo_open_list(3), xo_format(5)

HISTORY

     The libxo library was added in FreeBSD 11.0.

AUTHOR

     Phil Shafer

BSD
								 December 4, 2014							       BSD