Get data from GitHub using Curl


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Get data from GitHub using Curl
# 1  
Old 04-02-2017
Get data from GitHub using Curl

Dear friends,

I am working on a bash script to get data from the github by hitting the API using curl and this is quite straight forward. However the authentication mechanism we have in our github servers makes this intricate.

Here is the situation :

* The github uses SSO / SAML / opensso for authentication.
* When a user tries to authenticate, it redirects the user to the SSO page and it uses the system certificate to authenticate
* After successfully authenticating, it redirects back to the github page

When I try "curl'ing" the URL, it redirects to SSO. So I passed the -L but that too didn't help.
I have also tried passing the cert and key using --cacert and --key but no luck.
Then I accessed the git using firefox, turned-on the developer tools and copied the "CURL URL" from the network tab which gave me something like below.

Code:
curl -k  "https://xyz.com/repo/Support/issues/" -H "Host: xyz.com" -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" -H "Accept-Language: en-US,en;q=0.5" --compressed -H "DNT: 1" -H "Cookie: logged_in=yes; user_session=jhdfudhfdjfugn6387384knfsf_8lKjQ7Ye5WGDQC5lySTHvuvEWlkdmfHYFklm976876fknJGF79fkkw9MKVx; dotcom_user=username-name; _gh_render=jBiMzdkMzFhMDYyYjYG"%"0AOwBGSSIPdXNlcl9sb2dpbgY7AEZJIhZmYWhlZW0tcG9ubmFtYmF0aAY7AFQ"%"3D"%"0ABAh7B0kiD3Nlc3Npb25faWQGOgZFVEkiRThhNWJhZWE2OTI5NDc2NWQ0NDkx"%"0ANWUyYTQyNTJkM2QwOWQwNDI5ZDU4NzlhZTYzMzI3N--f73aac91ec9d307fcb9c72f1b532427604c76f00; _fi_sess=GY3ZTgiLCJzcHlfcmVwbyI6IkFnaWxlRGVsaXZlcnlQbGF0Zm9AyZDA2OTIxYzE4YWZhZWE3OTIzMJ0Iiwic3B5X3JlcG9ybS9TdXBwb3eyJzZXNzaW9uX2lkIjoiMmQzYjkwYTfYXQiOjE0OTA5NDI2MTV9--77f8e92a1bac271c284f2b11f269b57a099f1992; tz=Asia"%"2FKolkata" -H "Connection: keep-alive" -H "Cache-Control: max-age=0"

This worked! and I believe its the user_session key which got me in.
Please can someone help to find how I can generate this user_session using the curl? so that I can use it and curl again to authenticate and get the data.
I have generate a Personal Access Token from the GIT and passed to curl but that too didnt help. Below are my system details.
Code:
hostname:/home/user $ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.6 (Santiago)
hostname:/home/user $ uname
Linux user 2.6.32-642.11.1.el6.x86_64 #1 SMP Wed Oct 26 10:25:23 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
hostname:/home/user $ curl -V
curl 7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Protocols: tftp ftp telnet dict ldap ldaps http file https ftps scp sftp
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz

Kind Regards,
Kochappa.

Last edited by jim mcnamara; 04-02-2017 at 01:16 PM.. Reason: extra code tags
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. News, Links, Events and Announcements

Microsoft buys github

Another opensource bastion goes to Microsoft. Confirmed—Microsoft Buys GitHub For $7.5 Billion It's official: Microsoft buys open-source coding community GitHub for a reported $7.5 billion | Alphr Microsoft Is Buying GitHub. Here'''s Why. | Fortune Why Microsoft is buying GitHub: It's all... (2 Replies)
Discussion started by: MadeInGermany
2 Replies

2. Shell Programming and Scripting

How to post content file of data using curl?

HI All, i want to asking about my case , how to post content file of data using curl. currently if i wanna post data file , i can use this command below : curl --cacert maxaj.cer -X POST -F 'fileupload=@/data/report_3300_xxx/log/traffic_3300_20180319_1051.txt'... (0 Replies)
Discussion started by: fajar_3t3
0 Replies

3. Shell Programming and Scripting

Wget and curl to post data

i'm using this command to post data to a remote host: wget --post-data="My Data" http://<my-ip>:80 -O /dev/null -q and curl --data "My Data" http://<my-ip>:80 however, when i run the above, i see the following in my access log on the remote host: Wget: 10.10.10.10 - - "POST /... (1 Reply)
Discussion started by: SkySmart
1 Replies

4. Web Development

Data Post with curl

how to make the bash script ? http://server.com/mysql.php POST /mysql.php HTTP/1.1 Host: server.com User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:26.0) Gecko/20100101 Firefox/26.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language:... (1 Reply)
Discussion started by: iddfcr
1 Replies

5. Shell Programming and Scripting

Curl Script - Post a file (multipart/form-data)

Hi All, I am trying to post an xml file (utf-16 encoded) using curl to a REST service. The REST service is expecting 'multipart/form-data' content type. curl -k -i -H "Content-Type=multipart/form-data" -F "filename=@file.xml;type=text/xml" -X POST -u <username>:<password> <endPointURL> ... (0 Replies)
Discussion started by: Anooja G
0 Replies

6. Shell Programming and Scripting

Read URL data from UNIX-CLI without Wget,CURL,w3m,LWP

Hi Experts, Problem statement : We have an URL for which we need to read the data and get parsed inside the shell scripts.My Aix has very limited perl utility, i cant install any utility as well. Precisely, wget,cURL,Lynx,w3m and Lwp cant be used as i got these utilities only when i googled it.... (12 Replies)
Discussion started by: scott_cog
12 Replies

7. UNIX for Dummies Questions & Answers

Read URL data from UNIX without wget,curl,lynx,w3m.

Hi Experts, Problem statement : We have an URL for which we need to read the data and get parsed inside the shell scripts. My Aix has very limited perl utility, i cant install any utility as well. Precisely, wget,cURL,Lynx,w3m and Lwp cant be used as i got these utilities only when i googled... (0 Replies)
Discussion started by: scott_cog
0 Replies

8. UNIX for Dummies Questions & Answers

GitHub documentation :: GitHub for dummies

Hi I wish to "develop" for brew. It is hosted on GitHub. What book (or online documentation) do you recommend me for GitHub and/or Git? Prefer small physical book. (1 Reply)
Discussion started by: slashdotweenie
1 Replies

9. UNIX for Dummies Questions & Answers

Posting data to a form using curl

Hello all. I have an incredible number of servers that I need to change a parameter on using a web interface. I'd like to be able to do this via curl, but I'm having some trouble. I filled out the form and hit update while snooping (tcpdump) my interface. That gave the the following as what is... (0 Replies)
Discussion started by: DeCoTwc
0 Replies
Login or Register to Ask a Question