Get data from GitHub using Curl Post: 302995154

Sponsored Content

Top Forums Shell Programming and Scripting Get data from GitHub using Curl Post 302995154 by Kochappa on Sunday 2nd of April 2017 04:13:06 AM

04-02-2017

Registered User

Get data from GitHub using Curl

Dear friends,

I am working on a bash script to get data from the github by hitting the API using curl and this is quite straight forward. However the authentication mechanism we have in our github servers makes this intricate.

Here is the situation :

* The github uses SSO / SAML / opensso for authentication.
* When a user tries to authenticate, it redirects the user to the SSO page and it uses the system certificate to authenticate
* After successfully authenticating, it redirects back to the github page

When I try "curl'ing" the URL, it redirects to SSO. So I passed the -L but that too didn't help.
I have also tried passing the cert and key using --cacert and --key but no luck.
Then I accessed the git using firefox, turned-on the developer tools and copied the "CURL URL" from the network tab which gave me something like below.

Code:

curl -k  "https://xyz.com/repo/Support/issues/" -H "Host: xyz.com" -H "User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:45.0) Gecko/20100101 Firefox/45.0" -H "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" -H "Accept-Language: en-US,en;q=0.5" --compressed -H "DNT: 1" -H "Cookie: logged_in=yes; user_session=jhdfudhfdjfugn6387384knfsf_8lKjQ7Ye5WGDQC5lySTHvuvEWlkdmfHYFklm976876fknJGF79fkkw9MKVx; dotcom_user=username-name; _gh_render=jBiMzdkMzFhMDYyYjYG"%"0AOwBGSSIPdXNlcl9sb2dpbgY7AEZJIhZmYWhlZW0tcG9ubmFtYmF0aAY7AFQ"%"3D"%"0ABAh7B0kiD3Nlc3Npb25faWQGOgZFVEkiRThhNWJhZWE2OTI5NDc2NWQ0NDkx"%"0ANWUyYTQyNTJkM2QwOWQwNDI5ZDU4NzlhZTYzMzI3N--f73aac91ec9d307fcb9c72f1b532427604c76f00; _fi_sess=GY3ZTgiLCJzcHlfcmVwbyI6IkFnaWxlRGVsaXZlcnlQbGF0Zm9AyZDA2OTIxYzE4YWZhZWE3OTIzMJ0Iiwic3B5X3JlcG9ybS9TdXBwb3eyJzZXNzaW9uX2lkIjoiMmQzYjkwYTfYXQiOjE0OTA5NDI2MTV9--77f8e92a1bac271c284f2b11f269b57a099f1992; tz=Asia"%"2FKolkata" -H "Connection: keep-alive" -H "Cache-Control: max-age=0"

This worked! and I believe its the user_session key which got me in.
Please can someone help to find how I can generate this user_session using the curl? so that I can use it and curl again to authenticate and get the data.
I have generate a Personal Access Token from the GIT and passed to curl but that too didnt help. Below are my system details.

Code:

hostname:/home/user $ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.6 (Santiago)
hostname:/home/user $ uname
Linux user 2.6.32-642.11.1.el6.x86_64 #1 SMP Wed Oct 26 10:25:23 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
hostname:/home/user $ curl -V
curl 7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Protocols: tftp ftp telnet dict ldap ldaps http file https ftps scp sftp
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz

Kind Regards,
Kochappa.

Last edited by jim mcnamara; 04-02-2017 at 01:16 PM.. Reason: extra code tags

Kochappa

View Public Profile for Kochappa

Find all posts by Kochappa

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Posting data to a form using curl

Hello all. I have an incredible number of servers that I need to change a parameter on using a web interface. I'd like to be able to do this via curl, but I'm having some trouble. I filled out the form and hit update while snooping (tcpdump) my interface. That gave the the following as what is...

2. UNIX for Dummies Questions & Answers

GitHub documentation :: GitHub for dummies

Hi I wish to "develop" for brew. It is hosted on GitHub. What book (or online documentation) do you recommend me for GitHub and/or Git? Prefer small physical book.

3. UNIX for Dummies Questions & Answers

Read URL data from UNIX without wget,curl,lynx,w3m.

Hi Experts, Problem statement : We have an URL for which we need to read the data and get parsed inside the shell scripts. My Aix has very limited perl utility, i cant install any utility as well. Precisely, wget,cURL,Lynx,w3m and Lwp cant be used as i got these utilities only when i googled...

4. Shell Programming and Scripting

Read URL data from UNIX-CLI without Wget,CURL,w3m,LWP

Hi Experts, Problem statement : We have an URL for which we need to read the data and get parsed inside the shell scripts.My Aix has very limited perl utility, i cant install any utility as well. Precisely, wget,cURL,Lynx,w3m and Lwp cant be used as i got these utilities only when i googled it....

5. Shell Programming and Scripting

Curl Script - Post a file (multipart/form-data)

Hi All, I am trying to post an xml file (utf-16 encoded) using curl to a REST service. The REST service is expecting 'multipart/form-data' content type. curl -k -i -H "Content-Type=multipart/form-data" -F "filename=@file.xml;type=text/xml" -X POST -u <username>:<password> <endPointURL> ...

6. Web Development

Data Post with curl

how to make the bash script ? http://server.com/mysql.php POST /mysql.php HTTP/1.1 Host: server.com User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:26.0) Gecko/20100101 Firefox/26.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language:...

7. Shell Programming and Scripting

Wget and curl to post data

i'm using this command to post data to a remote host: wget --post-data="My Data" http://<my-ip>:80 -O /dev/null -q and curl --data "My Data" http://<my-ip>:80 however, when i run the above, i see the following in my access log on the remote host: Wget: 10.10.10.10 - - "POST /...

8. Shell Programming and Scripting

How to post content file of data using curl?

HI All, i want to asking about my case , how to post content file of data using curl. currently if i wanna post data file , i can use this command below : curl --cacert maxaj.cer -X POST -F 'fileupload=@/data/report_3300_xxx/log/traffic_3300_20180319_1051.txt'...

9. News, Links, Events and Announcements

Microsoft buys github

Another opensource bastion goes to Microsoft. Confirmed—Microsoft Buys GitHub For $7.5 Billion It's official: Microsoft buys open-source coding community GitHub for a reported $7.5 billion | Alphr Microsoft Is Buying GitHub. Here'''s Why. | Fortune Why Microsoft is buying GitHub: It's all...

LEARN ABOUT MOJAVE

curlopt_useragent

CURLOPT_USERAGENT(3)					     curl_easy_setopt options					      CURLOPT_USERAGENT(3)

NAME

       CURLOPT_USERAGENT - set HTTP user-agent header

SYNOPSIS

       #include <curl/curl.h>

       CURLcode curl_easy_setopt(CURL *handle, CURLOPT_USERAGENT, char *ua);

DESCRIPTION

       Pass  a	pointer  to  a	zero terminated string as parameter. It will be used to set the User-Agent: header in the HTTP request sent to the
       remote server. This can be used to fool servers or scripts. You can also set any custom header with CURLOPT_HTTPHEADER(3).

       The application does not have to keep the string around after setting this option.

DEFAULT

       NULL, no User-Agent: header is used by default.

PROTOCOLS

       HTTP, HTTPS

EXAMPLE

       CURL *curl = curl_easy_init();
       if(curl) {
	 curl_easy_setopt(curl, CURLOPT_URL, "http://example.com");

	 curl_easy_setopt(curl, CURLOPT_USERAGENT, "Dark Secret Ninja/1.0");

	 curl_easy_perform(curl);
       }

AVAILABILITY

       As long as HTTP is supported

RETURN VALUE

       Returns CURLE_OK if HTTP is supported, CURLE_UNKNOWN_OPTION if not, or CURLE_OUT_OF_MEMORY if there was insufficient heap space.

SEE ALSO

       CURLOPT_REFERER(3), CURLOPT_HTTPHEADER(3),

libcurl 7.54.0							 December 21, 2016					      CURLOPT_USERAGENT(3)