Bash script - cygwin (powershell?) pull from GitHub API Parse JSON


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Bash script - cygwin (powershell?) pull from GitHub API Parse JSON
# 1  
Old 08-20-2014
Bash script - cygwin (powershell?) pull from GitHub API Parse JSON

All,

Have a weird issue where i need to generate a report from GitHub monthly detailing user accounts and the last time they logged in. I'm using a windows box to do this (work issued) and would like to know if anyone has any experience scripting for GitAPI using windows / cygwin / powershell?

I would normally curl for the stuff i wanted and use jsawk to parse the returned JSON, but that's not working (and i don't see an option to add jsawk to Cygwin) so i'm at a loss: Any help greatly appreciated.

Edit: thought i would add that the code block below is working except for the 'jsawk' operation. I was just including that as an example of how i would do this in a real terminal.



Code:
#!/bin/bash
IFS_bak=$IFS
IFS=$'\r\n'
uid='xxxxxxx'
GH_OAUTH='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
curl -i --silent https://api.github.com:443/orgs/xxxxxxxxxxx/repos?access_token=$GH_OAUTH
for RepoLine in `curl --silent https://api.github.com:443/orgs/xxxxxxxxxxxxxxx/repos?access_token=$GH_OAUTH | _
  jsawk -n 'out(this.name)'`
do 
   RepoName=$(echo $RepoLine | awk -F" : " '{ print $1 }')
curl -i --silent https://api.github.com/repos/xxxxxxxxxxxx/$RepoName/teams?access_token=$GH_OAUTH


Last edited by ChocoTaco; 08-20-2014 at 07:23 PM..
# 2  
Old 08-21-2014
A bit of a hack, but without jsawk, you might try and get away with something like:
Code:
awk -F\" '$2==s{print $(NF-1)}' s=name

It may be enough for your application..
This User Gave Thanks to Scrutinizer For This Post:
# 3  
Old 08-21-2014
Scrutinizer, thank you for the suggestion. That worked although i ran into some other issues after the fact. Long story - in the end i broke down and had the IT guys bring me an old PC destined for the recycle bin and I installed CentOS 6.5 on it so we can do this the right way Smilie . Now i'm running into another issue and i'm not sure what what may be wrong here: Original code block -

Code:
#!/bin/bash
IFS_bak=$IFS
IFS=$'\r\n'
uid='xxxxxxxxxxxxxxxxxxxxxxxxxxx'
GH_OAUTH='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
curl -i --silent https://api.github.com:443/orgs/xxxxxxxxxxxxxx/repos?access_token=$GH_OAUTH
for RepoLine in `curl --silent https://api.github.com:443/orgs/xxxxxxxxxxx/repos?access_token=$GH_OAUTH | _
  jsawk -n 'out(this.name)'`
do 
   RepoName=$(echo $RepoLine | awk -F" : " '{ print $1 }')
curl -i --silent https://api.github.com/repos/xxxxxxxxxxxxxx/$RepoName/teams?access_token=$GH_OAUTH

Getting an error in terminal"

"Line 12: Syntax error: Unexpected End of File. "

Seems like something simple, any ideas?
# 4  
Old 08-21-2014
You're welcome! There seems to be a done missing. What is the underscore doing?
# 5  
Old 08-27-2014
Right on both counts good sir! Thank you for your time.

---------- Post updated 08-27-14 at 12:14 PM ---------- Previous update was 08-26-14 at 04:49 PM ----------

Didn't want to start a new thread for this, sorry if this reply is too old but wanted to ask another question.

I've got the script below working, however it's not really doing what i want and i must be missing something.

these are the requirement requirements:

Initial curl ---- return list of repos for our org
"foreach" repository name in list returned (JSON) --- curl for list of users with access
"foreach" repo user list returned (JSON) ---- output usernames to a txt, csv, whatever file for auditing.

I'm struggling a little here as i'm not very experienced with bash shell scripting and jsawk in particular. Turns out there is no direct way to "for each" is bash (as far as i can tell) and i think the script may not be looping correctly. Any help appreciated, thanks for having a read.




Code:
#!/bin/bash -x
IFS_bak=$IFS
IFS=$'\r\n'
uid='xxxxxxxxxxxxxxxxxxxx'
GH_OAUTH='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
curl -i  https://api.github.com:443/orgs/xxxxxxxxxxxxxxxxxxxxxxxxxxx/repos?access_token=$GH_OAUTH
for RepoLine in `curl --silent https://api.github.com:443/orgs/xxxxxxxxxxxxxxxxxxxxx/repos?access_token=$GH_OAUTH |
  jsawk -n 'out(this.name)'`
do
  RepoName=$(echo $RepoLine | awk -F" : " '{ print $1 }')
   curl -i https://api.github.com/repos/xxxxxxxxxxxxxxxxxxxxxxxxx/$RepoName/teams?access_token=$GH_OAUTH | 
  jsawk -n 'out(this.login)' > output.txt
done


Last edited by ChocoTaco; 08-27-2014 at 01:32 PM.. Reason: Edited code block, wrong paste
# 6  
Old 08-27-2014
Can you post a sample of what your first curl downloads and also what your first jsawk produces with this as input?

There is a second do statement in your post, that I think should not be there..
# 7  
Old 08-27-2014
Scrutinizer, here are the files requested: The first shows the API curl response, the second outputs the jsawk stuff from mem, had to change the code a little to get the jsawk out, here's what i did --->

sorry not that good with awk / jsawk still learning:

Code:
#!/bin/bash
FS_bak=$IFS
IFS=$'\r\n'
uid='XXXXXXXXXXXXXXXXXX'
GH_OAUTH='XXXXXXXXXXXXXXXXXXXXXXXXX'
curl --silent  https://api.github.com:443/orgs/XXXXXXXXXXXXX/repos?access_token=$GH_OAUTH > output.txt
cat output.txt | jsawk -n 'out(this.name)' | jsawk 'return this.name' | > outputjsawk.txt

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Parse property from json file

Hello All, Greetings.. I have a json file that I need to pase its URLs and other values. The match should start with "notifications" and output URLs and settings values. I tried with python or awk but hardly could get URLs only. Or whole URLs from full json file. Could not match... (2 Replies)
Discussion started by: 7adi
2 Replies

2. Shell Programming and Scripting

Building JSON command with bash script

Hello. I'm new to bash script and I'm learning the basics by writing some scripts. Recently a friend of mine asked me if I could try to write a script to him to automate a couple of processes that uses JSON RPCs. I'll try to explain in few words the workflow just to contextualize the problem.... (48 Replies)
Discussion started by: psysc0rpi0n
48 Replies

3. UNIX for Advanced & Expert Users

Running Powershell Script from Linux through Cygwin

Hello Experts, I am creating a run time powershell script on Linux machine and copying that powershell script to Windows machine. To connect to windows through "ssh", I am using Cygwin tool. To make the connection password less I copied my public in authorized_keys in windows Administrator... (5 Replies)
Discussion started by: shekhar_4_u
5 Replies

4. Shell Programming and Scripting

Parsing and Editing a json file with bash script

I am trying to automate editing of a json file using bash script. The file I initially receive is { "appMap": { "URL1": { "name": "a" }, "URL2": { "name": "b" }, "URL3": { "name": "c" }, } WHat I would like to do is replace... (5 Replies)
Discussion started by: Junaid Subhani
5 Replies

5. Shell Programming and Scripting

Bash Script to pull ipa server name on 500 servers

Hello All, I need help writing a bash script that will run on 500 LINUX servers and do the following: 1. Capture the ipa_server name from /etc/sssd/sssd.conf on a list of 500 servers in the ipahosts file. 2. Write to a file outputing only server name and IPA server name. Root ssh keys... (3 Replies)
Discussion started by: vtowntechy
3 Replies

6. Shell Programming and Scripting

How to define a variable in a BASH script by using a JSON file online?

Hello, I would like to modify an existing script of mine that uses a manually defined "MCVERSION" variable and make it define that variable instead based on this JSON file stored online: https://s3.amazonaws.com/Minecraft.Download/versions/versions.json Within that JSON, I 'm looking for... (4 Replies)
Discussion started by: nbsparks
4 Replies

7. Shell Programming and Scripting

How to start powershell with shebang from windows/cygwin/bash?

I would like to the the windws8/cygwin/bash shebang feature to start a powershell script. I do a "chmod +x set-sound.ps1" and then at a bash prompt I do ./set-sound.ps1 The first line of ./set-sound.ps1 #!powershell.exe -ExecutionPolicy unrestricted The result is the result: ... (5 Replies)
Discussion started by: siegfried
5 Replies

8. UNIX for Dummies Questions & Answers

GitHub documentation :: GitHub for dummies

Hi I wish to "develop" for brew. It is hosted on GitHub. What book (or online documentation) do you recommend me for GitHub and/or Git? Prefer small physical book. (1 Reply)
Discussion started by: slashdotweenie
1 Replies

9. Shell Programming and Scripting

shell / bash / script api ?

Hi is there a good dokumentation for shell scripting ? like the api in java ? didnt find a good one yet (5 Replies)
Discussion started by: Turrican
5 Replies

10. UNIX for Dummies Questions & Answers

How to use cygwin to run bash script

Hi, all, I try to run a quite simple bash script mytest.sh in cygwin, it's content is: #!/bin/bash echo "It is my first bash shell" there are three lines in the script. The second line is blank line. When I run it use command: bash c:/mytest.sh, ... (6 Replies)
Discussion started by: Jenny.palmy
6 Replies
Login or Register to Ask a Question