Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Bash script - cygwin (powershell?) pull from GitHub API Parse JSON

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Bash script - cygwin (powershell?) pull from GitHub API Parse JSON
# 1  
Old 08-20-2014
Bash script - cygwin (powershell?) pull from GitHub API Parse JSON
All,

Have a weird issue where i need to generate a report from GitHub monthly detailing user accounts and the last time they logged in. I'm using a windows box to do this (work issued) and would like to know if anyone has any experience scripting for GitAPI using windows / cygwin / powershell?

I would normally curl for the stuff i wanted and use jsawk to parse the returned JSON, but that's not working (and i don't see an option to add jsawk to Cygwin) so i'm at a loss: Any help greatly appreciated.

Edit: thought i would add that the code block below is working except for the 'jsawk' operation. I was just including that as an example of how i would do this in a real terminal.



Code:
#!/bin/bash
IFS_bak=$IFS
IFS=$'\r\n'
uid='xxxxxxx'
GH_OAUTH='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
curl -i --silent https://api.github.com:443/orgs/xxxxxxxxxxx/repos?access_token=$GH_OAUTH
for RepoLine in `curl --silent https://api.github.com:443/orgs/xxxxxxxxxxxxxxx/repos?access_token=$GH_OAUTH | _
  jsawk -n 'out(this.name)'`
do 
   RepoName=$(echo $RepoLine | awk -F" : " '{ print $1 }')
curl -i --silent https://api.github.com/repos/xxxxxxxxxxxx/$RepoName/teams?access_token=$GH_OAUTH

Last edited by ChocoTaco; 08-20-2014 at 07:23 PM..
# 2  
Old 08-21-2014
A bit of a hack, but without jsawk, you might try and get away with something like:
Code:
awk -F\" '$2==s{print $(NF-1)}' s=name

It may be enough for your application..
This User Gave Thanks to Scrutinizer For This Post:
ChocoTaco
# 3  
Old 08-21-2014
Scrutinizer, thank you for the suggestion. That worked although i ran into some other issues after the fact. Long story - in the end i broke down and had the IT guys bring me an old PC destined for the recycle bin and I installed CentOS 6.5 on it so we can do this the right way Smilie . Now i'm running into another issue and i'm not sure what what may be wrong here: Original code block -

Code:
#!/bin/bash
IFS_bak=$IFS
IFS=$'\r\n'
uid='xxxxxxxxxxxxxxxxxxxxxxxxxxx'
GH_OAUTH='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
curl -i --silent https://api.github.com:443/orgs/xxxxxxxxxxxxxx/repos?access_token=$GH_OAUTH
for RepoLine in `curl --silent https://api.github.com:443/orgs/xxxxxxxxxxx/repos?access_token=$GH_OAUTH | _
  jsawk -n 'out(this.name)'`
do 
   RepoName=$(echo $RepoLine | awk -F" : " '{ print $1 }')
curl -i --silent https://api.github.com/repos/xxxxxxxxxxxxxx/$RepoName/teams?access_token=$GH_OAUTH

Getting an error in terminal"

"Line 12: Syntax error: Unexpected End of File. "

Seems like something simple, any ideas?
# 4  
Old 08-21-2014
You're welcome! There seems to be a done missing. What is the underscore doing?
# 5  
Old 08-27-2014
Right on both counts good sir! Thank you for your time.

---------- Post updated 08-27-14 at 12:14 PM ---------- Previous update was 08-26-14 at 04:49 PM ----------

Didn't want to start a new thread for this, sorry if this reply is too old but wanted to ask another question.

I've got the script below working, however it's not really doing what i want and i must be missing something.

these are the requirement requirements:

Initial curl ---- return list of repos for our org
"foreach" repository name in list returned (JSON) --- curl for list of users with access
"foreach" repo user list returned (JSON) ---- output usernames to a txt, csv, whatever file for auditing.

I'm struggling a little here as i'm not very experienced with bash shell scripting and jsawk in particular. Turns out there is no direct way to "for each" is bash (as far as i can tell) and i think the script may not be looping correctly. Any help appreciated, thanks for having a read.




Code:
#!/bin/bash -x
IFS_bak=$IFS
IFS=$'\r\n'
uid='xxxxxxxxxxxxxxxxxxxx'
GH_OAUTH='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
curl -i  https://api.github.com:443/orgs/xxxxxxxxxxxxxxxxxxxxxxxxxxx/repos?access_token=$GH_OAUTH
for RepoLine in `curl --silent https://api.github.com:443/orgs/xxxxxxxxxxxxxxxxxxxxx/repos?access_token=$GH_OAUTH |
  jsawk -n 'out(this.name)'`
do
  RepoName=$(echo $RepoLine | awk -F" : " '{ print $1 }')
   curl -i https://api.github.com/repos/xxxxxxxxxxxxxxxxxxxxxxxxx/$RepoName/teams?access_token=$GH_OAUTH | 
  jsawk -n 'out(this.login)' > output.txt
done

Last edited by ChocoTaco; 08-27-2014 at 01:32 PM.. Reason: Edited code block, wrong paste
# 6  
Old 08-27-2014
Can you post a sample of what your first curl downloads and also what your first jsawk produces with this as input?

There is a second do statement in your post, that I think should not be there..
# 7  
Old 08-27-2014
Scrutinizer, here are the files requested: The first shows the API curl response, the second outputs the jsawk stuff from mem, had to change the code a little to get the jsawk out, here's what i did --->

sorry not that good with awk / jsawk still learning:

Code:
#!/bin/bash
FS_bak=$IFS
IFS=$'\r\n'
uid='XXXXXXXXXXXXXXXXXX'
GH_OAUTH='XXXXXXXXXXXXXXXXXXXXXXXXX'
curl --silent  https://api.github.com:443/orgs/XXXXXXXXXXXXX/repos?access_token=$GH_OAUTH > output.txt
cat output.txt | jsawk -n 'out(this.name)' | jsawk 'return this.name' | > outputjsawk.txt

APIoutputScrub.txt (49.1 KB) 
outputjsawk.txt (454 Bytes) 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Parse property from json file

Hello All, Greetings.. I have a json file that I need to pase its URLs and other values. The match should start with "notifications" and output URLs and settings values. I tried with python or awk but hardly could get URLs only. Or whole URLs from full json file. Could not match... (2 Replies)
Discussion started by: 7adi
2 Replies

2. Shell Programming and Scripting

Building JSON command with bash script

Hello. I'm new to bash script and I'm learning the basics by writing some scripts. Recently a friend of mine asked me if I could try to write a script to him to automate a couple of processes that uses JSON RPCs. I'll try to explain in few words the workflow just to contextualize the problem.... (48 Replies)
Discussion started by: psysc0rpi0n
48 Replies

3. UNIX for Advanced & Expert Users

Running Powershell Script from Linux through Cygwin

Hello Experts, I am creating a run time powershell script on Linux machine and copying that powershell script to Windows machine. To connect to windows through "ssh", I am using Cygwin tool. To make the connection password less I copied my public in authorized_keys in windows Administrator... (5 Replies)
Discussion started by: shekhar_4_u
5 Replies

4. Shell Programming and Scripting

Parsing and Editing a json file with bash script

I am trying to automate editing of a json file using bash script. The file I initially receive is { "appMap": { "URL1": { "name": "a" }, "URL2": { "name": "b" }, "URL3": { "name": "c" }, } WHat I would like to do is replace... (5 Replies)
Discussion started by: Junaid Subhani
5 Replies

5. Shell Programming and Scripting

Bash Script to pull ipa server name on 500 servers

Hello All, I need help writing a bash script that will run on 500 LINUX servers and do the following: 1. Capture the ipa_server name from /etc/sssd/sssd.conf on a list of 500 servers in the ipahosts file. 2. Write to a file outputing only server name and IPA server name. Root ssh keys... (3 Replies)
Discussion started by: vtowntechy
3 Replies

6. Shell Programming and Scripting

How to define a variable in a BASH script by using a JSON file online?

Hello, I would like to modify an existing script of mine that uses a manually defined "MCVERSION" variable and make it define that variable instead based on this JSON file stored online: https://s3.amazonaws.com/Minecraft.Download/versions/versions.json Within that JSON, I 'm looking for... (4 Replies)
Discussion started by: nbsparks
4 Replies

7. Shell Programming and Scripting

How to start powershell with shebang from windows/cygwin/bash?

I would like to the the windws8/cygwin/bash shebang feature to start a powershell script. I do a "chmod +x set-sound.ps1" and then at a bash prompt I do ./set-sound.ps1 The first line of ./set-sound.ps1 #!powershell.exe -ExecutionPolicy unrestricted The result is the result: ... (5 Replies)
Discussion started by: siegfried
5 Replies

8. UNIX for Dummies Questions & Answers

GitHub documentation :: GitHub for dummies

Hi I wish to "develop" for brew. It is hosted on GitHub. What book (or online documentation) do you recommend me for GitHub and/or Git? Prefer small physical book. (1 Reply)
Discussion started by: slashdotweenie
1 Replies

9. Shell Programming and Scripting

shell / bash / script api ?

Hi is there a good dokumentation for shell scripting ? like the api in java ? didnt find a good one yet (5 Replies)
Discussion started by: Turrican
5 Replies

10. UNIX for Dummies Questions & Answers

How to use cygwin to run bash script

Hi, all, I try to run a quite simple bash script mytest.sh in cygwin, it's content is: #!/bin/bash echo "It is my first bash shell" there are three lines in the script. The second line is blank line. When I run it use command: bash c:/mytest.sh, ... (6 Replies)
Discussion started by: Jenny.palmy
6 Replies
Login or Register to Ask a Question