Visit Our UNIX and Linux User Community


Building JSON command with bash script


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Building JSON command with bash script
# 1  
Old 09-17-2019
Building JSON command with bash script

Hello.
I'm new to bash script and I'm learning the basics by writing some scripts.
Recently a friend of mine asked me if I could try to write a script to him to automate a couple of processes that uses JSON RPCs.

I'll try to explain in few words the workflow just to contextualize the problem.
There is an app which runs as a deamon. This app is the one which parses the commands and sends them to the remote server. It also receives the response and then sends it to another more userfriendly app which is the app the user interacts with.

Something like this:
  • daemon starts
  • user-firendly app send RPC command to daemon
  • daemon parses it and sends to the remote server
  • daemon gets the response and parses it to user friendly format
  • user-friendly app prints the JSON object nicely formatted.

One of these commands takes up to 8 parameters and 2 of these 8 parameters are lists of items.
The format is as follows:
Code:
app-name option1 "param1" "{\"item1\":num1,\"item2\":num2, \"item_n\":num_n}" num3 "param2" "[\"item3\",\"item4\",\"item_n\"]" param4 num5 "param5"

The goal here is to make the items inside those two lists, dynamic, I mean, I load those items from a file, line by line, then num1, num2 and num_n are evaluated in the code and used there. Other params are kind of static.

I already have a version that builds this command correctly with all brackets and double quotes and all special chars in place but the problem is that in the way I did it, JSON parser sees the whole command as a single long string and cannot be that way.
What I did in the code I have, was simply append the static parts of the command before and after of dynamically add the list items with for loops. But as I said, I did it in a way that it's just a huge single string.

JSON RPC expects each parameter as separate strings.

The other factor adding some complexity is what I mentioned of adding items to the lists dynamically. As those items are loaded from a file, it all depends on how many items there are in the file!

Does anyone knows a way of doing this with bash script?
# 2  
Old 09-18-2019
Quote:
Originally Posted by psysc0rpi0n

Does anyone knows a way of doing this with bash script?
Yes, you can do this in bash, but most people working with JSON are serializing data to be displayed on the web and so, they generally do not program web-based applications using bash.

What I find at unix.com is a lot of people, especially during the start of the school year in September, tend to have these "odd problems" where they are required to use one scripting language or another to do a homework assignment and then they attempt to hide the fact it is homework by coming up with some odd reason to do something that, in the real world, is simply not done this way.

For example, I write a lot of code to process JSON and I do this either in Javascript or in PHP; because 99% of all the code I write is related to web projects; because most people (like me) use JSON APIs to retrieve JSON objects across the network and use those objects in an application, generally written for the web (PHP, Javascript, Python) or maybe a modern gaming engine (which will use C++, Javascript, or C#, for example).

When you ask, "is this possible" my thoughts are why do you ask this? Because just about everything in a computing environment is possible.

The trick is to use the right tools for the job based on the application, the skill sets of the "shop" you are working in, etc.

However, during this time of year, more often than not, the driving requirement we see from new members is the strict and narrow task in a homework problem.

Now, I'm not staying for sure you are posting a vailed homework assignment, but I can tell you that this time of year, every year, we see a lot of homework posts, often written like yours, in a kind of obscure post about "a friend wants me to do this or that" or "I must do this in bash, because only bash is permitted" ; and it turns out that 95% of the time, these are all some forms of classwork where the member is new and posts something in some "disguised post" to get answers to homework assignments.

So, having said that....

What are you actually doing? For what actual project?

Please provide the project details of what you are really doing.

Thanks.
This User Gave Thanks to Neo For This Post:
# 3  
Old 09-18-2019
Thank you for replying...

Well, in fact it looks alot like that. However I can assure it's not an homework. I have already graduated earlier this year in February. And I'm not someone that just finished school in the 20's. I'm a full time worker, parents a enthusiast in electronics and programming.

This was supposed to be a simple script to process payments in bitcoins but this last command is messing with my nerves. I'm trying to start a simple project if a friend of mine and I told him I could do this script to automate some tasks.

I would appreciate any help. I'll come around here later.

Thanks
Psy
These 2 Users Gave Thanks to psysc0rpi0n For This Post:
# 4  
Old 09-18-2019
For simple JSON tasks jq is a versatile tool for querying, manipulating and creating JSON within a Shell script. You'll find a lot of usage examples in the web and of course in the excellent jq documentation.

For anything more complex: Use a decent scripting language of your choice, which supports JSON-processing.

Last edited by stomp; 09-18-2019 at 05:10 AM..
These 3 Users Gave Thanks to stomp For This Post:
# 5  
Old 09-18-2019
Quote:
Originally Posted by psysc0rpi0n
I would appreciate any help. I'll come around here later.
Thanks for the reply. That's great!

I'm sure our resident expert on JSON in shell scripts, stomp can help you then, as long as you use his favorite jq tool (see above).
# 6  
Old 09-18-2019
Hello again.

Well, I'm already using it (since yesterday which was when I found it - jq I mean).

So, I'll be a little bit more accurate about the thing.
The app is to process bitcoin payments to several bitcoin addresses. These addresses are in a file. One address per line.
The code I have (I cannot share it right now because I'm at work and the code I have in Github is not updated with the one I have in my personal laptop) reads the file, line by line, counts how many lines it read, then checks how much BTC there is in a certain wallet and divides that value by the number of addresses loaded from the file. This way, the available BTC amount is equally sent to as many addresses that were loaded from the file.

The only thing missing in my script is building this long command.
I'll share the code I have at the moment, later when I get home.

But the way I built it was by preppending the leading special chars to a variable, then appending each address loaded from the file already wrapped in braces, backslash and double quotes. Something like:

Code:
load_addresses # this is a function
BTCsendVal=$(printf "%.8f" $(bitcoin-cli -getbalance / $numberAddr | bc -l)) # evaluate the amount of BTC to send to each loaded address
com_params="\"\" \"{"
count=0
for i in $addr_list
count++
do
    if [[ $count -eq $numberAddr ]]; then
         com_param+="\\\"$i\\\""
    else
        com_param+="\\\"$i\\\","
done
com_params+="$params_n"
com_params+="params_[n+1]"
#etc etc

........

I might have missed some details in this code. I can't remember all the details accuratelly of the code I have, but this way, the command parameters will be seen as a single very long string!

I have some questions before I can even start playing with 'jq'.

The first parameter/argument is a double quotes with nothing inside it. Docs says its for back compatibility. But is this first parameter also considered a JSON element/or whatever it can be called?

What about the other parameters that are not wrapped in {}s or []s ???? Are those also to be processed by 'jq'?
# 7  
Old 09-18-2019
Notes:
  • maybe you can spare the escaping of " if you mix ' (single quotes) and " (double quotes).
  • simple shell scripting will suffice for what you want to achieve, jq seems not necessary(but hey! Read the docs maybe it's way simpler! I'm not an jq expert, I just use it occasionally)
  • if you want to know if jq accepts the input, in other words to validate json, just pipe your data into jq . like echo "$data" | jq .. jq will complain if the data is not well formed.

Last edited by stomp; 09-18-2019 at 05:25 PM..
This User Gave Thanks to stomp For This Post:

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to convert any shell command output to JSON format?

Hi All, I am new to shell scripting, Need your help in creating a shell script which converts any unix command output to JSON format output. example: sample df -h command ouput : Filesystem size used avail capacity Mounted /dev/dsk/c1t0d0s0 8.1G 4.0G 4.0G 50% /... (13 Replies)
Discussion started by: balu1234
13 Replies

2. Shell Programming and Scripting

Fun with terminal plotting JSON data at the command line

One of the great thing about unix is the ability to pipe multiple programs together to manipulate data. Plain, unstructured text is the most common type of data that is passed between programs, but these days JSON is becoming more popular. I thought it would be fun to pipe together some command... (1 Reply)
Discussion started by: kbrazil
1 Replies

3. Shell Programming and Scripting

JSON structure to table form in awk, bash

Hello guys, I want to parse a JSON file in order to get the data in a table form. My JSON file is like this: { "document":{ "page": }, { "column": } ] }, { ... (6 Replies)
Discussion started by: Gescad
6 Replies

4. UNIX for Beginners Questions & Answers

Json field grap via shell script/awk

i have a json data that looks like this: { "ip": "16.66.35.10", "hostname": "No Hostname", "city": "Stepney", "region": "England", "country": "GB", "loc": "51.57,-0.0333", "org": "AS6871 British Telecommunications PLC", "postal": "E1" } im looking for a way to assign... (9 Replies)
Discussion started by: SkySmart
9 Replies

5. Shell Programming and Scripting

Parsing and Editing a json file with bash script

I am trying to automate editing of a json file using bash script. The file I initially receive is { "appMap": { "URL1": { "name": "a" }, "URL2": { "name": "b" }, "URL3": { "name": "c" }, } WHat I would like to do is replace... (5 Replies)
Discussion started by: Junaid Subhani
5 Replies

6. Shell Programming and Scripting

UNIX or Perl script to convert JSON to CSV

Is there a Unix or Perl script that converts JSON files to CSV or tab delimited format? We are running AIX 6.1. Thanks in advance! (1 Reply)
Discussion started by: warpmail
1 Replies

7. Shell Programming and Scripting

Bash script - cygwin (powershell?) pull from GitHub API Parse JSON

All, Have a weird issue where i need to generate a report from GitHub monthly detailing user accounts and the last time they logged in. I'm using a windows box to do this (work issued) and would like to know if anyone has any experience scripting for GitAPI using windows / cygwin / powershell?... (9 Replies)
Discussion started by: ChocoTaco
9 Replies

8. Shell Programming and Scripting

How to define a variable in a BASH script by using a JSON file online?

Hello, I would like to modify an existing script of mine that uses a manually defined "MCVERSION" variable and make it define that variable instead based on this JSON file stored online: https://s3.amazonaws.com/Minecraft.Download/versions/versions.json Within that JSON, I 'm looking for... (4 Replies)
Discussion started by: nbsparks
4 Replies

9. Shell Programming and Scripting

BASH SCRIPT of LS command

I need help in writing a BASH SCRIPT of ls command. for example: $ ./do_ls.sh files f1.txt f2.jpeg f3.doc $ ./do_ls.sh dirs folder1 folder2 folder3 My attempt: #!/bin/bash # if test $# -d file then echo $dirs else (3 Replies)
Discussion started by: above8k
3 Replies

10. Shell Programming and Scripting

Building command line parameters of arbitrary length

I couldn't find an existing thread that addressed this question, so hopefully this isn't redundant with anything previously posted. Here goes: I am writing a C-Shell script that runs a program that takes an arbitrary number of parameters: myprog -a file1 \ -b file2 \ -c file3 ... \ -n... (2 Replies)
Discussion started by: cmcnorgan
2 Replies

Featured Tech Videos