Building JSON command with bash script


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Building JSON command with bash script
# 29  
Old 09-25-2019
With some more mucking around, I now have a version without eval that also escapes the reserved JSON characters:

Code:
#!/bin/bash
mk_json_object(){
   local items rest

   printf -v items '  "%s": "%s"' $(json_quote "$1" "$2")
   shift 2
   [ $# -gt 1 ] && printf -v rest ',\n  "%s": "%s"' $(json_quote "$@")
   printf "{\n%s\n}\n" "$items$rest"
}

mk_json_lst(){
   local list rest
   printf -v list '  "%s"' $(json_quote "$1")
   shift
   [ $# -gt 0 ] && printf -v rest ',\n  "%s"' $(json_quote "$@")
   printf "[\n%s\n]\n" "$list$rest"
}

json_quote() {
    while [ $# -gt 0 ]
    do
        val=${1//\\/\\\\}
        val=${val//\"/\\\"}
        val=${val//$'\n'/\\\n}
        val=${val//$'\t'/\\\t}
        val=${val//$'\r'/\\\r}
        val=${val//$'\f'/\\\f}
        val=${val//$'\b'/\\\b}
        printf "%s" "$val"
        shift
        [ $# -gt 0 ] && printf " "
    done
}

mk_json_object_one_val() {
        local key=$1
        while [ $# -gt 1 ]
        do
            shift
            args=( "${args[@]}" "$key" "$1" )
        done
        mk_json_object "${args[@]}"
}

mk_json_object_one_val '$(uname)' "key
1" '"test"' more$'\r'
mk_json_lst test$'\b' key-2

output:
Code:
{
  "$(uname)": "key\n1",
  "$(uname)": "\"test\"",
  "$(uname)": "more\r"
}
[
  "test\b",
  "key-2"
]

Note there are some extra space and newline characters added to the output in mk_json_object() and mk_json_lst() that you many want to remove if you a building variables to send as paramters to another program.

--- Post updated at 09:50 AM ---

Quote:
Originally Posted by psysc0rpi0n
Now, according to the examples from bitcoin core:
Code:
> bitcoin-cli sendmany "" "{\"1D1ZrZNe3JUo7ZycKEYQQiQAWd9y54F4XX\":0.01,\"1353tsE8YMTA4EuV7dgUXGjNFf9KpVvKHz\":0.02}" 1 "" "[\"1D1ZrZNe3JUo7ZycKEYQQiQAWd9y54F4XX\",\"1353tsE8YMTA4EuV7dgUXGjNFf9KpVvKHz\"]"

So the functions outputs are not matching exactly the docs example. The thing is that the docs example wouldn't also pass in a json checker tool like jsonlint. This said,I think the backslashes escaping the inner double quotes of each address can be removed.

Now, there is a question that might be asked at this point. Will these backslashes be handled by bitcoin core and removed later to pass the command to the RPC server? Are they really needed? I would take a chance in saying they are not needed.
These escaped quotes are simply to protect them from the shell the actual argument values that will react the bitcoin-cli process will be

Code:
1:sendmany
2:
3:{"1D1ZrZNe3JUo7ZycKEYQQiQAWd9y54F4XX":0.01,"1353tsE8YMTA4EuV7dgUXGjNFf9KpVvKHz":0.02}
4:1
5:
6:["1D1ZrZNe3JUo7ZycKEYQQiQAWd9y54F4XX","1353tsE8YMTA4EuV7dgUXGjNFf9KpVvKHz"]


Last edited by Chubler_XL; 09-25-2019 at 08:00 PM..
This User Gave Thanks to Chubler_XL For This Post:
# 30  
Old 09-25-2019
But I still need to understand something else. But it's something related with the way bitcoin core processes the command.

Stomp functions don't add any backslashes and the script don't return any script related errors. The error returned is from bitcoin core which complains about insufficient funds. However, when I tried to run the commands above manually without the backslashes, bitcoin core complained about json parsing error. So, why running the command by hand with no backslashes, produces errors and running the command without backslashes using the script, don't produce any errors (related with the script code)????

Tomorrow I need to look into this new issue!

ok, enough for today. Time to sleep.

Thanks to both of you.

Last edited by psysc0rpi0n; 09-25-2019 at 08:09 PM..
# 31  
Old 09-25-2019
Quote:
Originally Posted by psysc0rpi0n
But I still need to understand something else. But it's something related with the way bitcoin core processes the command.

Stomp functions don't add any backslashes and the script don't return any script related errors. The error returned is from bitcoin core which complains about insufficient funds. However, when I tried to run the commands above manually without the backslashes, bitcoin core complained about json parsing error. So, why running the command by hand with no backslashes, produces errors and running the command without backslashes using the script, don't produce any errors (related with the script code)????

Tomorrow I need to look into this new issue!

ok, enough for today. Time to sleep.

Thanks to both of you.
The reason for this is that quote removal only occurs on the original input words, not the result of expansions. To demonstrate:

Code:
$ echo "Hello, world"
Hello, world
$ VAR='echo "Hello, world"'
$ $VAR
"Hello, world"

It's also worth noting that quote matching is done before any expansion, for example:
Code:
$ QT='"'
$ echo "$QT"
"
$ echo $QT
"

If you save this code a pparam(or make it a function in your script) you can use it to debug what the shell is doing to your parameters before they are passed on to scripts/functions:

Code:
for((i=1;i<=${#};i++))
do
   echo Param $i: ${!i}
done

Code:
$ VAR='"one" "two"'
$ ./pparam $VAR
Param 1: "one"
Param 2: "two"
$ ./pparam "$VAR"
Param 1: "one" "two"
$ ./pparam "one" "two"
Param 1: one
Param 2: two

These 3 Users Gave Thanks to Chubler_XL For This Post:
# 32  
Old 09-26-2019
Quote:
Attempt 3 - readded backslashes because it's clear that they are needed.
Code:
bitcoin-cli -testnet sendmany "" "{\"n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC\":"0.079319635", \"2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ\":"0.079319635"}" 6 "Payments" "[\"n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC\", \"2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ\"]" true 6 CONSERVATIVE

Result:
Code:
error code: -3
error message:
Invalid amount

I think that is also wrong quoting, because the amounts are not escaped(prepended with slashes). But this causes no errors, because numbers should not be quoted in json. Maybe it will cause an error if you fix the quoting. :-)
# 33  
Old 09-26-2019
Quote:
Originally Posted by stomp
Code:
bitcoin-cli -testnet sendmany "" "{\"n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC\":"0.079319635", \"2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ\":"0.079319635"}" 6 "Payments" "[\"n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC\", \"2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ\"]" true 6 CONSERVATIVE

Result:
Code:
error code: -3
error message:
Invalid amount

I think that is also wrong quoting, because the amounts are not escaped(prepended with slashes). But this causes no errors, because numbers should not be quoted in json. Maybe it will cause an error if you fix the quoting. :-)
In bitcoin core docs, says that the amount can be strings too.

Later after work, I'll look into this.
Thanks
Psy
# 34  
Old 09-26-2019
Quote:
Originally Posted by Chubler_XL

...

These escaped quotes are simply to protect them from the shell the actual argument values that will react the bitcoin-cli process will be
Ok, I'm going to try your examples. This is not trivial for a starter. I've just been reading but not practicing much, so this is not clear at first sight!

--- Post updated at 10:02 PM ---

Quote:
Originally Posted by Chubler_XL
The reason for this is that quote removal only occurs on the original input words, not the result of expansions. To demonstrate:

Code:
$ echo "Hello, world"
Hello, world
$ VAR='echo "Hello, world"'
$ $VAR
"Hello, world"

It's also worth noting that quote matching is done before any expansion, for example:
Code:
$ QT='"'
$ echo "$QT"
"
$ echo $QT
"

If you save this code a pparam(or make it a function in your script) you can use it to debug what the shell is doing to your parameters before they are passed on to scripts/functions:

Code:
for((i=1;i<=${#};i++))
do
   echo Param $i: ${!i}
done

Code:
$ VAR='"one" "two"'
$ ./pparam $VAR
Param 1: "one"
Param 2: "two"
$ ./pparam "$VAR"
Param 1: "one" "two"
$ ./pparam "one" "two"
Param 1: one
Param 2: two

I don't fully understand what you mean by:
Quote:
that quote removal only occurs on the original input words, not the result of expansions.
and
Quote:
It's also worth noting that quote matching is done before any expansion, for example:
.

I need to read about expansions!
# 35  
Old 09-26-2019
I finally reached my goal...

Some changes in the script in the meantime.

I share the link in github:
TugaRevoBTC/TugaRevoTestnet.sh at master . PsySc0rpi0n/TugaRevoBTC . GitHub

Edited;
But this is not over... Heheh
I'll be doing some changes and I'll be doing tutorials and asking questions here.

Last edited by psysc0rpi0n; 09-27-2019 at 05:00 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to convert any shell command output to JSON format?

Hi All, I am new to shell scripting, Need your help in creating a shell script which converts any unix command output to JSON format output. example: sample df -h command ouput : Filesystem size used avail capacity Mounted /dev/dsk/c1t0d0s0 8.1G 4.0G 4.0G 50% /... (13 Replies)
Discussion started by: balu1234
13 Replies

2. Shell Programming and Scripting

Fun with terminal plotting JSON data at the command line

One of the great thing about unix is the ability to pipe multiple programs together to manipulate data. Plain, unstructured text is the most common type of data that is passed between programs, but these days JSON is becoming more popular. I thought it would be fun to pipe together some command... (1 Reply)
Discussion started by: kbrazil
1 Replies

3. Shell Programming and Scripting

JSON structure to table form in awk, bash

Hello guys, I want to parse a JSON file in order to get the data in a table form. My JSON file is like this: { "document":{ "page": }, { "column": } ] }, { ... (6 Replies)
Discussion started by: Gescad
6 Replies

4. UNIX for Beginners Questions & Answers

Json field grap via shell script/awk

i have a json data that looks like this: { "ip": "16.66.35.10", "hostname": "No Hostname", "city": "Stepney", "region": "England", "country": "GB", "loc": "51.57,-0.0333", "org": "AS6871 British Telecommunications PLC", "postal": "E1" } im looking for a way to assign... (9 Replies)
Discussion started by: SkySmart
9 Replies

5. Shell Programming and Scripting

Parsing and Editing a json file with bash script

I am trying to automate editing of a json file using bash script. The file I initially receive is { "appMap": { "URL1": { "name": "a" }, "URL2": { "name": "b" }, "URL3": { "name": "c" }, } WHat I would like to do is replace... (5 Replies)
Discussion started by: Junaid Subhani
5 Replies

6. Shell Programming and Scripting

UNIX or Perl script to convert JSON to CSV

Is there a Unix or Perl script that converts JSON files to CSV or tab delimited format? We are running AIX 6.1. Thanks in advance! (1 Reply)
Discussion started by: warpmail
1 Replies

7. Shell Programming and Scripting

Bash script - cygwin (powershell?) pull from GitHub API Parse JSON

All, Have a weird issue where i need to generate a report from GitHub monthly detailing user accounts and the last time they logged in. I'm using a windows box to do this (work issued) and would like to know if anyone has any experience scripting for GitAPI using windows / cygwin / powershell?... (9 Replies)
Discussion started by: ChocoTaco
9 Replies

8. Shell Programming and Scripting

How to define a variable in a BASH script by using a JSON file online?

Hello, I would like to modify an existing script of mine that uses a manually defined "MCVERSION" variable and make it define that variable instead based on this JSON file stored online: https://s3.amazonaws.com/Minecraft.Download/versions/versions.json Within that JSON, I 'm looking for... (4 Replies)
Discussion started by: nbsparks
4 Replies

9. Shell Programming and Scripting

BASH SCRIPT of LS command

I need help in writing a BASH SCRIPT of ls command. for example: $ ./do_ls.sh files f1.txt f2.jpeg f3.doc $ ./do_ls.sh dirs folder1 folder2 folder3 My attempt: #!/bin/bash # if test $# -d file then echo $dirs else (3 Replies)
Discussion started by: above8k
3 Replies

10. Shell Programming and Scripting

Building command line parameters of arbitrary length

I couldn't find an existing thread that addressed this question, so hopefully this isn't redundant with anything previously posted. Here goes: I am writing a C-Shell script that runs a program that takes an arbitrary number of parameters: myprog -a file1 \ -b file2 \ -c file3 ... \ -n... (2 Replies)
Discussion started by: cmcnorgan
2 Replies
Login or Register to Ask a Question