Building JSON command with bash script


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Building JSON command with bash script
# 22  
Old 09-22-2019
Ok, I'll read about eval more. I just know one of its usages which I think it is to append "parameters" to build a command. When I say this, it may also mean to add/append whatever you want to add/append to whatever you want. Append a string to another string, etc.
But this might be a very inaccurate description. Or bad wording choice!

Anyways, I'll also try to build the versions of your functions to create non-ordered JSON lists (["item_1", "item_2", "item_n"].

Thanks
Psy
# 23  
Old 09-24-2019
A short comment on debugging:

To check if a function does what it should do, the easiest method is to simply put the function in a small extra program, call it with some valid test data and just print the result to terminal and check if it matches the expectations, or generally speaking, check if the function succesfully executed the expected actions causing the expected results.
This User Gave Thanks to stomp For This Post:
# 24  
Old 09-24-2019
Quote:
Originally Posted by stomp
A short comment on debugging:

To check if a function does what it should do, the easiest method is to simply put the function in a small extra program, call it with some valid test data and just print the result to terminal and check if it matches the expectations, or generally speaking, check if the function succesfully executed the expected actions causing the expected results.
Yeah, I've been trying to do something like that sometimes. I also printf and echo many things to try to see where results go wrong.

I think I made it with the json list. I used your 2 functions and change them to try to build my json list in the following format: ["key_1", "key_2", ... , "key_n"]
Tell me if this can be improved.

Code:
mk_json_lst(){
   items=""
   for(( i = 1; i <= ${#addr_arr[@]}; i++))
   do
     [ -z "$items" ] || items+=","
     items+="$(eval echo \'\"\'${addr_arr[$i-1]}\'\"\')"
     echo "items is $items"
   done
   echo "[$items]"
 }

Code:
mk_json_lst_one_val(){
   local args=""
   for lst in "$@"
   do
     args+="$lst "
   done
   mk_json_lst $args
 }

--- Post updated 09-25-19 at 12:19 AM ---

Ok, building the json object and json list is ok but still bitcoin-cli is not accepting the $val value for some reason I cannot figure out!

I'm evaluating this variable in this line of code:
Code:
val=$(printf "%.9f" $(echo $(bitcoin-cli -testnet getbalance) / "$num_addr" | bc -l))

I did it this way because if I used a simpler way I would get a value in the format of:
Quote:
.00751256000000
and not in the format of

Quote:
0.00751256
which is the normal value I see in bitcoin-cli docs. So I used the first approach to try to convert the format of that number to the format that looked like what is in bitcoin core docs. And probably I might have a problem there. No idea how to fix it or what else can I try.
When I try to run the command with the json obj and the json list built with the script like:

Code:
bitcoin-cli -testnet sendmany "" {$pairs} 6 Payments [$items] true 6 CONSERVATIVE

I get the following error from bitcoin core:
Quote:
error code: -3
error message:
Invalid amount
Any ideas would be appreciated as I'm out of knowledge and expertise!
Thanks
Psy
# 25  
Old 09-25-2019
Hello Psy,

I suggest for now to only check if the JSON you want to generate is ok and only after having this 100% correct then go on to build the complete program call.

Regarding the json_object, this should be currently ok:

Example test script:

Code:
mk_json_object() {

        local pairs=""
        for((i=1;i<=$#;i+=2)) ; do
                [ -z "$pairs" ]  || pairs+=","
                pairs+="$(eval echo \'\"\'\$$i\'\"\': \'\"\'\$$(($i+1))\'\"\')"
        done
        echo "{ $pairs }"
}

mk_json_object_one_val() {

        local args=""
        local val="$1"
        shift
        for key in "$@"; do
                args+="$key $val "
        done
        mk_json_object $args # this variable should stay unquoted
}

mk_json_object_one_val myvalue mykey1 mykey2 mykey3

expected output:

Code:
{ "mykey1": "myvalue","mykey2": "myvalue","mykey3": "myvalue" }

You should have the same for the list, to check it's really working as expected.

About your command line:

Code:
bitcoin-cli -testnet sendmany "" {$pairs} 6 Payments [$items] true 6 CONSERVATIVE

If that is the final command line before calling the command, it is wrong, because {$pairs} is not a valid JSON object and [$items] is not a valid JSON Array. If this is not the final command, but the command that is to be executed(and evaluated) it is also most likely wrong, because {$pairs} and [$items] are not quoted and will result in wrong data after shell evaluation. Use "{$pairs}" and "[$items]". Aside from that the so far introduced functions already provide brackets. You probably have double brackets, which is not valid JSON.

Please look at the command and show it like it is executed, like this:

Code:
set -x
bitcoin-cli...

So you get the output from bash on the console how the final command really looks like.

Example:

Code:
object="$(mk_json_object_one_val myvalue mykey1 mykey2 mykey3)"
set -x
bitcoin-cli "$object" "par2 "par3"
set +x

Output:

Code:
[stomp@tu02c7dl log]$ ./testscript

+ bitcoin-cli '{ "mykey1": "myvalue","mykey2": "myvalue","mykey3": "myvalue" }' par2 par3
+ set +x

You see in the output the single quotes where the shell set the parameter boundary. The whole object is one parameter, which is exactly as desired.

Last edited by stomp; 09-25-2019 at 04:41 AM..
# 26  
Old 09-25-2019
Ok, I'll do that later when I get home.

I'll evaluate the list and array as suggested using 'jq' and will post results here!
Then I'll proceed to the 'set -x' thing.

Thanks.
Psy
# 27  
Old 09-25-2019
Had a go at simplifying Stomps two functions above. My goal was to get rid of eval, which I achieved for mk_json_object but couldn't achieve this for mk_json_object_one_val:

Code:
mk_json_object(){
   local items rest

   printf -v items '"%s": "%s"' "$1" "$2"
   shift 2
   [ $# -gt 1 ] && printf -v rest ',"%s": "%s"' "$@"
   echo "{ $items$rest }"
}

mk_json_object_one_val() {
   local val="$1"
   shift
   eval set -- "$(printf "\"$val\" \"%s\" " "$@")"
   mk_json_object "$@"
}


Last edited by Chubler_XL; 09-25-2019 at 06:43 PM..
These 2 Users Gave Thanks to Chubler_XL For This Post:
# 28  
Old 09-25-2019
I'm home now and almost going bed. That's my life. Today free time is about 1h...

I have just tested the json object and array created by the functions using 'jq' tool. I think this is a pretty fair way of testing if json objects are correctly built!
So, I just pipped the 'echo' command at the end of stopms both functions through "jq '.'" like this:

Code:
echo "{$pairs}" | jq '.'

Result is:
Code:
{
  "n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC": "0.079319635",
  "2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ": "0.079319635"
}

Which I think it's perfect.

and
Code:
echo "[$items]" | jq '.'

result is:
Code:
[
  "n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC",
  "2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ"
]

which I think its also perfect.

Now, according to the examples from bitcoin core:
Code:
> bitcoin-cli sendmany "" "{\"1D1ZrZNe3JUo7ZycKEYQQiQAWd9y54F4XX\":0.01,\"1353tsE8YMTA4EuV7dgUXGjNFf9KpVvKHz\":0.02}" 1 "" "[\"1D1ZrZNe3JUo7ZycKEYQQiQAWd9y54F4XX\",\"1353tsE8YMTA4EuV7dgUXGjNFf9KpVvKHz\"]"

So the functions outputs are not matching exactly the docs example. The thing is that the docs example wouldn't also pass in a json checker tool like jsonlint. This said,I think the backslashes escaping the inner double quotes of each address can be removed.
Example:
Code:
> bitcoin-cli sendmany "" "{"1D1ZrZNe3JUo7ZycKEYQQiQAWd9y54F4XX":0.01,"1353tsE8YMTA4EuV7dgUXGjNFf9KpVvKHz":0.02}" 1 "" "["1D1ZrZNe3JUo7ZycKEYQQiQAWd9y54F4XX","1353tsE8YMTA4EuV7dgUXGjNFf9KpVvKHz"]"

Now, there is a question that might be asked at this point. Will these backslashes be handled by bitcoin core and removed later to pass the command to the RPC server? Are they really needed? I would take a chance in saying they are not needed.
The same for the double quotes outside the {} and the []. But I think I can try these to some extent. I can try to issue the command manually in terminal and see if bitcoin core complains or it it runs the command smoothly.

The other thing I think I need to change is the value of each key because as they are numbers (floating point) they probably don't need to be wrapped around with double quotes. At least the example from the docs have no double quotes around the key values. So the command will become:
Code:
> bitcoin-cli sendmany "" {"1D1ZrZNe3JUo7ZycKEYQQiQAWd9y54F4XX":0.01,"1353tsE8YMTA4EuV7dgUXGjNFf9KpVvKHz":0.02} 1 "" ["1D1ZrZNe3JUo7ZycKEYQQiQAWd9y54F4XX","1353tsE8YMTA4EuV7dgUXGjNFf9KpVvKHz"]

I think this might be enough to bitcoin core run the command smoothly!

Summarizing, regarding the validity of json objects built by the functions, I think they are perfect! However, I'll test the command with the changes I said. No backslashes, no double quotes around key values and no outer double quotes wrapping {} and [] around.

In a couple of minutes I'll update this post!

About the improved stomp functions, I'll leave them for later after I have this actual version running and for after figuring out all tricks to make the command to work.


Update:
ok, so I tested 2 versions of the command directly in terminal and I typed in the command myself by hand (no scripts)

Attempt 1 - no backslashes and no outter double quotes wrapping {} and [] around and no double quotes around key values(the floating point numbers):
Code:
bitcoin-cli -testnet sendmany "" {"n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC":0.079319635, "2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ":0.079319635} 6 "Payments" ["n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC", "2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ"] true 6 CONSERVATIVE

Result:
Code:
error: Error parsing JSON:{n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC:0.079319635,

Attempt 2 - re-added back the outter double quotes wrapping {} and []. Still no backslashes and no double quotes wrapping around key values
Code:
bitcoin-cli -testnet sendmany "" "{"n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC":0.079319635, "2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ":0.079319635}" 6 "Payments" "["n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC", "2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ"]" true 6 CONSERVATIVE

Result:
Code:
error: Error parsing JSON:{n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC:0.079319635, 2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ:0.079319635}

Attempt 3 - readded backslashes because it's clear that they are needed.

Code:
bitcoin-cli -testnet sendmany "" "{\"n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC\":"0.079319635", \"2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ\":"0.079319635"}" 6 "Payments" "[\"n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC\", \"2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ\"]" true 6 CONSERVATIVE

Result:
Code:
error code: -3
error message:
Invalid amount

Ok, but no problem here because I foun out the problem... I'm just dumb and stupid. That's all. Bitcoin lowest decimal value is 1x10^(-8). Meaning 8 decimal places and in my code I was using "%.9f". Damn stupid of me!

I have already tried the command from the script and yes, I get an error but it's not from the script, is an "insuficient funds error". But it's a bug in my code. It's a matter of math. I mean, let's say I have 0.08444443 BTC to send to 2 addresses. What I'm doing is to divide that amount by 2 which would be 0.042222215. But this value is 9 decimal places long. So I format it to 8 decimal places and those to last decimal places becomes one by rounding up the 15 to 2 (or 20 if we consider 9 decimal places). This in turn generates insufficient amount error because 0.042222 * 2 is more than the total I had originally to send. So I have here a rounding problem I need to fix.

Last edited by psysc0rpi0n; 09-25-2019 at 07:49 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to convert any shell command output to JSON format?

Hi All, I am new to shell scripting, Need your help in creating a shell script which converts any unix command output to JSON format output. example: sample df -h command ouput : Filesystem size used avail capacity Mounted /dev/dsk/c1t0d0s0 8.1G 4.0G 4.0G 50% /... (13 Replies)
Discussion started by: balu1234
13 Replies

2. Shell Programming and Scripting

Fun with terminal plotting JSON data at the command line

One of the great thing about unix is the ability to pipe multiple programs together to manipulate data. Plain, unstructured text is the most common type of data that is passed between programs, but these days JSON is becoming more popular. I thought it would be fun to pipe together some command... (1 Reply)
Discussion started by: kbrazil
1 Replies

3. Shell Programming and Scripting

JSON structure to table form in awk, bash

Hello guys, I want to parse a JSON file in order to get the data in a table form. My JSON file is like this: { "document":{ "page": }, { "column": } ] }, { ... (6 Replies)
Discussion started by: Gescad
6 Replies

4. UNIX for Beginners Questions & Answers

Json field grap via shell script/awk

i have a json data that looks like this: { "ip": "16.66.35.10", "hostname": "No Hostname", "city": "Stepney", "region": "England", "country": "GB", "loc": "51.57,-0.0333", "org": "AS6871 British Telecommunications PLC", "postal": "E1" } im looking for a way to assign... (9 Replies)
Discussion started by: SkySmart
9 Replies

5. Shell Programming and Scripting

Parsing and Editing a json file with bash script

I am trying to automate editing of a json file using bash script. The file I initially receive is { "appMap": { "URL1": { "name": "a" }, "URL2": { "name": "b" }, "URL3": { "name": "c" }, } WHat I would like to do is replace... (5 Replies)
Discussion started by: Junaid Subhani
5 Replies

6. Shell Programming and Scripting

UNIX or Perl script to convert JSON to CSV

Is there a Unix or Perl script that converts JSON files to CSV or tab delimited format? We are running AIX 6.1. Thanks in advance! (1 Reply)
Discussion started by: warpmail
1 Replies

7. Shell Programming and Scripting

Bash script - cygwin (powershell?) pull from GitHub API Parse JSON

All, Have a weird issue where i need to generate a report from GitHub monthly detailing user accounts and the last time they logged in. I'm using a windows box to do this (work issued) and would like to know if anyone has any experience scripting for GitAPI using windows / cygwin / powershell?... (9 Replies)
Discussion started by: ChocoTaco
9 Replies

8. Shell Programming and Scripting

How to define a variable in a BASH script by using a JSON file online?

Hello, I would like to modify an existing script of mine that uses a manually defined "MCVERSION" variable and make it define that variable instead based on this JSON file stored online: https://s3.amazonaws.com/Minecraft.Download/versions/versions.json Within that JSON, I 'm looking for... (4 Replies)
Discussion started by: nbsparks
4 Replies

9. Shell Programming and Scripting

BASH SCRIPT of LS command

I need help in writing a BASH SCRIPT of ls command. for example: $ ./do_ls.sh files f1.txt f2.jpeg f3.doc $ ./do_ls.sh dirs folder1 folder2 folder3 My attempt: #!/bin/bash # if test $# -d file then echo $dirs else (3 Replies)
Discussion started by: above8k
3 Replies

10. Shell Programming and Scripting

Building command line parameters of arbitrary length

I couldn't find an existing thread that addressed this question, so hopefully this isn't redundant with anything previously posted. Here goes: I am writing a C-Shell script that runs a program that takes an arbitrary number of parameters: myprog -a file1 \ -b file2 \ -c file3 ... \ -n... (2 Replies)
Discussion started by: cmcnorgan
2 Replies
Login or Register to Ask a Question