Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Building JSON command with bash script

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Building JSON command with bash script
# 8  
Old 09-18-2019
Quote:
Originally Posted by stomp
Notes:
  • maybe you can spare the escaping of " if you mix ' (single quotes) and " (double quotes).
  • simple shell scripting will suffice for what you want to achieve, jq seems not necessary(but hey! Read the docs maybe it's way simpler! I'm not an jq expert, I just use it occasionally)
  • if you want to know if jq accepts the input, in other words to validate json, just pipe your data into jq . like echo "$data" | jq .. jq will complain if the data is not well formed.
But I cannot do it this way because this "com_params" variable that I set to store the command parameters is seen as a single long string, and if I do it this way, the app that parses the command to JSON complains.

How do you say this can be done with a simple bash script without using 'jq' ? Please give me a hint because I'm not getting there. I'm around this for like 2 or 3 weeks and I can't find a way to make the command to be accepted by the daemon that parses the command!

Note:
This is the code I actually have:
(I can't post links yet)
Search for my nick in Github anf for "TugaRevoBTC" repository if you want to see the code!
# 9  
Old 09-18-2019
Ok, I might have gotten somewhere.

Check my github repository named TugaRevoBTC for the entire code.

Code:
send_many(){
  load_addr_data
  count=0
  backcptb='""'
  min_conf=6
  comm1="Periodic payments"
  replcbl=true
  conf_targ=6
  est_mde=CONSERVATIVE

  val=$(printf "%.9f" $(echo $(bitcoin-cli -testnet getbalance) / "$num_addr" | bc -l))
  printf "Value to send: %.8f BTC\n" "$val"
  while true
  do
    echo "Confirm with YES or cancel with NO (caps matter)"
    read -r -p '> ' opt
    case $opt in
      "YES")
            obj_init="\""
            json_obj='{'
            for i in "${addr_arr[@]}"
            do
              ((count++))
              json_obj+="\"$i\""
              json_obj+=":"
              if [ "$count" -lt "$num_addr" ]; then
                json_obj+="$val,"
              else
                json_obj+="$val}"
              fi
            done
            obj_end+='"'
            json_final=$(echo $json_obj | jq '.')
            printf "bitcoin-cli -testnet sendmany  %s %s%s%s %d %s\n" "$backcptb" "$obj_init" "$json_final" "$obj_end" $min_conf "$comm1"
            return 1
            ;;
      "NO") echo "Action cancelled!"
            return 2
            ;;
    esac
  done
}

I just hope bitcoin-cli accepts the command the way I did it here! I haven't tested it yet. Bed time for now!
Any suggestions/corrections are greatly appreciated.

Thanks
Psy

Edited;
I just noticed something I can improve, at least visually. I can just add the leading ' " ' and the trailing ' " ' statically in that printf instead of passing each one as parameters.
These 2 Users Gave Thanks to psysc0rpi0n For This Post:
Neo stomp
# 10  
Old 09-19-2019
@psysc0rpi0n Great! You did it! Your code is good.

One of my goal in programming is trying to create code which is readable well. So I encapsule code into functions, even at the cost (in bash) to make code less resource efficient. Here the function for the creation of a json object:

This function operates on the given parameters($1, $2, ...):

Code:
mk_json_object() {

        local pairs=""
        for((i=1;i<=$#;i+=2)) ; do
                [ -z "$pairs" ]  || pairs+=","
                pairs+="$(eval echo \'\"\'\$$i\'\"\': \'\"\'\$$(($i+1))\'\"\')"
        done
        echo "{ $pairs }"
}

use it like this:

Code:
mydata="$(mk_json_object "key1" "value1" "key2" "value2")"

Since it is bash it's a bit of a quoting and eval mess here.

Since you have a special object where all values are equal, I create another function, so the main call stays simple:

Code:
mk_json_object_one_val() {

        local args=""
        local val="$1"
        shift
        for key in "$@"; do
                args+="$key $val "
        done
        mk_json_object $args # this variable should stay unquoted
}

use it like this:

Code:
mydata="$(mk_json_object_one_val "value" "key1" "key2" "key3")"

mydata will contain: { "key1": "value","key2": "value","key3": "value" }

The quoting is different from yours, but escaping the " with \ is not necessary, if you use the mydata variable in quotes this way "$mydata", since the content of mydata will not be parsed by the shell again.

Note: The function can not cope with values containing whitespace characters.
Last edited by stomp; 09-19-2019 at 04:42 AM..
These 2 Users Gave Thanks to stomp For This Post:
Neo psysc0rpi0n
# 11  
Old 09-19-2019
Ok, I'll have to go through your code and adapt it to my specific case. Later when I'm home, I'll try it.

Thanks
PsySc0rpi0n
# 12  
Old 09-19-2019
Quote:
Originally Posted by stomp
@psysc0rpi0n Great! You did it! Your code is good.

One of my goal in programming is trying to create code which is readable well. So I encapsule code into functions, even at the cost (in bash) to make code less resource efficient. Here the function for the creation of a json object:

This function operates on the given parameters($1, $2, ...):

Code:
mk_json_object() {

        local pairs=""
        for((i=1;i<=$#;i+=2)) ; do
                [ -z "$pairs" ]  || pairs+=","
                pairs+="$(eval echo \'\"\'\$$i\'\"\': \'\"\'\$$(($i+1))\'\"\')"
        done
        echo "{ $pairs }"
}

use it like this:

Code:
mydata="$(mk_json_object "key1" "value1" "key2" "value2")"

Since it is bash it's a bit of a quoting and eval mess here.

Since you have a special object where all values are equal, I create another function, so the main call stays simple:

Code:
mk_json_object_one_val() {

        local args=""
        local val="$1"
        shift
        for key in "$@"; do
                args+="$key $val "
        done
        mk_json_object $args # this variable should stay unquoted
}

use it like this:

Code:
mydata="$(mk_json_object_one_val "value" "key1" "key2" "key3")"

mydata will contain: { "key1": "value","key2": "value","key3": "value" }

The quoting is different from yours, but escaping the " with \ is not necessary, if you use the mydata variable in quotes this way "$mydata", since the content of mydata will not be parsed by the shell again.

Note: The function can not cope with values containing whitespace characters.
Hello.

I'm trying to apply the code to my specific case, and I think I have the same problem.
How can I add as many item pairs as lines that are loaded from the file?
The examples you posted are like static, right? You need to write each item pairs by hand as needed, right?

I tried my version of the code, but somehow I think there might be escaping issues because bitcoin-cli is still finding "json parsing errors" or "wrong amount" (wrong value in item pair), wrong number of arguments, etc.

Code:
send_many(){
   load_addr_data
   count=0
   min_conf=6
   comm1="Periodic payments"
   replcbl=true
   conf_targ=6
   est_mde=CONSERVATIVE

   val=$(printf "%.9f" $(echo $(bitcoin-cli -testnet getbalance) / "$num_addr" | bc -l))
   printf "Value to send: %.8f BTC\n" "$val"
   while true
   do
     echo "Confirm with YES or cancel with NO (caps matter)"
     read -r -p '> ' opt
     case $opt in
       "YES")
             json_obj='{'
             for i in "${addr_arr[@]}"
             do
               ((count++))
               json_obj+="\"$i\""
               json_obj+=":"
               if [ "$count" -lt "$num_addr" ]; then
                 json_obj+="$val,"
               else
                 json_obj+="$val}"
               fi
             done

             count=0
             json_lst="["
             for i in "${addr_arr[@]}"
             do
               ((count++))
               if [ "$count" -lt "$num_addr" ]; then
                 json_lst+="\"$i\","
               else
                 json_lst+="\"$i\"]"
               fi
             done
             #json_obj_final=$(echo $json_obj | jq '.')
             #json_lst_final=$(echo $json_lst | jq '.')
             #printf "bitcoin-cli -testnet sendmany  %s \"%s\" %d \"%s\" \"%s\" %s %d %s\n" "$backcptb" "$json_obj" $min_conf "$comm1" "$json_lst" "true" $min_conf "$est_mde"
             bitcoin-cli -testnet sendmany "" "$json_obj_final" $min_conf "$comm1" "$json_lst_final" "$replcbl" $min_conf "$est_mde"
             return 1
             ;;
       "NO") echo "Action cancelled!"
             return 2
             ;;
     esac
   done
 }

I tried to change "", add escaping chars, replacing "" by ' ' but I just can't make it work. I need to make sure that each of the 8 parameters are seen as strings. Param1 one string, param 2 another string, param3 another string and so on! I would like to try to make my version to work and then I could try your version and adapt it to load and create as many item pairs as needed, dynamically!

Thanks
Psy

Edited;
I know you already told me I can avoi escaping double quotes. But as I'm still trying to memorize the special chars that must be escaped and the ones that need no escaping, the way to save command outputs, the way to run commands in a nested way, etc, it's still too many details to just keep in mind at the same time and in a short period of time, I'll be escaping them because as of now that's the way I'm more comfortable with it.

--- Post updated at 11:00 PM ---

I tried something else and things gets even more weird.

The help menu for the command "sendmany" is:

Quote:
bitcoin-cli help sendmany
sendmany "" {"address":amount} ( minconf "comment" ["address",...] replaceable conf_target "estimate_mode" )

Send multiple times. Amounts are double-precision floating point numbers.

Arguments:
1. dummy (string, required) Must be set to "" for backwards compatibility.
2. amounts (json object, required) A json object with addresses and amounts
{
"address": amount, (numeric or string, required) The bitcoin address is the key, the numeric amount (can be string) in BTC is the value
}
3. minconf (numeric, optional, default=1) Only use the balance confirmed at least this many times.
4. comment (string, optional) A comment
5. subtractfeefrom (json array, optional) A json array with addresses.
The fee will be equally deducted from the amount of each selected address.
Those recipients will receive less bitcoins than you enter in their corresponding amount field.
If no addresses are specified here, the sender pays the fee.
[
"address", (string) Subtract fee from this address
...
]
6. replaceable (boolean, optional, default=fallback to wallet's default) Allow this transaction to be replaced by a transaction with higher fees via BIP 125
7. conf_target (numeric, optional, default=fallback to wallet's default) Confirmation target (in blocks)
8. estimate_mode (string, optional, default=UNSET) The fee estimate mode, must be one of:
"UNSET"
"ECONOMICAL"
"CONSERVATIVE"

Result:
"txid" (string) The transaction id for the send. Only 1 transaction is created regardless of
the number of addresses.

and knowing that the bitcoin-cli command syntax is (from bitcoin core docs):
Quote:
Examples:

Send two amounts to two different addresses:
> bitcoin-cli sendmany "" "{\"1D1ZrZNe3JUo7ZycKEYQQiQAWd9y54F4XX\":0.01,\"1353tsE8YMTA4EuV7dgUXGjNFf9KpVvKHz\":0.02}"

Send two amounts to two different addresses setting the confirmation and comment:
> bitcoin-cli sendmany "" "{\"1D1ZrZNe3JUo7ZycKEYQQiQAWd9y54F4XX\":0.01,\"1353tsE8YMTA4EuV7dgUXGjNFf9KpVvKHz\":0.02}" 6 "testing"

Send two amounts to two different addresses, subtract fee from amount:
> bitcoin-cli sendmany "" "{\"1D1ZrZNe3JUo7ZycKEYQQiQAWd9y54F4XX\":0.01,\"1353tsE8YMTA4EuV7dgUXGjNFf9KpVvKHz\":0.02}" 1 "" "[\"1D1ZrZNe3JUo7ZycKEYQQiQAWd9y54F4XX\",\"1353tsE8YMTA4EuV7dgUXGjNFf9KpVvKHz\"]"

As a JSON-RPC call
> curl --user myusername --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "sendmany", "params": ["", {"1D1ZrZNe3JUo7ZycKEYQQiQAWd9y54F4XX":0.01,"1353tsE8YMTA4EuV7dgUXGjNFf9KpVvKHz":0.02}, 6, "testing"] }' -H 'content-type: text/plain;' http://127.0.0.1:8332/
I decided to try to set all 8 parameters by hand directly on terminal.
After playing a while with incluing/excluding '' and "" and \ I get to a point where the only way 'jq' validates the objects and non-ordered lists is as follows:

Code:
backcomp='""'
json_obj='{"n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC":0.07931964,"2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ":0.07931964}'
json_lst='["n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC", "2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ"]'
min_conf=6
comment='"Periodic payments"'
replc=true
est_mde=CONSERVATIVE

Then I used 'jq' to validate the object and list:

Code:
echo $json_obj | jq '.'
{
  "n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC": 0.07931964,
  "2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ": 0.07931964
}

[
  "n1yswZYByRv3okGDHs1oDaevq8gsDaYZzC",
  "2NFafKRugHFdYEib7xVfkT6bvtkUKK8oShZ"
]

This looks perfect, however, when I try to issue the command with all these parameters:

Code:
$ bitcoin-cli -testnet sendmany $backcomp $json_obj $min_conf $comment $json_lst $replc $min_conf $est_mde

Code:
error: Error parsing JSON:payments"$ bitcoin-cli -testnet sendmany $backcomp $json_obj $min_conf '$comment' $json_lst $replc $min_conf $est_mde

Looks like the command is broken on variable $comment.
But to be honest, and as I said I'm still grasping about the escapings, expansions and etc, so I either try all possible combinations of escaping and expansions to see when it works or you guys help me figuring this out!

Thanks
Psy
# 13  
Old 09-19-2019
Just use the function with the same var as in your code:

Code:
mk_json... "$var" "${addr_arr[@]}"

Quote:
'$comment'
That doesn't work. Single quotes do not execute variable substitution. You need double quotes for that.

(Did not read your full post)
# 14  
Old 09-20-2019
Quote:
Originally Posted by stomp
Just use the function with the same var as in your code:

Code:
mk_json... "$var" "${addr_arr[@]}"



That doesn't work. Single quotes do not execute variable substitution. You need double quotes for that.

(Did not read your full post)
You mean to add double quotes when I run the command? Because I have already added double quotes when I assigned the variable due to the existence of 2 separate words.

Won't "$comment" expand to ""Periodic payments""? Or should I remove the double quotes at the assignment of the comment variable?
Last edited by psysc0rpi0n; 09-20-2019 at 05:08 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to convert any shell command output to JSON format?

Hi All, I am new to shell scripting, Need your help in creating a shell script which converts any unix command output to JSON format output. example: sample df -h command ouput : Filesystem size used avail capacity Mounted /dev/dsk/c1t0d0s0 8.1G 4.0G 4.0G 50% /... (13 Replies)
Discussion started by: balu1234
13 Replies

2. Shell Programming and Scripting

Fun with terminal plotting JSON data at the command line

One of the great thing about unix is the ability to pipe multiple programs together to manipulate data. Plain, unstructured text is the most common type of data that is passed between programs, but these days JSON is becoming more popular. I thought it would be fun to pipe together some command... (1 Reply)
Discussion started by: kbrazil
1 Replies

3. Shell Programming and Scripting

JSON structure to table form in awk, bash

Hello guys, I want to parse a JSON file in order to get the data in a table form. My JSON file is like this: { "document":{ "page": }, { "column": } ] }, { ... (6 Replies)
Discussion started by: Gescad
6 Replies

4. UNIX for Beginners Questions & Answers

Json field grap via shell script/awk

i have a json data that looks like this: { "ip": "16.66.35.10", "hostname": "No Hostname", "city": "Stepney", "region": "England", "country": "GB", "loc": "51.57,-0.0333", "org": "AS6871 British Telecommunications PLC", "postal": "E1" } im looking for a way to assign... (9 Replies)
Discussion started by: SkySmart
9 Replies

5. Shell Programming and Scripting

Parsing and Editing a json file with bash script

I am trying to automate editing of a json file using bash script. The file I initially receive is { "appMap": { "URL1": { "name": "a" }, "URL2": { "name": "b" }, "URL3": { "name": "c" }, } WHat I would like to do is replace... (5 Replies)
Discussion started by: Junaid Subhani
5 Replies

6. Shell Programming and Scripting

UNIX or Perl script to convert JSON to CSV

Is there a Unix or Perl script that converts JSON files to CSV or tab delimited format? We are running AIX 6.1. Thanks in advance! (1 Reply)
Discussion started by: warpmail
1 Replies

7. Shell Programming and Scripting

Bash script - cygwin (powershell?) pull from GitHub API Parse JSON

All, Have a weird issue where i need to generate a report from GitHub monthly detailing user accounts and the last time they logged in. I'm using a windows box to do this (work issued) and would like to know if anyone has any experience scripting for GitAPI using windows / cygwin / powershell?... (9 Replies)
Discussion started by: ChocoTaco
9 Replies

8. Shell Programming and Scripting

How to define a variable in a BASH script by using a JSON file online?

Hello, I would like to modify an existing script of mine that uses a manually defined "MCVERSION" variable and make it define that variable instead based on this JSON file stored online: https://s3.amazonaws.com/Minecraft.Download/versions/versions.json Within that JSON, I 'm looking for... (4 Replies)
Discussion started by: nbsparks
4 Replies

9. Shell Programming and Scripting

BASH SCRIPT of LS command

I need help in writing a BASH SCRIPT of ls command. for example: $ ./do_ls.sh files f1.txt f2.jpeg f3.doc $ ./do_ls.sh dirs folder1 folder2 folder3 My attempt: #!/bin/bash # if test $# -d file then echo $dirs else (3 Replies)
Discussion started by: above8k
3 Replies

10. Shell Programming and Scripting

Building command line parameters of arbitrary length

I couldn't find an existing thread that addressed this question, so hopefully this isn't redundant with anything previously posted. Here goes: I am writing a C-Shell script that runs a program that takes an arbitrary number of parameters: myprog -a file1 \ -b file2 \ -c file3 ... \ -n... (2 Replies)
Discussion started by: cmcnorgan
2 Replies
Login or Register to Ask a Question