Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

This function (decode64) runs on gawk but not on busybox awk

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting This function (decode64) runs on gawk but not on busybox awk
# 1  
Old 03-25-2015
This function (decode64) runs on gawk but not on busybox awk
Hello,

I'm trying to figure out a way to use a decode64 function in an embedded system who has few utilities, including busybox.

Right now have something like this (taken from "google base64-and-base85-encoding-awk-scripts" sorry, I'm not able to post urls yet)

Code:
_decode64()
{
    [ ! -t 0 ] && set -- "${@}" $(cat)
    [ -z "${1}" ] && return 1
    printf "%s\\n" "${@}" | awk 'function _decode64_awk() {
    while(getline) { for(i=1;i<=length($0);i++) {
        c=index(b64,substr($0,i,1)); if(c--) { for(b=0;b<6;b++) {
        o=o*2+int(c/32); c=(c*2)%64;
        if(++obc==8) { printf "%c",o; obc=0; o=0; }}}}}}
    BEGIN {b64="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
     _decode64_awk(); exit;}'
}

This works great with simple strings such as:

Code:
echo "aGVsbG8gd29ybGQhCg==" | _decode64
hello world!

But it doesn't in busybox with more complicated examples:

Code:
cat<<E=O=F | _decode64 | zcat | tar xvf -
H4sIAEdTE1UAA+3TW2+bMBQA4DznV5yhtDUlqECBdHItbd3arrt093unChuTeGHQ
glnSZvnvM+kSrZOmvayaJp3vAUvnpoMFeTKu06RzozwjjsP29AeR9/O5MIiCjh9E
XhhHfhibuB8MwqgD3s2udaWpdVIBdMRI5Uo0Kv9N3Z/y/6nTVIoylXFI7O6sC8ZH
uAWuBg8+wfo61FKD64LVm92ZW9AjItH2jzL3sg37JryorKRuqgL8RfasUoXOwFqr
T04Ka9n+DXhTX/ByCslkDBtZUwitygJWS5yaOLHhapGJuXFJhlLnqpAmCFlZEcV8
qnZZLouhHpGeZ1PlOMuOlmCqSOWU8Djs1w2vdWWq+qrv2zYFlRHhustZnHmU78aU
X59QsnIzcMz+RGxtB6ZLMCI2A3stDumqyExynJILxnbacav3FVa/pNAmPHO0j/nC
qrG1t394dAwzsyKz7u7du79/cPjg6OGjx0+Onz57/uLlq9dv3r57/yHhIpXZcKQ+
j/MvRXl2XtW6+TqZXlyaH2Q7jOLBzm1ny7o2+ZebpCCnStP5Rnfe
/ddfGkIIIYQQ QgghhBBCCCGEEEIIIYQQQuhv+w5cA6uTACgAAA==
E=O=F

The above code works great with gawk so I think it could have some portability issues, thoughts?

NOTE: Right now, I'm using a pure shell implementation but it's quite slow, so I'm trying to improve the speed by using awk but still keeping the portability
# 2  
Old 03-26-2015
Busybox awk seems to be unable to output null characters, this breaks your complex example.

You can find a bit about it in the second answer's comments here:
shell - How can I output null-terminated strings in Awk? - Stack Overflow
Last edited by Walter Misar; 03-26-2015 at 12:00 PM..
# 3  
Old 03-26-2015
what awk version do you have on a busybox?
awk --version
# 4  
Old 03-26-2015
Busybox uses its own implementation of awk, seemingly one that uses null-terminated strings internally (as opposed to length prefixed, which could handle null characters inside strings). I used busybox 1.20.2 for my tests.

BusyBox

---------- Post updated at 09:03 PM ---------- Previous update was at 06:37 PM ----------

Converting base64 to uuencode, then uudecoding the result works with busybox and might or might not be faster than doing it all in the shell.

Code:
_decode64()
{
  [ ! -t 0 ] && set -- "${@}" $(cat)
  [ -z "${1}" ] && return 1
  printf "%s\\n" "${@}" | sed 's/=//g' | (
    echo begin 644 -
    awk 'function _decode64_awk() {
      inp = 1;
      out = "";
      while (getline) {
        for(i=1; i<=length($0); i++) {
          c = substr(uu, index(b64,substr($0,i,1)), 1);
          inp++;
          out = (out c);
          if (inp==61) {
            print "M" out;
            inp = 1;
            out = "";
          }
        }
      }      
      if (length(out)) {
        printf "%c", (32+3*length(out)/4);
        print out;
      }
    }
    BEGIN {
      b64="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
      uu="`!\"#$%&'\''()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_"
      _decode64_awk();
      exit;
    }'
    echo \`
    echo end
  ) | uudecode
}

Alternatively one could do the character translation with tr, and just do the regrouping with awk.
Last edited by Walter Misar; 03-26-2015 at 05:12 PM..
These 2 Users Gave Thanks to Walter Misar For This Post:
chilicuil vgersh99
# 5  
Old 03-26-2015
Thanks Walter,

Your modified code works great with busybox v1.13.3 and way faster than my current shell implementation which I'm posting below in case someone is curious about.

Code:
_decode64_sh() {
    [ ! -t 0 ] && set -- "${@}" "$(cat)"
    _decode64_sh__b64='ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'
    _decode64_sh__n="0"; _decode64_sh__v="0"; _decode64_sh__args="${@}"
    while [ "${_decode64_sh__args}" ]; do #process char by char
        _decode64_sh__char="${_decode64_sh__args%${_decode64_sh__args#?}}"
        case "${_decode64_sh__char}" in
            '\n') continue ;;
             '=') _decode64_sh__v="$(($_decode64_sh__v << 6))" ;;
               *) _decode64_sh__char="${_decode64_sh__b64#*$_decode64_sh__char}"
                  _decode64_sh__char="$((${#_decode64_sh__b64}-${#_decode64_sh__char}))"
                  if [ "${_decode64_sh__char}" -eq "0" ]; then
                    [ X"${_decode64_sh__args}" = X"${_decode64_sh__char}" ] &&\
                        _decode64_sh__args='' || _decode64_sh__args="${_decode64_sh__args#?}"
                    continue
                  fi
                  _decode64_sh__v="$(($_decode64_sh__v << 6 | $_decode64_sh__char-1))"
        esac
        _decode64_sh__n="$(($_decode64_sh__n+1))"
        if [ "${_decode64_sh__n}" -eq "4" ]; then
            _decode64_sh__s="16"; while [ "${_decode64_sh__s}" -gt "-1" ]; do
                _decode64_sh__char="$(($_decode64_sh__v >> $_decode64_sh__s & 255))"
                printf \\$(($_decode64_sh__char/64*100+$_decode64_sh__char%64/8*10+$_decode64_sh__char%8))
                _decode64_sh__s="$(($_decode64_sh__s-8))"
            done
            _decode64_sh__v="0"; _decode64_sh__n="0"
        fi
        [ X"${_decode64_sh__args}" = X"${_decode64_sh__char}" ] &&\
            _decode64_sh__args='' || _decode64_sh__args="${_decode64_sh__args#?}"
    done
}

Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk runs and produces output but with error

When I run the awk below, I get an error message awk -v OFS='\t' '$(NF-1)=="Benign" || ($(NF-2) OFS $(NF-1))=="Likely Benign" {$(NF)=$(NF-2) OFS $(NF-1)} {print $0 }' input awk: cmd. line:1: (FILENAME=VUS FNR=8) fatal: attempt to access field -1 input Chr Start End Ref ... (6 Replies)
Discussion started by: cmccabe
6 Replies

2. Shell Programming and Scripting

Why does the gawk-function close return -1 ?

Hallo, why does the gawk-function close return -1 ? :confused: the file sortiert.txt was created and exists (with the sorted print-string). The source Code: BEGIN{ print "b\n\na\nx\n9" | "sort > sortiert.txt" error = close ("sort > sortiert.txt") print error if (error) {print... (10 Replies)
Discussion started by: wilmaed
10 Replies

3. Shell Programming and Scripting

awk runs but output is empty

The awk below runs, however the output file is 0 bytes. It is basically matching input files that are 21 - 259 records to a file of 11,137,660 records. Basically, what it does is use the input files of which there are 4 to search and match in a large 11,000,000 record file and output the... (4 Replies)
Discussion started by: cmccabe
4 Replies

4. UNIX for Dummies Questions & Answers

Doubts About awk, and Gawk

well i have some doubts about the use of this commands: my first doubt is to know if there is a way to execute a awk program from a file? (now i do copy paste, i copy the script of a notepad on the terminal and then i press enter, but i want to put this scripts in some folder and execute them)... (3 Replies)
Discussion started by: matius_88
3 Replies

5. Shell Programming and Scripting

ash busybox read command not working inside function....

I have a script that has to execute a read command in a function, this is in an ash busybox. The code is... trapcatch () { echo "Ctl-c Detected, what do you want to do?" echo "Please choose the number of one of the following options" echo "1. Jump past this Set" echo "2. Exit... (8 Replies)
Discussion started by: tesser
8 Replies

6. Shell Programming and Scripting

awk command in script gives error while same awk command at prompt runs fine: Why?

Hello all, Here is what my bash script does: sums number columns, saves the tot in new column, outputs if tot >= threshold val: > cat getnon0file.sh #!/bin/bash this="getnon0file.sh" USAGE=$this" InFile="xyz.38" Min="0.05" # awk '{sum=0; for(n=2; n<=NF; n++){sum+=$n};... (4 Replies)
Discussion started by: catalys
4 Replies

7. Shell Programming and Scripting

Removing \n within a record (awk/gawk)

I am using a solution that was provided by a member: awk '{s=$0;if(length(s) < 700){getline; s=s " " $0}printf("%s\n",s)}' This scans through a file and removes '\n' within a record but not the record delimiter. However, there are instances where there are MULTIPLE instances of '\n'... (10 Replies)
Discussion started by: CKT_newbie88
10 Replies

8. Shell Programming and Scripting

Substitution using awk/gawk

Hello, I have a file containing lines such as: (1 104 (16) (17) (18) (102))$ (1 105 (16) (17) (19:21) (102))$ I would like to extract the numbers, only by using awk (or gawk). I do not want to use "sed" as it is very slow. For now my solution consists in... (2 Replies)
Discussion started by: jolecanard
2 Replies

9. Shell Programming and Scripting

awk,gawk in bat file

Hi. I'm trying to convert bat file into shell script. Bat file invokes awk file in one section: c:\upg\exe\gawk -f c:\upg\awk\gen_sae.awk -v OP=C:\\upg\\lod\\... ...c:\upg\ref\saaxi.ref c:\upg\log\SAAEPWO.log c:\upg\ref\saaepref.log First of all I issued unix2dos command on that awk file.... (0 Replies)
Discussion started by: andrej
0 Replies
Login or Register to Ask a Question