Login or Register to Ask a Question and Join Our Community


UNIX for Beginners Questions & Answers

String has * as the field delimiter and I need echo/awk to escape it, how?

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers String has * as the field delimiter and I need echo/awk to escape it, how?
# 1  
Old 01-06-2017
String has * as the field delimiter and I need echo/awk to escape it, how?
Hi,

I am trying to read an Oracle listener log file line by line and need to separate the lines into several fields. The field delimiter for the line happens to be an asterisk.

I have the script below to start with but when running it, the echo command is globbing it to include other information that I don't need.

Below is a sample run of the script z.ksh

Code:
$
$ ls -altr
total 16
drwxr-xr-x 3 oracle oinstall 4096 Jan  7 03:59 ..
-rw-r--r-- 1 oracle oinstall  243 Jan  7 04:03 x.out
-rwxr--r-- 1 oracle oinstall  586 Jan  7 04:12 z.ksh
drwxr-xr-x 2 oracle oinstall 4096 Jan  7 04:12 .
$ ./z.ksh
- Processing  --> 15-DEC-2016 10:19:24 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=testuser))(SERVER=DEDICATED)(SERVICE_NAME=test_app.x.y.z)) * (ADDRESS=(PROTOCOL=tcp)(HOST=60.11.22.123)(PORT=55440)) * establish * test_app.x.y.z * 12666
- timestamp = 15-DEC-2016 10:19:24 x.out z.ksh (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=testuser))(SERVER=DEDICATED)(SERVICE_NAME=test_app.x.y.z)) x.out z.ksh (ADDRESS=(PROTOCOL=tcp)(HOST=60.11.22.123)(PORT=55440)) x.out z.ksh establish x.out z.ksh test_app.x.y.z x.out z.ksh 12666
- connectstring =
- result =
- service =
- returncode =

$ cat z.ksh
#!/bin/ksh

LOG=x.out

while read line
do
   echo "- Processing  --> $line"
   timestamp=`echo $line | awk -F"[*]" '{ print $1 }'`
   connectstring=`echo $line | awk -F"[*]" '{ print $2 }'`
   result=`echo $line | awk -F"[*]" '{ print $3 }'`
   service=`echo $line | awk -F"[*]" '{ print $4 }'`
   returncode=`echo $line | awk -F"[*]" '{ print $5 }'`

   echo "- timestamp = $timestamp"
   echo "- connectstring = $connectstring"
   echo "- result = $result"
   echo "- service = $service"
   echo "- returncode = $returncode"

   echo
done < $LOG

###########
# THE END #
###########

I've also tried doing awk -F "\*" and that does not make any difference besides giving the warning awk: warning: escape sequence `\*' treated as plain `*'

I also need to somehow extract the line below to each respective fields, i.e. CONNECT_DATA, PROGRAM, USER, SERVER, SERVICE_NAME,HOST and PORT Smilie.

Code:
(CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=testuser))(SERVER=DEDICATED)(SERVICE_NAME=test_app.x.y.z)) * (ADDRESS=(PROTOCOL=tcp)(HOST=60.11.22.123)(PORT=55440))

Here's wishing Oracle could have provided something to parse their own log. Maybe there is a program/script/utility out there that can parse log files of any format?

I will have to somehow change the timestamp to YYYYMMDD. For the time being, I need to be able to get around the asterisk globbing to start with.

Can't install Splunk/logstash unfortunately.

Any advice much appreciated. Thanks in advance.
# 2  
Old 01-06-2017
-F'*' worked for gawk. What OS are you on?
try echo "$line"
Why do you need so many awk-s? Cannot you do it with just one?
Last edited by vgersh99; 01-06-2017 at 11:54 AM..
This User Gave Thanks to vgersh99 For This Post:
newbie_01
# 3  
Old 01-06-2017
You do not need to escape it in awk. With awk a single character field separator, that is not a space character, is not treated as a regular expression string, but as a literal character.

Quote:
An extended regular expression can be used to separate fields by assigning a string containing the expression to the built-in variable FS, either directly or as a consequence of using the -F sepstring option. The default value of the FS variable shall be a single <space>. The following describes FS behavior:

1. If FS is a null string, the behavior is unspecified.

2. If FS is a single character:
a. If FS is <space>, skip leading and trailing <blank> and <newline> characters; fields shall be delimited by sets of one or more <blank> or <newline> characters.

b. Otherwise, if FS is any other character c, fields shall be delimited by each single occurrence of c.
3. Otherwise, the string value of FS shall be considered to be an extended regular expression. Each occurrence of a sequence matching the extended regular expression shall delimit fields.
awk:regular expressions
Last edited by Scrutinizer; 01-06-2017 at 12:21 PM..
# 4  
Old 01-06-2017
When double quoting $line, the * chars will be preserved, and your awk scripts will work. Did you consider reading the variables immediately with bash ?

Code:
while IFS="*" read TS CS RS SV RC REST
  do    echo "- timestamp = $TS"
        echo "- connectstring = $CS"
        echo "- result = $RS"
        echo "- service = $SV"
        echo "- returncode = $RC"
  done <  $LOG
- timestamp = 15-DEC-2016 10:19:24 
- connectstring =  (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=testuser))(SERVER=DEDICATED)(SERVICE_NAME=test_app.x.y.z)) 
- result =  (ADDRESS=(PROTOCOL=tcp)(HOST=60.11.22.123)(PORT=55440)) 
- service =  establish 
- returncode =  test_app.x.y.z


EDIT: Can't you, BTW, make ORACLE use other delimiters?
Last edited by RudiC; 01-06-2017 at 12:06 PM..
This User Gave Thanks to RudiC For This Post:
newbie_01
# 5  
Old 01-06-2017
Hi,

Thanks for your reply.

The awk I have is actually gawk, see below. Didn't know that is the case.
OS is Red Hat Enterprise Linux Server release 5.11 (Tikanga)
The multple awk-s is 'coz I am trying to assign each field to a variable that I can further need to awk again Smilie. Not sure if I can just use one awk to assign them to multiple variables. Can I replace the multiple awks to just a single awk?

After extracting to the timestamp variable, I will be converting that to YYYYMMDD.

For the connectstring variable, I will need to further break that down to CONNECT_DATA, PROGRAM, USER, SERVER, SERVICE_NAME,HOST and PORT. Don't know how to do that yet. Trying to get around the asterisk problem for the time being.

Code:
 $ cat z.ksh
#!/bin/ksh

LOG=x.out

while read line
do
   echo "- Processing  --> $line"
   timestamp=`echo $line | awk -F"*" '{ print $1 }'`
   connectstring=`echo $line | awk -F"*" '{ print $2 }'`
   result=`echo $line | awk -F"*" '{ print $3 }'`
   service=`echo $line | awk -F"*" '{ print $4 }'`
   returncode=`echo $line | awk -F"*" '{ print $5 }'`

   echo "- timestamp = $timestamp"
   echo "- connectstring = $connectstring"
   echo "- result = $result"
   echo "- service = $service"
   echo "- returncode = $returncode"

   echo
done < $LOG

###########
# THE END #
###########

 $ ./z.ksh
- Processing  --> 15-DEC-2016 10:19:24 * (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=testuser))(SERVER=DEDICATED)(SERVICE_NAME=test_app.x.y.z)) * (ADDRESS=(PROTOCOL=tcp)(HOST=60.11.22.123)(PORT=55440)) * establish * test_app.x.y.z * 12666
- timestamp = 15-DEC-2016 10:19:24 x.out z.ksh (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=testuser))(SERVER=DEDICATED)(SERVICE_NAME=test_app.x.y.z)) x.out z.ksh (ADDRESS=(PROTOCOL=tcp)(HOST=60.11.22.123)(PORT=55440)) x.out z.ksh establish x.out z.ksh test_app.x.y.z x.out z.ksh 12666
- connectstring =
- result =
- service =
- returncode =

 $ which awk
/bin/awk
 $ ls -l /bin/*awk*
lrwxrwxrwx 1 root root      4 Feb 11  2013 /bin/awk -> gawk
-rwxr-xr-x 1 root root 338744 Jun 13  2012 /bin/gawk
-rwxr-xr-x 1 root root   3089 Jun 13  2012 /bin/igawk
-rwxr-xr-x 1 root root 338760 Jun 13  2012 /bin/pgawk
 $
$ awk --version
GNU Awk 3.1.5
Copyright (C) 1989, 1991-2005 Free Software Foundation.

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.

Trying awk -F"*" does work out fine from the command line as you mentioned. It is during the echo run that it is failing.

Code:
$ awk -F"*" '{ printf "%-20s %-40s %-15s %-10s %-10s \n", $1 , $2 , $3 , $4 , $5 }' x.out
15-DEC-2016 10:19:24   (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=testuser))(SERVER=DEDICATED)(SERVICE_NAME=test_app.x.y.z))   (ADDRESS=(PROTOCOL=tcp)(HOST=60.11.22.123)(PORT=55440))   establish   test_app.x.y.z

$ awk -F"*" '{ print $2, $3 }' x.out
 (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=testuser))(SERVER=DEDICATED)(SERVICE_NAME=test_app.x.y.z))   (ADDRESS=(PROTOCOL=tcp)(HOST=60.11.22.123)(PORT=55440))

$ export x=`head -1 x.out`
$ echo $x
15-DEC-2016 10:19:24 x.out z.ksh (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=testuser))(SERVER=DEDICATED)(SERVICE_NAME=test_app.x.y.z)) x.out z.ksh (ADDRESS=(PROTOCOL=tcp)(HOST=60.11.22.123)(PORT=55440)) x.out z.ksh establish x.out z.ksh test_app.x.y.z x.out z.ksh 12666
$ echo $x | awk -F"*" '{ print $2 }'

$ echo $x | awk -F"*" '{ print $0 }'
15-DEC-2016 10:19:24 x.out z.ksh (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=testuser))(SERVER=DEDICATED)(SERVICE_NAME=test_app.x.y.z)) x.out z.ksh (ADDRESS=(PROTOCOL=tcp)(HOST=60.11.22.123)(PORT=55440)) x.out z.ksh establish x.out z.ksh test_app.x.y.z x.out z.ksh 12666
$ echo $x | awk -F"*" '{ print $1 }'
15-DEC-2016 10:19:24 x.out z.ksh (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=testuser))(SERVER=DEDICATED)(SERVICE_NAME=test_app.x.y.z)) x.out z.ksh (ADDRESS=(PROTOCOL=tcp)(HOST=60.11.22.123)(PORT=55440)) x.out z.ksh establish x.out z.ksh test_app.x.y.z x.out z.ksh 12666
$ echo $x | awk -F"*" '{ print $3 }'

Can you actually use awk to wrap a field so that the string below ...

Code:
15-DEC-2016 10:19:24   (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=testuser))(SERVER=DEDICATED)(SERVICE_NAME=test_app.x.y.z))   (ADDRESS=(PROTOCOL=tcp)(HOST=60.11.22.123)(PORT=55440))   establish   test_app.x.y.z

Can be printed to be as below?

Code:
15-DEC-2016 10:19:24   (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)					establish   test_app.x.y.z
                       (USER=testuser))(SERVER=DEDICATED)(SERVICE_NAME=test_app.x.y.z))   
		       (ADDRESS=(PROTOCOL=tcp)(HOST=60.11.22.123)(PORT=55440))

---------- Post updated at 11:26 AM ---------- Previous update was at 11:20 AM ----------

Hi,

Unfortunately, can't get ORACLE to use a different delimiter.

---------- Post updated at 11:39 AM ---------- Previous update was at 11:26 AM ----------

Thanks a lot RudiC, I am using the code that you posted at the moment and that works just fine at the moment.

Now I need to break down the following strings further?

- connectstring = (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=testuser))(SERVER=DEDICATED)(SERVICE_NAME=test_app.x.y.z))
- result = (ADDRESS=(PROTOCOL=tcp)(HOST=60.11.22.123)(PORT=55440))

Can you actually use awk's printf to wrap a field/column so that for example if a field is 50 characters i want it to print to 2 lines of 25 characters each each of which are printed as the second field?

Quote:
Originally Posted by RudiC
When double quoting $line, the * chars will be preserved, and your awk scripts will work. Did you consider reading the variables immediately with bash ?

Code:
while IFS="*" read TS CS RS SV RC REST
  do    echo "- timestamp = $TS"
        echo "- connectstring = $CS"
        echo "- result = $RS"
        echo "- service = $SV"
        echo "- returncode = $RC"
  done <  $LOG
- timestamp = 15-DEC-2016 10:19:24 
- connectstring =  (CONNECT_DATA=(CID=(PROGRAM=JDBC Thin Client)(HOST=__jdbc__)(USER=testuser))(SERVER=DEDICATED)(SERVICE_NAME=test_app.x.y.z)) 
- result =  (ADDRESS=(PROTOCOL=tcp)(HOST=60.11.22.123)(PORT=55440)) 
- service =  establish 
- returncode =  test_app.x.y.z


EDIT: Can't you, BTW, make ORACLE use other delimiters?
# 6  
Old 01-06-2017
And, I guess, you don't want to break at 25 chars but at the nearest parenthesis?

I don't know of such a function in awk; you need to program it step by step yourself. Mayhap perl provides sth. alike?
# 7  
Old 01-06-2017
Something along this line?
Code:
awk -F\* '
        {while (length($2) > 0) {T[++C] = substr ($2, 1, 20)
                                 $2 = substr ($2, 21)
                                }
         printf "%20s\t%20s\t%s\t%s\n", $1, T[1], $3, $4
         for (i=2; i<=C; i++) printf "%20s\t%20s\n", "", T[i]
        }
' file
15-DEC-2016 10:19:24 	 (CONNECT_DATA=(CID=	 (ADDRESS=(PROTOCOL=tcp)(HOST=60.11.22.123)(PORT=55440)) 	 establish 
                    	(PROGRAM=JDBC Thin C
                    	lient)(HOST=__jdbc__
                    	)(USER=testuser))(SE
                    	RVER=DEDICATED)(SERV
                    	ICE_NAME=test_app.x.
                    	              y.z))

This User Gave Thanks to RudiC For This Post:
newbie_01
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

awk Associative Array and/or Referring to Field by String (Nonconstant String Value)

I will start with an example of what I'm trying to do and then describe how I am approaching the issue. File PS028,005 Lexeme HRS # M # PhraseType 1(1:1) 7(7) PhraseLab 501 503 ClauseType ZYq0 PS028,005 Lexeme W # L> # BNH # M #... (17 Replies)
Discussion started by: jvoot
17 Replies

2. Shell Programming and Scripting

How can awk ignore the field delimiter like comma inside a field?

We have a csv file as mentioned below and the requirement is to change the date format in file as mentioned below. Current file (file.csv) ---------------------- empname,date_of_join,dept,date_of_resignation ram,08/09/2015,sales,21/06/2016 "akash,sahu",08/10/2015,IT,21/07/2016 ... (6 Replies)
Discussion started by: gopal.biswal
6 Replies

3. Shell Programming and Scripting

awk :how to change delimiter without giving all field name

Hi Experts, i need to change delimiter from tab to "," sample test file cat test A0000368 A29938511 072569352 5 Any 2 for Ģ1.00 BUTCHERS|CAT FOOD|400G Sep 12 2012 12:00AM Jan 5 2014 11:59PM Sep 7 2012 12:00AM M 2.000 group 5 ... (2 Replies)
Discussion started by: Lakshman_Gupta
2 Replies

4. Shell Programming and Scripting

perform echo and awk inside a string

hi, just wanted to make a shortcut of this one a="a b c" b=`echo $a | awk '{print $2}'` echo "the middle is $b" why can't i do this: a="a b c" echo "the middle is ${`echo $a | awk '{print $2}'`}" <- bad substitution :wall: thanks (6 Replies)
Discussion started by: h0ujun
6 Replies

5. Shell Programming and Scripting

Awk Search text string in field, not all in field.

Hello, I am using awk to match text in a tab separated field and am able to do so when matching the exact word. My problem is that I would like to match any sequence of text in the tab-separated field without having to match it all. Any help will be appreciated. Please see the code below. awk... (3 Replies)
Discussion started by: rocket_dog
3 Replies

6. Shell Programming and Scripting

how to find the nth field value in delimiter file in unix using awk

Hi All, I wanted to find 200th field value in delimiter file using awk.? awk '{print $200}' inputfile I am getting error message :- awk: The field 200 must be in the range 0 to 199. The source line number is 1. The error context is {print >>> $200 <<< } using... (4 Replies)
Discussion started by: Jairaj
4 Replies

7. Shell Programming and Scripting

awk output field delimiter

Dear All, 1.txt (tab in between each value in a line) a b c a b c a c d you can see below, why with ~ i can output with tab, but = cannot? # awk -F'\t' '$2 ~ /b/' 1 a b c a b c # awk -F'\t' '$2 = "b"' 1 a b c a b c a b d ... (1 Reply)
Discussion started by: jimmy_y
1 Replies

8. Shell Programming and Scripting

need to convert a decimal value to an ascii char with awk for the field delimiter

Hello, I need an awk script to receive a variable that's an decimal value such as 009 or 031 and then convert this value to an ascii character to use as the FS (field separator for the input file). For example, 009 should be converted to an ascii tab 031 should be converted to an ascii... (1 Reply)
Discussion started by: script_op2a
1 Replies

9. UNIX for Advanced & Expert Users

Printing Field with Delimiter in AWK/cut

Hello, I had posted earlier about printing fields using AWK, but now I have a slightly different problem. I have text files in the format: 1*2,3,4,5 and wish to print the first, third, and fifth fields, including the asterisk and commas. In other words, after filtering it should look... (1 Reply)
Discussion started by: Jahn
1 Replies

10. Shell Programming and Scripting

Set a variable field delimiter using awk

How can i set a variable field delimiter using awk?? I wanna do something like this ,but i canīt get the correct syntaxis : VARI=TEST echo "0121212TESTxvcshaashd"|awk 'FS="$VARI" {print $2}' Thanks. (2 Replies)
Discussion started by: Klashxx
2 Replies
Login or Register to Ask a Question