Visit The New, Modern Unix Linux Community


AIX to RHEL migration - awk treating 0e[0-9]+ as 0 instead of string issue


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting AIX to RHEL migration - awk treating 0e[0-9]+ as 0 instead of string issue
# 1  
AIX to RHEL migration - awk treating 0e[0-9]+ as 0 instead of string issue

Greetings Experts,
We are migrating from AIX to RHEL Linux. I have created a script to verify and report the NULLs and SPACEs in the key columns and duplicates on key combination of "|" delimited set of big files. Following is the code that was successfully running in AIX.

Code:
awk -F "|" 'BEGIN { OFS="|" ; null_blank_flag="NO"; duplicate_flag="NO"} {
if (NR == 1) # Header record, get the key column names
  {if (substr(FILENAME,length(FILENAME)-6,3) == "abc") { a_key_cols[$1 OFS $2]=$1 "," $2 }
  {if (substr(FILENAME,length(FILENAME)-6,3) == "def") { a_key_cols[$1 OFS $2 OFS $3]=$1 "," $2 "," $3}
  {if (substr(FILENAME,length(FILENAME)-6,3) == "ghi") { a_key_cols[$1 OFS $2 OFS $3 OFS $4]=$1 "," $2 "," $3 "," $4}
}
if (NR >= 2)
{  if (substr(FILENAME,length(FILENAME)-6,3) == "abc") { a[$1 OFS $2]++ }
  {if (substr(FILENAME,length(FILENAME)-6,3) == "def") { a[$1 OFS $2 OFS $3]++ }
  {if (substr(FILENAME,length(FILENAME)-6,3) == "ghi") { a[$1 OFS $2 OFS $3 OFS $4]++}
}
}
END { for (i in a) { n=split(i,arry,OFS);
for (k=1;k<=n;k++) { 
gsub(" ","",arry[k]);
if ( (! arry[k]) && ( arry[k] != "0" )) { null_blank_flag="YES" } }
if ( a[i] >= 2 ) { duplicate_flag="YES" }
print "Filename " FILENAME " null/blank flag: " null_blank_flag " and duplicate flag: " duplicate_flag
} ' file_name_*.txt

I had to use the condition arry[k] != "0" because as AWK is treating 0 as NULL and it may be valid value in the file. Please note that the key values change depending on the file name and the key column has SHA_encrypted account numbers like 02djfdf93ikdkjdfkdf3 and 0e123458939393 etc. Please ignore any syntax issues in script as I am not able to copy/paste the working code as it in another machine and the number of characters in the encrypted account_number

When ran this script in Linux, the encrypted account numbers which follows the format 0e[0-9]+ is being interpreted as scientific notation as 0*10 power of [0-9]+ where as in AIX this is interpreted as string itself and doesn't set the null_blank_flag to yes.

Code:
**AIX**
echo "abc|0e123456789|xyz|1234|kdkd|dfs" | awk -F "|" '{ if ( ! $2 ) { print $2 " is empty or NULL " } else { print $2 " is not empty "}}'
output: 0e123456789 is not empty

**RHEL LINUX**
echo "abc|0e123456789|xyz|1234|kdkd|dfs" | awk -F "|" '{ if ( ! $2 ) { print $2 " is empty or NULL " } else { print $2 " is not empty "}}'
output: 0e123456789 is empty or NULL

grep may be a choice and I think I need to read the file twice to check for the null/blank and duplicates on the keys and hence resorted to awk.

As a[$1 OFS $2 OFS $3 OFS $4]++ array index is string for most of the data in the file, I had expected the array index that was built will also be string $1 OFS $2 OFS $3 OFS $4. Will it change from string to integers even though the first record file entry after header is string. Will the strings convert to integer during the split n=split(i,arry,OFS); ...; arry[k] The interesting part is the output from the Linux version is also 0e123456789 which confirms that $2 is 0e123456789. Now where is the issue occurring which transforms $2 value to 0

Can you please explain how to make awk/shell to interpret 0e[0-9]+ as string instead of number. Thank you for your time.
# 2  
Try a string conversion:
Code:
echo "abc|0e123456789|xyz|1234|kdkd|dfs" | awk -F "|" '{ if ( ! ($2 "")) { print $2 " is empty or NULL " } else { print $2 " is not empty "}}'
0e123456789 is not empty

But - no integer conversion is done for sheer assignments or usage in e.g. constucting an array index .
This User Gave Thanks to RudiC For This Post:
# 3  
Comparing with a string should work as well.
Code:
echo "abc|0e123456789|xyz|1234|kdkd|dfs" | awk -F "|" '{ if ($2=="") { print $2 " is empty or NULL " } else { print $2 " is not empty "}}'

This User Gave Thanks to MadeInGermany For This Post:
# 4  
Thank you RudiC and MadeInGermany. That works. Just wanted to know that during reading $2 is 0e123456789 but why does awk interpret $2 to 0 i.e., process $2 as scientific notation during ! $2 in earlier code. Can you please explain this behavior. Is this again specific to Linux as this was executing without issues in AIX. Thank you for your time.
# 5  
As I said - usage alone, be it for assignment or reference in an index, takes the value as is. Only evaluation, e.g. for a boolean expression or a numerical computation, converts the string to a number, using the starting characters up to a non-convertible one.
This User Gave Thanks to RudiC For This Post:
# 6  
awk has automatic type conversion.
The ! operator is a boolean that is rather a number (being 1 or 0) than a string, so there is a decent hint to treat the variable as a number.
In border cases there can be differences between awk versions. E.g. the awk in AIX is a derivate from nawk, where awk in Linux is mostly gawk, but some Linux have mawk that is again little different.
The bottom line is, make the data type clear, e.g. typical casts are
var "" append a null string, result is a string
var+0 add a zero, result is a number
This User Gave Thanks to MadeInGermany For This Post:

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #258
Difficulty: Medium
Iannis Xenakis wrote programs in the FORTRAN language that generated numeric data that he transcribed into scores to be played by traditional musical instruments.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk treating variables differently in UNIX-Linux

Hi, awk seem to be acting differently in Unix and Linux when it comes to formatting. This is making it difficult to migrate scripts. for example: UNIX: echo "123" |awk '{printf ("%05s\n" ,$1)}' 00123 echo "123" |awk '{printf ("%05d\n" ,$1)}' 00123 echo "S12" |awk '{printf ("%05s\n"... (9 Replies)
Discussion started by: wanderingmind16
9 Replies

2. AIX

AIX Migration issue with EMC ODM sets

Hi Experts , I want to start migrating our AIX 6.1 to AIX 7.1 . I am planning to use alt_disk_migration . Chris gibson has awesome documentation in the internet. However I am running into an issue with EMC odm filesets . So my current OS is AIX 6.1. and I have this : lslpp -l | grep EMC ... (7 Replies)
Discussion started by: JME2015
7 Replies

3. AIX

AIX - FC Switch migration, SAN Migration question!

I'm New to AIX / VIOS We're doing a FC switch cutover on an ibm device, connected via SAN. How do I tell if one path to my remote disk is lost? (aix lvm) How do I tell when my link is down on my HBA port? Appreciate your help, very much! (4 Replies)
Discussion started by: BG_JrAdmin
4 Replies

4. Shell Programming and Scripting

Search several string and convert into a single line for each search string using awk command AIX?.

I need to search the file using strings "Request Type" , " Request Method" , "Response Type" and by using result set find the xml tags and convert into a single line?. below are the scenarios. Cat test Nov 10, 2012 5:17:53 AM INFO: Request Type Line 1.... (5 Replies)
Discussion started by: laknar
5 Replies

5. Shell Programming and Scripting

awk - treating remaining columns as one

Hi all, For no particular reason, I would like to use awk on a file that contains multiple columns, but let's say only columns 1 and 2 have some text values, and the remainder of the line contains text that I would like to treat as one column, considering I have spaces as delimiter for the... (33 Replies)
Discussion started by: ppucci
33 Replies

6. Shell Programming and Scripting

Treating string as date ?

Is there a way to treat a string as date and compare it to the current date? lets assum inpu lik $ cat myfile Name Last login ************************** Sara 2/13/2012 kalpeer 2/15/2012 ygemici 2/14/2012 we want to display the name who logged in during the last #... (4 Replies)
Discussion started by: Sara_84
4 Replies

7. AIX

IY17981 fix required for aix 4.3.3 to aix 5L migration but not found

Hi, redbook documentation is telling that IY17981 fix is required for aix 4.3.3 to aix 5L migration. But there is no mention about that fix in any ML installation packages. - My system is ML11 : oslevel –r 4330-11 - But xlC.rte is on wrong version : lslpp -L xlC.rte xlC.rte ... (3 Replies)
Discussion started by: astjen
3 Replies

8. AIX

AIX 5.2 to 5.3 migration

Hello All, We want to upgrade our 44p Model 270 from AIX 5.2 to 5.3. This is a standalone devlopment server but downtime is something we don't want because we have a short development deadline looming. I have no tape drive to make backups to. I myself am a developer and don't have any... (4 Replies)
Discussion started by: Fred Vogel
4 Replies

9. Shell Programming and Scripting

awk issue on AIX

Hi, two teams at two locations - A & B. At location A, we have AIX 5.3 and at location B, we have AIX 5.1. We execute the below awk command in loc A and it executes successfully(part of a larger script). But the same does not get executed in the loc B server. We are not able to access the loc B... (1 Reply)
Discussion started by: ranj@chn
1 Replies

10. UNIX for Dummies Questions & Answers

AIX Migration

Hi , Migrating AIX 4.3.3 ML10 to 5.3 ML1 (retaining 32 bit) after successfully going through 5 of the 5.3 Install CD's. After "All fileset's processed successfully " message I was told that system would reboot and then I would get prompted for setting TERm type ( i have an ascii ibm3151) and... (1 Reply)
Discussion started by: Student37
1 Replies

Featured Tech Videos