Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Appending ErrorCodes to the corresponding error record

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Appending ErrorCodes to the corresponding error record
# 1  
Old 06-13-2014
Appending ErrorCodes to the corresponding error record
Hi,

Here i'm using a awk inside bash script to validate the datafile by referring to the configuration file(schema file).
Here the validation check is done for datatype, field length and null values.
Once the validation is done on data file the error records are moved to the bad file. So till here everything is working fine .
Now i wanted to add a error code to each bad record in the badfile along with the error field.

Below i have shown the details of datafile, confile, code tried, output getting and output expected:
configurationfile:
Code:
id,Integer(3),NOT NULL
name,String(20)
state,String(5),NOT NULL
phone_no,Integer(4)
gender,Char(1)

datafile:
Code:
201,John,MI,4589,M
202,Lilly,FL,589,F
20w,Taylor,,5888,M
210,8888,OK,456
215,Madav,,4454,M
2165,ram,MI,4589,M
21734,Leena,,589,F
218,Rohan,CA,2212,M

Script/Code:
Code:
#!/bin/bash
awk -F "," -vDT="$(date +%m%d%Y%H%M)" 'BEGIN { 
GOOD = "good_" DT; #Adding timestamp into a GOOD file
BAD = "bad_" DT; #Adding timestamp into a BAD file
putB = "hadoop fs -put /home/user/data/" BAD " /user/user/bad/"} 
NR == FNR{
gsub("[)(]", "-", $2); 
split($2, a, "-"); 
split($1, c,",");
hh[NR] = c[1]; d[NR] = a[1]; l[NR] = a[2]; n[NR] = ($3 == "NOT NULL") ? 1 : 0; next} 
{
for(i = 1; i <= NF; i++)
{

if(((d[i] == "Integer" && (($i + 0) == $i || $i == "")) || (d[i] == "String" && ($i + 0) != $i) || (d[i] == "Char" && ($i + 0) != $i)) && (length($i) <= l[i]) && (length($i) >= n[i]))
{f = 1} else {f = 0};
if(f == 0) {print $0 > BAD; b++; next} 
}
print $0 > GOOD; g++
}
END {
print "Count of Bad Records : " b;
#system(putB);
}' configfile.txt datafile2.txt

Output getting without errorcode:
Code:
20w,Taylor,,5888,M
210,8888,OK,456
215,Madav,,4454,M
2165,ram,MI,4589,M
21734,Leena,,589,F

Expected output along with the error detail along with the error field name:
Code:
20w,Taylor,,5888,M,datatypeerror|id
210,8888,OK,456,datatyprerror|name
215,Madav,,4454,M,nullerror|state
2165,ram,MI,4589,M,columnwidthError|id
21734,Leena,,589,F,columnwidthError|id,nullerror|state

In the above expected result, at the end of each record the error details along with the eoorr field. So this can be achieved?

Thanks,
Shree
# 2  
Old 06-13-2014
You must split your long all-in-one condition into several smaller conditions, and each time add an individual error message to an error-message variable.
For example:
Code:
f=0
error=""
if (d[i] == "Integer" && (($i + 0) == $i || $i == "")) {f=1} else {error = error ",datatypeerror|" hh[i]}
if (d[i] == "String" && ($i + 0) != $i) {f=1} else {error = error ",nullerror|" hh[i]}
if (d[i] == "Char" && ($i + 0) != $i) {f=1} else {error = error ",datatyperror|" hh[i]}
if ((length($i) <= l[i]) && (length($i) >= n[i])) {f=1} else {error = error ",columnwidthError|" hh[i]}

# 3  
Old 06-16-2014
Hi,
I tried the similar thing in my code. May be because some syantax checks it's giving me an error.
I got the logic you suggeseted but while implementing the same in the entire code somewhere it's going wrong.
Also i dint get how to insert the piece of code
Code:
f=0
error=""

I'm sure somewhere here its going wrong! How this can be acheived?

Thanks,
Shree
Last edited by Scrutinizer; 06-24-2014 at 04:51 AM.. Reason: quote -> code tags
# 4  
Old 06-20-2014
Hi, Is there any updates on the above post ?
# 5  
Old 06-20-2014
Post your updated script, and the error you are getting.
# 6  
Old 06-20-2014
Try this
Code:
awk -F, 'NR==FNR        {NM[NR]=$1
                           split ($2,X,"[)(]")
                         TP[NR]=toupper(X[1])
                         MX[NR]=X[2]
                         MN[NR]=$3!=""
                         FLDS  =NR
                         next}
                        {OUT=$0
                         for (i=1; i<=FLDS; i++) {
                           L=length($i)
                           N=$i==$i+0
                           if (TP[i] == "INTEGER" && !N) OUT = OUT ",dattyperr|" NM[i]
                           if (TP[i] == "STRING" &&   N) OUT = OUT ",dattyperr|" NM[i]
                           if (L > MX[i])                OUT = OUT ",colwiderr|" NM[i]
                           if (L < MN[i])                OUT = OUT ",nullerr|" NM[i]
                          }
                         print OUT
                        }
        ' file1 file2
201,John,MI,4589,M
202,Lilly,FL,589,F
20w,Taylor,,5888,M,dattyperr|id,nullerr|state
210,8888,OK,456,dattyperr|name
215,Madav,,4454,M,nullerr|state
2165,ram,MI,4589,M,colwiderr|id
21734,Leena,,589,F,colwiderr|id,nullerr|state
218,Rohan,CA,2212,M

Checking for CHAR or FLOAT has not yet been implemented. Redirection shouldn't be a problem to you.
# 7  
Old 06-23-2014
Hi RudiC ,

Your code alone is working fine but i'm facing problem in redirection.When i implement the similar thing into my code its going wrong. Also in your code, you are printing the results on screen but i wanted to print those good and bad records on different files.

Thanks
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Getting error not able remove the record

export PATH=$PATH:/opt/IBM/db2/V9.5/95fp4/bin DB2INSTANCE=bancbcb1 export DB2INSTANCE db2 "connect to dyaxprpt user xyz using byx" echo "Load lastest data start" Row=$(db2 –x “delete from xyz.ROL_yxpwhere axl_ING_ID in(50127,50455,503458,175748)”) If ; then echo “no data deleted” elif ;... (2 Replies)
Discussion started by: jagu
2 Replies

2. Shell Programming and Scripting

Appending CRLF to end of record

I need to append |\r\n (a pipe character and CRLF) at end of each record in Unix to all records where they are not already present. So first check for the presence of |\r\n and if absent append it else do nothing (3 Replies)
Discussion started by: abhilashnair
3 Replies

3. Shell Programming and Scripting

Extract timestamp from first record in xml file and it checks if not it will replace first record

I have test.xml <emp><id>101</id><name>AAA</name><date>06/06/14 1811</date></emp> <Join><id>101</id><city>london</city><date>06/06/14 2011</date></join> <Join><id>101</id><city>new york</city><date>06/06/14 1811</date></join> <Join><id>101</id><city>sydney</city><date>06/06/14... (2 Replies)
Discussion started by: vsraju
2 Replies

4. Shell Programming and Scripting

How to compare current record,with next and previous record in awk without using array?

Hi! all can any one tell me how to compare current record of column with next and previous record in awk without using array my case is like this input.txt 0 32 1 26 2 27 3 34 4 26 5 25 6 24 9 23 0 32 1 28 2 15 3 26 4 24 (7 Replies)
Discussion started by: Dona Clara
7 Replies

5. Shell Programming and Scripting

Splitting record into multiple records by appending values from an input field (AWK)

Hello, For the input file, I am trying to split those records which have multiple values seperated by '|' in the last input field, into multiple records and each record corresponds to the common input fields + one of the value from the last field. I was trying with an example on this forum... (4 Replies)
Discussion started by: imtiaz99
4 Replies

6. Shell Programming and Scripting

Appending error messages from log file next to the corresponding error record

Hi Everyone, I have an issue and trying to get a solution but was not succesful yet. Any help is greatly appreciated. I am using ksh to inoke sql loader to load data from txt file into two oracle tables based on the condition written in the control file. If an error occurs while loading into... (8 Replies)
Discussion started by: vpv0002
8 Replies

7. UNIX for Dummies Questions & Answers

Appending error

Hi All, I just want to append the value in variable at the end of the file. var=1234 sed -e "$a $var" file1 > file 2. But I get this error sed: -e expression #1, char 4: unknown command: `1' Kindly let m know how can I do that... (5 Replies)
Discussion started by: waqar1
5 Replies

8. Shell Programming and Scripting

Error while appending records to a file

Hi, I have a sample file which contains records. Input File : 1 user1 username1\password@database-name 2 user2 username2\password@database-name 3 user3 username1\password@database-name I should search for a 'username1\' in those records. If 'username1\' is found in those records, that record... (7 Replies)
Discussion started by: siri_886
7 Replies

9. AIX

pax error on appending data to LTO3

I have problem when I use the command "pax -awvf /dev/rmt0 ./data1" in AIX 5.3.0.0. The command with parameter -a allow me to append the tape but when I try to retrieve the data that I append, it will show me error. I would like to know if anyone have the same problem and any solution found? Tq. (0 Replies)
Discussion started by: kwliew999
0 Replies

10. Shell Programming and Scripting

appending spaces to first line based on second record.

Hi, I have a situation to append spaces to end of first record (header)and last record (footer) based on second record length. The first record length is always 20.The second record will be different for different files.I have to append spaces for the first line based on second record... (2 Replies)
Discussion started by: ammu
2 Replies
Login or Register to Ask a Question