Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Appending ErrorCodes to the corresponding error record

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Appending ErrorCodes to the corresponding error record
# 8  
Old 06-23-2014
Try e.g.
Code:
print OUT > (OUT==$0)?GOOD:BAD

# 9  
Old 06-23-2014
Hi RudiC ,

My code is not giving the proper output. I have 20 records in my file .Out of 20 ,4 records are error records and rest 16 are good records. When i run the script which aontains the errorCode appending mechanism, all my 20 records are written to the good file and bad record file is not even getting created.
Below is the script:
Code:
#!/bin/bash
awk -F "," -vDT="$(date +%m%d%Y%H%M)" 'BEGIN { 
GOOD = "good_" DT; #Adding timestamp into a GOOD file
BAD = "bad_" DT; #Adding timestamp into a BAD file
putB = "hadoop fs -put /home/user/data/" BAD " /user/user/bad/"} 
NR == FNR{
gsub("[)(]", "-", $2); 
split($2, a, "-"); 
split($1, c,",");
hh[NR] = c[1]; d[NR] = a[1]; l[NR] = a[2]; n[NR] = ($3 == "NOT NULL") ? 1 : 0; next} 
{
 
for(i = 1; i <= NF; i++)
{
OUT=$0
if(d[i] == "Integer" && (($i + 0) == $i || $i == "")) {f=1} else{OUT = OUT ",dattyperr|" hh[i]}
if(d[i] == "String" && ($i + 0) != $i){f=1} else{OUT = OUT ",dattyperr|" hh[i]}
if(d[i] == "Char" && ($i + 0) != $i) {f=1} else{OUT = OUT ",dattyperr|" hh[i]}
if((length($i) <= l[i]) && (length($i) >= n[i])){f=1} else{OUT = OUT ",colwiderr|" hh[i]}
if(f == 0) {print OUT > BAD; b++; next} 
}
print OUT > GOOD; g++
}
END {
print "Count of Bad Records : " b;
print "Count of Good Records : " g;
#system(putB);
}' a.txt b.txt

My a.txt file is
Code:
id,Integer(3),NOT NULL
name,String(20)
state,String(5),NOT NULL
phone_no,Integer(4)
gender,String(1)

b.txt is:
Code:
201,John,MI,4589,M
202,Lilly,FL,589,F
203,Richard,CA,2212,M
204,Cruse,VA,2222,M
20w,Taylor,,5888,M
206,Merry,TX,6969,F
207,,CO,5656,F
208,Tom,AL,5555,M
209,Sam,FL,2586,M
210,8888,OK,456
211,George11,MI,5555,M
212,Reet,MI,4589,M
213,8888a,FL,5899,F
214,Meera,NY,2546,F
215,Madav,,4454,M
216,ram,MI,4589,M
217,Leena,FL,589,F
218,Rohan,CA,2212,M
219,Raj,VA,2222,M
TaylorSmith,NY,5888,M

The output written on the good file is :
Code:
201,John,MI,4589,M,dattyperr|gender,dattyperr|gender
202,Lilly,FL,589,F,dattyperr|gender,dattyperr|gender
203,Richard,CA,2212,M,dattyperr|gender,dattyperr|gender
204,Cruse,VA,2222,M,dattyperr|gender,dattyperr|gender
20w,Taylor,,5888,M,dattyperr|gender,dattyperr|gender
206,Merry,TX,6969,F,dattyperr|gender,dattyperr|gender
207,,CO,5656,F,dattyperr|gender,dattyperr|gender
208,Tom,AL,5555,M,dattyperr|gender,dattyperr|gender
209,Sam,FL,2586,M,dattyperr|gender,dattyperr|gender
210,8888,OK,456,dattyperr|phone_no,dattyperr|phone_no
211,George11,MI,5555,M,dattyperr|gender,dattyperr|gender
212,Reet,MI,4589,M,dattyperr|gender,dattyperr|gender
213,8888a,FL,5899,F,dattyperr|gender,dattyperr|gender
214,Meera,NY,2546,F,dattyperr|gender,dattyperr|gender
215,Madav,,4454,M,dattyperr|gender,dattyperr|gender
216,ram,MI,4589,M,dattyperr|gender,dattyperr|gender
217,Leena,FL,589,F,dattyperr|gender,dattyperr|gender
218,Rohan,CA,2212,M,dattyperr|gender,dattyperr|gender
219,Raj,VA,2222,M,dattyperr|gender,dattyperr|gender
TaylorSmith,NY,5888,M,dattyperr|phone_no,dattyperr|phone_no,dattyperr|phone_no

Expected output in bad file is :
Code:
20w,Taylor,,5888,M,dattyperr|id,nullerror|state
210,8888,OK,456,dattyperr|name
215,Madav,,4454,M,nullerror|state
TaylorSmith,NY,5888,M,dattyperr|id,dattyperr|state,dattyperr|phone_no

Also in my case, how can i append the nullerror.

Thanks,
Last edited by Scrutinizer; 06-24-2014 at 04:50 AM.. Reason: quote -> code tags
# 10  
Old 06-23-2014
Applied to your latest files, my proposal gives:
Code:
GOOD:
201,John,MI,4589,M
202,Lilly,FL,589,F
203,Richard,CA,2212,M
204,Cruse,VA,2222,M
206,Merry,TX,6969,F
207,,CO,5656,F
208,Tom,AL,5555,M
209,Sam,FL,2586,M
211,George11,MI,5555,M
212,Reet,MI,4589,M
213,8888a,FL,5899,F
214,Meera,NY,2546,F
216,ram,MI,4589,M
217,Leena,FL,589,F
218,Rohan,CA,2212,M
219,Raj,VA,2222,M
BAD:
20w,Taylor,,5888,M,dattyperr|id,nullerr|state
210,8888,OK,456,dattyperr|name
215,Madav,,4454,M,nullerr|state
TaylorSmith,NY,5888,M ,dattyperr|id,colwiderr|id,dattyperr|state,dattyperr|phone_no

So, what does not work?
# 11  
Old 06-23-2014
Hi, your proposal is working fine.
But how can i add the good and bad records on the different files along with the timestamp attached to the good & bad filename. Alos i wanted to count the good and bad records and show it on the console. If i add code related the above requirements i'm getting the error.

Script used for this is:
Code:
#!/bin/bash
awk -F "," -vDT="$(date +%m%d%Y%H%M)" 'BEGIN {    
GOOD = "good_" DT;   #Adding timestamp into a GOOD file
BAD = "bad_" DT;     #Adding timestamp into a BAD file
putB = "hadoop fs -put /home/user/data/" BAD " /user/user/bad/"}     
NR == FNR{
gsub("[)(]", "-", $2);    
split($2, a, "-");    
split($1, c,",");
hh[NR] = c[1]; d[NR] = a[1]; l[NR] = a[2];MN[NR]=$3!=""; next}   
{
  
  for(i = 1; i <= NF; i++)
  {
     OUT=$0
     L=length($i)
     N=$i==$i+0
     if (d[i] == "INTEGER" && !N) OUT = OUT ",dattyperr|" hh[i]
     if (d[i] == "STRING" &&   N) OUT = OUT ",dattyperr|" hh[i]
     if (L > l[i])                OUT = OUT ",colwiderr|" hh[i]
     if (L < MN[i])                OUT = OUT ",nullerr|" hh[i]
 }
      print OUT > (OUT==$0)?GOOD:BAD
}
END {
print "Count of Bad Records : " b;
print "Count of Good Records : " g;
#system(putB);
    }' configfile.txt datafile2.txt

Also how can i count the good and bad records?
# 12  
Old 06-23-2014
Try
Code:
         OUT==$0        {print OUT > GOOD
                         g++
                         next}
                        {print OUT > BAD
                         b++}
         END            {print "Good: ", g, ", bad: ", b}

# 13  
Old 06-23-2014
Hi,

Still i'm getting some syntax error in the below code:
Code:
#!/bin/bash
awk -F "," -vDT="$(date +%m%d%Y%H%M)" 'BEGIN {    
GOOD = "good_" DT;   #Adding timestamp into a GOOD file
BAD = "bad_" DT;     #Adding timestamp into a BAD file
putB = "hadoop fs -put /home/user/data/" BAD " /user/user/bad/"}     
NR == FNR{
gsub("[)(]", "-", $2);    
split($2, a, "-");    
split($1, c,",");
hh[NR] = c[1]; d[NR] = a[1]; l[NR] = a[2];MN[NR]=$3!=""; next}   
{
  for(i = 1; i <= NF; i++)
  {
     OUT=$0
     L=length($i)
     N=$i==$i+0
     if (d[i] == "INTEGER" && !N) OUT = OUT ",dattyperr|" hh[i]
     if (d[i] == "STRING" &&   N) OUT = OUT ",dattyperr|" hh[i]
     if (L > l[i])                OUT = OUT ",colwiderr|" hh[i]
     if (L < MN[i])                OUT = OUT ",nullerr|" hh[i]
     OUT==$0 {print OUT > GOOD; g++;next}
  }
     {print OUT > BAD: b++}
}
END {      
print "Good: ", g, ", bad: ", b
print "parsing sucess!!";
print "Count of Bad Records : " b;
print "Count of Good Records : " g;
#system(putB);
    }' a.txt b.txt

# 14  
Old 06-23-2014
Quote:
Originally Posted by shree11
Still i'm getting some syntax error in the below code:
It's much easier for people to help you if you provide the actual error message.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Getting error not able remove the record

export PATH=$PATH:/opt/IBM/db2/V9.5/95fp4/bin DB2INSTANCE=bancbcb1 export DB2INSTANCE db2 "connect to dyaxprpt user xyz using byx" echo "Load lastest data start" Row=$(db2 –x “delete from xyz.ROL_yxpwhere axl_ING_ID in(50127,50455,503458,175748)”) If ; then echo “no data deleted” elif ;... (2 Replies)
Discussion started by: jagu
2 Replies

2. Shell Programming and Scripting

Appending CRLF to end of record

I need to append |\r\n (a pipe character and CRLF) at end of each record in Unix to all records where they are not already present. So first check for the presence of |\r\n and if absent append it else do nothing (3 Replies)
Discussion started by: abhilashnair
3 Replies

3. Shell Programming and Scripting

Extract timestamp from first record in xml file and it checks if not it will replace first record

I have test.xml <emp><id>101</id><name>AAA</name><date>06/06/14 1811</date></emp> <Join><id>101</id><city>london</city><date>06/06/14 2011</date></join> <Join><id>101</id><city>new york</city><date>06/06/14 1811</date></join> <Join><id>101</id><city>sydney</city><date>06/06/14... (2 Replies)
Discussion started by: vsraju
2 Replies

4. Shell Programming and Scripting

How to compare current record,with next and previous record in awk without using array?

Hi! all can any one tell me how to compare current record of column with next and previous record in awk without using array my case is like this input.txt 0 32 1 26 2 27 3 34 4 26 5 25 6 24 9 23 0 32 1 28 2 15 3 26 4 24 (7 Replies)
Discussion started by: Dona Clara
7 Replies

5. Shell Programming and Scripting

Splitting record into multiple records by appending values from an input field (AWK)

Hello, For the input file, I am trying to split those records which have multiple values seperated by '|' in the last input field, into multiple records and each record corresponds to the common input fields + one of the value from the last field. I was trying with an example on this forum... (4 Replies)
Discussion started by: imtiaz99
4 Replies

6. Shell Programming and Scripting

Appending error messages from log file next to the corresponding error record

Hi Everyone, I have an issue and trying to get a solution but was not succesful yet. Any help is greatly appreciated. I am using ksh to inoke sql loader to load data from txt file into two oracle tables based on the condition written in the control file. If an error occurs while loading into... (8 Replies)
Discussion started by: vpv0002
8 Replies

7. UNIX for Dummies Questions & Answers

Appending error

Hi All, I just want to append the value in variable at the end of the file. var=1234 sed -e "$a $var" file1 > file 2. But I get this error sed: -e expression #1, char 4: unknown command: `1' Kindly let m know how can I do that... (5 Replies)
Discussion started by: waqar1
5 Replies

8. Shell Programming and Scripting

Error while appending records to a file

Hi, I have a sample file which contains records. Input File : 1 user1 username1\password@database-name 2 user2 username2\password@database-name 3 user3 username1\password@database-name I should search for a 'username1\' in those records. If 'username1\' is found in those records, that record... (7 Replies)
Discussion started by: siri_886
7 Replies

9. AIX

pax error on appending data to LTO3

I have problem when I use the command "pax -awvf /dev/rmt0 ./data1" in AIX 5.3.0.0. The command with parameter -a allow me to append the tape but when I try to retrieve the data that I append, it will show me error. I would like to know if anyone have the same problem and any solution found? Tq. (0 Replies)
Discussion started by: kwliew999
0 Replies

10. Shell Programming and Scripting

appending spaces to first line based on second record.

Hi, I have a situation to append spaces to end of first record (header)and last record (footer) based on second record length. The first record length is always 20.The second record will be different for different files.I have to append spaces for the first line based on second record... (2 Replies)
Discussion started by: ammu
2 Replies
Login or Register to Ask a Question