Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Source data validation

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Source data validation
# 1  
Old 08-13-2015
Source data validation
I am using below logic to validate whether i am expecting the correct data from source,if not logic should give which column has error.i am running below logic in linux

Code:
awk -F, '
NR==1{next}
{f=" "}
$1!~/^[0-9. -]{0,5}$|^$/{f=f?f" emp_id-error":"emp_id-error"}
$4!~/^[0-9 -]{0,6}$|^$/{f=f?f" tot_amt-error":"tot_amt-error"}
$5!~/^[0-9" "]{0,4}-[0-9" "]{0,2}-[0-9" "]{0,2} [0-9" "]{0,2}:[0-9" "]{0,2}:[0-9" "]{0,2}{0,29}$|^$/{f=f?f" current_time-error":"current_time-error"}
{print f,$0}
' /home/vivek/Documents/datagentest_10.txt

datagentest_10.txt file

Code:
emp_id,emp_name,market_name,tot_amt,current_time
1,Brandon Buckner,avil,526,2015-08-11 00:47:00
11,ram,avil,633,2015-08-11 00:47:00
11,raffm,avil,633,2015-08-11 00:47:00
11,ramrtt,avil,633,2015-08-11 00:47:00

data looks good but i am getting error for all columns and for all rows.please help me.


Code:
error message:

emp_id-error tot_amt-error current_time-error 1,Brandon Buckner,avil,526,2015-08-11 00:47:00

Moderator's Comments:
Mod Comment Note that after receiving ten infractions for not using CODE tags (as required by forum rules) and a few other infractions as well, the submitter has been placed in read-only mode for two weeks.
Last edited by Don Cragun; 08-13-2015 at 04:51 PM.. Reason: Add CODE and ICODE tags.
# 2  
Old 08-13-2015
Ignore
# 3  
Old 08-13-2015
when i run below code in linux i am getting error for all columns.not sure why getting error in linux.please help me.

Code:
awk -F, '
NR==1{next}
{f=""}
$1!~/^[0-9]{0,12}$/{f=f?f" ServiID-error":"ServiID-error"} 
$4!~/^[0-9.-]{0,12}$/{f=f?f" ServicerID-error":"ServicerID-error"} 
$5!~/^[0-9]{0,4}-[0-9]{0,2}-[0-9]{0,2} [0-9]{0,2}:[0-9]{0,2}:[0-9]{0,2}$|^[" "]{0,38}$/{f=f?f" DataAsOfDate-error":"DataAsOfDate-error"} 
{print f,$0}
' /home/vivek/Documents/viv.txt


sample data
Code:
emp_id,emp_name,market_name,tot_amt,current_time
1,Brandon Buckner,avil,526,2015-08-11 00:47:00
2,ram,avil,633,2015-08-11 00:47:00
3,raffm,avil,633,2015-08-11 00:47:00
4,ramrtt,avil,633,2015-08-11 00:47:00

# 4  
Old 08-13-2015
Looks like you've assumed a second identity in time, no?
# 5  
Old 08-13-2015
I am getting error for all columns,in first column I have value 1 but still I am getting error..not sure what the problem is..
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help With UNIX Shell Scripting For Data Validation

Hi All, I am completely new to Unix Shell Scripting. I m validating(Basic File Validation) an .HHT file in TIBCO. After that i need to do Data Validation through UNIX Shell scripting. Rules in DataValidation: 1.) Need to Check Every field wheather it is a Char or Number?(Fields are... (1 Reply)
Discussion started by: Chaitanya K
1 Replies

2. Shell Programming and Scripting

Data validation engine

Generic Data validator Data file: Name,Sal,Dept ABC,1234,D1 AYX,12356,D2 DHF,345,ED3 123,4565,FGJG Config File: Delimiter-"," Rule1-Name- Rule2-Sal- Rule3-Dept-* Can be used to match any regex including date different format and numbers. (3 Replies)
Discussion started by: dikesm
3 Replies

3. Shell Programming and Scripting

basic data validation

hpux. older version. don't have alot of the newer features in some utilities. How do I verify that a variable starts with the letter A. I would like to make it case insensitive. Also, if I have a variable that has letters and numbers. I want to change all the lower case letters to upper case.... (2 Replies)
Discussion started by: guessingo
2 Replies

4. Shell Programming and Scripting

File and Data Validation.

Hello, I am working on an interface between a legacy system and an ERP system.The format of the data extracted into the staging folder from the legacy system is a follows. One control file named ExtractDataControl.txt Multiple entity files eg R1001.txt, R1002.txt, R1020.txt The control... (5 Replies)
Discussion started by: Savio_Saldanha
5 Replies

5. Shell Programming and Scripting

Data Validation

I have a comma delimited file that I need to validate the data in one two columns in. Sample File: 1234,1234,1234,DESCRIPTION,1,1,2 1234,1234,1234,DESCRIPTION,1,1,2 1234,1234,1234,DESCRIPTION,1,1,2 1234,1234,1234,DESCRIPTION,1,1,2 I need to make sure that the second column's entries are... (3 Replies)
Discussion started by: hmnetonline
3 Replies

6. Shell Programming and Scripting

shell script data & time validation

How to validate a date and optionly a time in shell scripting when i get the date and time as pararmeters that sent out with the call of the file? (in my case sh union.sh `first parameter ,second parameter...` (4 Replies)
Discussion started by: tal
4 Replies

7. Shell Programming and Scripting

validation of data using filter (awk or other that works...) in csv files

Hello People I have the following file.csv: date,string,float,number,boolean 20080303,abc,1.5,123,Y 20080304,abc,1.2,345,N 20080229,nvh,1.4,098,Y 20080319,ugy,1.9,586,N 20080315,gyh,2.4,345,Y 20080316,erf,3.1,932,N I need to filter the date field where I have a data bigger than I... (1 Reply)
Discussion started by: Rafael.Buria
1 Replies

8. UNIX for Dummies Questions & Answers

Data Validation

Hello, I am trying to use data validation with a program. I have everything else working fine. I just can't figure out what I am doing wrong with the data validation in one of my files. Here is the code: # data validation loop while : do # get input from keyboard ... (2 Replies)
Discussion started by: ericelysia
2 Replies

9. UNIX for Advanced & Expert Users

awk data validation

Hi , This is a general doubt.... Is there any way to do data validation inside an awk script.. Let me make it more clear.... I have a string variable inside awk script .. Is there any way to check whether it is number or string etc... Thanks in advance. Shihab (1 Reply)
Discussion started by: shihabvk
1 Replies
Login or Register to Ask a Question