Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Source data validation
Top Forums Shell Programming and Scripting Source data validation Post 302952098 by katakamvivek on Thursday 13th of August 2015 03:09:59 PM
Old 08-13-2015
Source data validation
I am using below logic to validate whether i am expecting the correct data from source,if not logic should give which column has error.i am running below logic in linux

Code:
awk -F, '
NR==1{next}
{f=" "}
$1!~/^[0-9. -]{0,5}$|^$/{f=f?f" emp_id-error":"emp_id-error"}
$4!~/^[0-9 -]{0,6}$|^$/{f=f?f" tot_amt-error":"tot_amt-error"}
$5!~/^[0-9" "]{0,4}-[0-9" "]{0,2}-[0-9" "]{0,2} [0-9" "]{0,2}:[0-9" "]{0,2}:[0-9" "]{0,2}{0,29}$|^$/{f=f?f" current_time-error":"current_time-error"}
{print f,$0}
' /home/vivek/Documents/datagentest_10.txt

datagentest_10.txt file

Code:
emp_id,emp_name,market_name,tot_amt,current_time
1,Brandon Buckner,avil,526,2015-08-11 00:47:00
11,ram,avil,633,2015-08-11 00:47:00
11,raffm,avil,633,2015-08-11 00:47:00
11,ramrtt,avil,633,2015-08-11 00:47:00

data looks good but i am getting error for all columns and for all rows.please help me.


Code:
error message:

emp_id-error tot_amt-error current_time-error 1,Brandon Buckner,avil,526,2015-08-11 00:47:00

Moderator's Comments:
Mod Comment Note that after receiving ten infractions for not using CODE tags (as required by forum rules) and a few other infractions as well, the submitter has been placed in read-only mode for two weeks.
Last edited by Don Cragun; 08-13-2015 at 04:51 PM.. Reason: Add CODE and ICODE tags.
 

9 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

awk data validation

Hi , This is a general doubt.... Is there any way to do data validation inside an awk script.. Let me make it more clear.... I have a string variable inside awk script .. Is there any way to check whether it is number or string etc... Thanks in advance. Shihab (1 Reply)
Discussion started by: shihabvk
1 Replies

2. UNIX for Dummies Questions & Answers

Data Validation

Hello, I am trying to use data validation with a program. I have everything else working fine. I just can't figure out what I am doing wrong with the data validation in one of my files. Here is the code: # data validation loop while : do # get input from keyboard ... (2 Replies)
Discussion started by: ericelysia
2 Replies

3. Shell Programming and Scripting

validation of data using filter (awk or other that works...) in csv files

Hello People I have the following file.csv: date,string,float,number,boolean 20080303,abc,1.5,123,Y 20080304,abc,1.2,345,N 20080229,nvh,1.4,098,Y 20080319,ugy,1.9,586,N 20080315,gyh,2.4,345,Y 20080316,erf,3.1,932,N I need to filter the date field where I have a data bigger than I... (1 Reply)
Discussion started by: Rafael.Buria
1 Replies

4. Shell Programming and Scripting

shell script data & time validation

How to validate a date and optionly a time in shell scripting when i get the date and time as pararmeters that sent out with the call of the file? (in my case sh union.sh `first parameter ,second parameter...` (4 Replies)
Discussion started by: tal
4 Replies

5. Shell Programming and Scripting

Data Validation

I have a comma delimited file that I need to validate the data in one two columns in. Sample File: 1234,1234,1234,DESCRIPTION,1,1,2 1234,1234,1234,DESCRIPTION,1,1,2 1234,1234,1234,DESCRIPTION,1,1,2 1234,1234,1234,DESCRIPTION,1,1,2 I need to make sure that the second column's entries are... (3 Replies)
Discussion started by: hmnetonline
3 Replies

6. Shell Programming and Scripting

File and Data Validation.

Hello, I am working on an interface between a legacy system and an ERP system.The format of the data extracted into the staging folder from the legacy system is a follows. One control file named ExtractDataControl.txt Multiple entity files eg R1001.txt, R1002.txt, R1020.txt The control... (5 Replies)
Discussion started by: Savio_Saldanha
5 Replies

7. Shell Programming and Scripting

basic data validation

hpux. older version. don't have alot of the newer features in some utilities. How do I verify that a variable starts with the letter A. I would like to make it case insensitive. Also, if I have a variable that has letters and numbers. I want to change all the lower case letters to upper case.... (2 Replies)
Discussion started by: guessingo
2 Replies

8. Shell Programming and Scripting

Data validation engine

Generic Data validator Data file: Name,Sal,Dept ABC,1234,D1 AYX,12356,D2 DHF,345,ED3 123,4565,FGJG Config File: Delimiter-"," Rule1-Name- Rule2-Sal- Rule3-Dept-* Can be used to match any regex including date different format and numbers. (3 Replies)
Discussion started by: dikesm
3 Replies

9. Shell Programming and Scripting

Help With UNIX Shell Scripting For Data Validation

Hi All, I am completely new to Unix Shell Scripting. I m validating(Basic File Validation) an .HHT file in TIBCO. After that i need to do Data Validation through UNIX Shell scripting. Rules in DataValidation: 1.) Need to Check Every field wheather it is a Char or Number?(Fields are... (1 Reply)
Discussion started by: Chaitanya K
1 Replies

LEARN ABOUT DEBIAN

lbt

LBT(1)							      General Commands Manual							    LBT(1)

NAME

       lbt - LTL to Buchi Translator

SYNOPSIS

       lbt < formula.txt > automaton.txt
       lbt2dot < automaton.txt > automaton.dot

DESCRIPTION

       This  manual  page  documents  briefly  the  lbt  and lbt2dot commands.	This manual page was written for the Debian GNU/Linux distribution
       because the original program does not have a manual page.  Instead, it has documentation in HTML format; see below.

       lbt is a filter that translates a linear temporal logic (LTL) formula to a corresponding generalized Buchi automaton.  The  translation	is
       based  on  the algorithm by Gerth, Peled and Vardi presented at PSTV'95, Simple on-the-fly automatic verification of linear temporal logic.
       Hardly any optimizations are implemented, and the generated automaton is often bigger than necessary.  But on the  other  hand,	it  should
       always be correct.
       The filter lbt2dot can be used to translate Buchi automata from the lbt output format to GraphViz format for visualization.

EXAMPLE

       echo G p0 | lbt | lbt2dot | dotty -

SEE ALSO

       dotty(1).

FILES

       /usr/share/doc/lbt/html/index.html
	      The real documentation for LBT.

AUTHOR

       This  manual  page was written by Marko Makela <msmakela@tcs.hut.fi>, for the Debian GNU/Linux system (but may be used by others).  The lbt
       program was written by Mauno Ronkko and Heikki Tauriainen, and it was optimized by Marko Makela, who also wrote the lbt2dot filter.  Please
       see the copyright file in /usr/share/doc/lbt for details.

								  August 10, 2001							    LBT(1)
All times are GMT -4. The time now is 01:42 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy