Need a ready Shell script to validate a high volume data file

Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Need a ready Shell script to validate a high volume data file
# 1  
Old 10-02-2013
Need a ready Shell script to validate a high volume data file


I am looking for a ready shell script that can help in loading and validating a high volume (around 4 GB) .Dat file . The data in the file has to be validated at each of its column, like the data constraint on each of the data type on each of its 60 columns and also a few other constraints on each of the column.

After validation, the rows wiith error records need to be written to an exception report and the correct data has to be loaded separately.

If anyone has an idea or any ready to use shell script which can customised for my requirement it will be greatly appreciated.
# 2  
Old 10-02-2013
There is no "ready shell script" for such a task. Provide a sample of the data file. Be specific about the rules for validating each column (field) for a line (record) of the data file.
# 3  
Old 10-03-2013
Originally Posted by blackrageous
There is no "ready shell script" for such a task. Provide a sample of the data file. Be specific about the rules for validating each column (field) for a line (record) of the data file.

I have attached zip file which contains two files.
1.) ipbcdg_check_fotp_20130909_2.dat ( This file contains data which we have to validate. Tilde sign(~) is used to seperate the different Columns values.
2.)Data_Validation_Sheet.xls ( This files contains all the rules, Against which we have to validate the Data.)

In Data validation file we have different sheet which are for different-2 interfaces e.g "cash","check" and two differnet sheets for Rules and lookup tables.

How to find for which interface we have to validate :-
In dat file name we have "check_fotp" which is our interface name.
If you open Data validation file we found one sheet with name "Check" and in that sheet for first column we have two different values . "Check_fot" and "check_fotp".

Now corresponding to "Check_fotp" values we have columns name which we have to validate in the dat file.

E.g we take first row value corresponing to "Check_fotp"
check_fotp, Transaction Reference Identifier, CHARNUM(50) ,Y, "RC_1", "RC_2", "RC_3", "RC_7"

The values "RC_1", "RC_2", "RC_3", "RC_7" have rules which we have to validate and these descriptions are present in different sheet in "Data validation file" , sheet name s"REF_CD_DETAIL".

Some time we have differnt values in the Rule column. like in 3 row corresponding to "check_fotp" we have one value "Party_Role"

This means if any value occured we have to check "lookup" sheet in the "data validation sheet " and lookup sheet we have one table corresponding to "Party_Role" which have some values.
So in dat file for this columns values should be from this "Party_Role" table only.

I know its complicated. If you need any other information please let me know.

Thanks in advance for your help.

Last edited by Guruprasad; 10-03-2013 at 06:13 AM.. Reason: updation
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell Script to validate the access for Servers

Hi Experts, Please advise to Write a Shell script, To check , whether the user has access to UNIX PRODUCTION server or not? If no, then need to provide the access to UNIX PRODUCTION server and then need to provide access to print only few FILES. After connecting to ther server, again... (5 Replies)
Discussion started by: vasuvv
5 Replies

2. Shell Programming and Scripting

Bash to verify and validate file header and data type

The below bash is a file validation check executed that will verify the correct header count of 10 and the correct data type in each field of the tab-delimited file. The key has the data type of each field in it. My real data has 58 headers in it but only the header and next row need to be... (6 Replies)
Discussion started by: cmccabe
6 Replies

3. Shell Programming and Scripting

Output large volume of data to CSV file

I have a program that output the ownership and permission on each directory and file on the server to a csv file. I am getting error message when I run the program. The program is not outputting to the csv file. Error: the file access permissions do not allow the specified action cannot... (2 Replies)
Discussion started by: dellanicholson
2 Replies

4. Shell Programming and Scripting

Script to ingest a csv, validate data and insert into Oracle

Hi all i would appreciate your help... I am looking for a set of unix commands which i can use to 1) ingest a csv file with a known format 2) validate the filename 3) validate the data/datatypes 4) Insert into an oracle db Can you help get me started? yogz888 (1 Reply)
Discussion started by: yogz888
1 Replies

5. UNIX for Dummies Questions & Answers

Shell script to read lines in a text file and filter user data Shell Programming and Scripting

sxsaaas (3 Replies)
Discussion started by: VikrantD
3 Replies

6. Shell Programming and Scripting

Validate file count in korn shell script

Hi, I have files in the directory like below which I need to validate if all the required files are present. A_B_001 of 002_time1.txt A_B_002 of 002_time1.txt A_B_001 of 001_time2.txt Scenarios- a)If file with 001 of 002_time1 or 002 of 002_time1 is missing in the folder,script should... (6 Replies)
Discussion started by: aneeta13
6 Replies

7. Shell Programming and Scripting

how to validate data of excel using perl or shell

I have a software which generates excel report with some specific data. The excel file format is .xls (old 2003 format) The data are in the forms like differenct cells contains numeric, string and alphanumeric data. The data per cell for specific input data is fixed. I need to retrive specific... (5 Replies)
Discussion started by: PratLinux
5 Replies

8. Shell Programming and Scripting

The scope of the shell/perl script is to read the input text file. Validate the expiry date of each

The scope of the shell/perl script is to read the input text file. Validate the expiry date of each certificate and send the mail to the user. The user takes action to add the new certificate to the storage file and user owns the responsibility to update the input text file with the new certificate... (5 Replies)
Discussion started by: casmo
5 Replies

9. Shell Programming and Scripting

Better way to Validate column data in file.

I am trying to validate the third column in a pipe delimited file. The column must be 10 char long and all digits 0-9. I am writing out two new files from the existing file, if it would be quicker, I could leave the bad rows in the file and ignore them in the next process. What I have is... (12 Replies)
Discussion started by: barry1
12 Replies

10. Shell Programming and Scripting

Want to execute rest of the script after the file is ready ...

Hi All I have a requirement like, where a file gets generated in a particular dir and once the file is ready and available then I want to execute rest of the script, because untill and unless the file exists and is available there is no use of running rest of the commands in that script. ... (5 Replies)
Discussion started by: csaha
5 Replies
Login or Register to Ask a Question