File and Data Validation.


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting File and Data Validation.
# 1  
Old 10-16-2010
File and Data Validation.

Hello,
I am working on an interface between a legacy system and an ERP system.The format of the data extracted into the staging folder from the legacy system is a follows.

One control file named ExtractDataControl.txt
Multiple entity files eg R1001.txt, R1002.txt, R1020.txt

The control file has the following information.
a)Entity Name (same as the filename for that entity but without the dot and extension)
b)Number of lines in the Entity extract file.
Pipe is the seperator between a) and b)

Eg of the control file is given below.
R1001|589
R1002|345
R1020|456

I need to validate the following
a) Correct number of entity files are received as mention in the control file.
b) The number of lines receiced in each entity file matches the number of lines mentioned in the control file.

I am having difficulty in reading the control file and then looping through the entity files. Would appreciate any help in this matter.

Forgive my ignorance. I am new to Shell Scripting word.Smilie
Thanks in advance,
Warm Regards,
Savio Saldanha.
# 2  
Old 10-16-2010
try this -spaces & punctuation exactly as entered below.
Code:
while IFS='|' && read fname size
 do
 if [ -f $fname ]  ; then    # does file exist?
   cat $fname | wc -l | read linecount  # not a UUOC; eliminates an extra dummy variable
   if [ $linecount -eq $size ]  ; then   # is file the right size
         echo "$fname is good"
   else
         echo "$fname is there - wrong line count"
   fi
 else
   echo "$fname does not exist"
 fi
done < ExtractDataControl.txt

# 3  
Old 10-17-2010
Code:
#!/usr/bin/env ruby 
files=Hash.new(0)
File.foreach("ExtractDataControl.txt") do |line|
 f,numlines = line.chomp.split("|")
 files[f+".txt"]=numlines.to_i
end
files.each_pair do |x,y|
  if File.exists?(x)
    count= File.read(x).split("\n").size
    if count != y
      print "Wrong size: #{x} should have #{y} lines, found #{count} lines\n"
    end
  end
end


Code:
$ wc -l R1020.txt R1001.txt  R1002.txt
 456 R1020.txt
 589 R1001.txt
 346 R1002.txt

$ cat ExtractDataControl.txt
R1001|589
R1002|345
R1020|456

$ruby test.rb
Wrong size: R1002.txt should have 345 lines, found 346 lines



---------- Post updated at 11:09 PM ---------- Previous update was at 11:05 PM ----------

Quote:
Originally Posted by jim mcnamara
Code:
cat $fname | wc -l | read linecount  # not a UUOC; eliminates an extra dummy variable

well, yes it is UUOCSmilie
Code:
linecount=$(wc -l < "$fname")

# 4  
Old 10-17-2010
Quote:
Originally Posted by jim mcnamara
Code:
   cat $fname | wc -l | read linecount  # not a UUOC; eliminates an extra dummy variable

If I pipe the output of a command into `read variable', why doesn't the output show up in $variable when the read command finishes?
# 5  
Old 10-17-2010
@aia. This is no problem in ksh
Code:
$ bash
$ cat infile | wc -l | read linecount ; echo $linecount

$ ksh
$ cat infile | wc -l | read linecount ; echo $linecount
14
$

# 6  
Old 10-17-2010
Bug Data and File Validation

Many thanks for the replies. I am waiting to reach office on Monday and implement the same.

Appreciate.Smilie

Regards,
Savio Saldanha.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Source data validation

I am using below logic to validate whether i am expecting the correct data from source,if not logic should give which column has error.i am running below logic in linux awk -F, ' NR==1{next} {f=" "} $1!~/^{0,5}$|^$/{f=f?f" emp_id-error":"emp_id-error"} $4!~/^{0,6}$|^$/{f=f?f"... (4 Replies)
Discussion started by: katakamvivek
4 Replies

2. Shell Programming and Scripting

Help With UNIX Shell Scripting For Data Validation

Hi All, I am completely new to Unix Shell Scripting. I m validating(Basic File Validation) an .HHT file in TIBCO. After that i need to do Data Validation through UNIX Shell scripting. Rules in DataValidation: 1.) Need to Check Every field wheather it is a Char or Number?(Fields are... (1 Reply)
Discussion started by: Chaitanya K
1 Replies

3. Shell Programming and Scripting

Data validation engine

Generic Data validator Data file: Name,Sal,Dept ABC,1234,D1 AYX,12356,D2 DHF,345,ED3 123,4565,FGJG Config File: Delimiter-"," Rule1-Name- Rule2-Sal- Rule3-Dept-* Can be used to match any regex including date different format and numbers. (3 Replies)
Discussion started by: dikesm
3 Replies

4. Shell Programming and Scripting

basic data validation

hpux. older version. don't have alot of the newer features in some utilities. How do I verify that a variable starts with the letter A. I would like to make it case insensitive. Also, if I have a variable that has letters and numbers. I want to change all the lower case letters to upper case.... (2 Replies)
Discussion started by: guessingo
2 Replies

5. Shell Programming and Scripting

Data Validation

I have a comma delimited file that I need to validate the data in one two columns in. Sample File: 1234,1234,1234,DESCRIPTION,1,1,2 1234,1234,1234,DESCRIPTION,1,1,2 1234,1234,1234,DESCRIPTION,1,1,2 1234,1234,1234,DESCRIPTION,1,1,2 I need to make sure that the second column's entries are... (3 Replies)
Discussion started by: hmnetonline
3 Replies

6. Shell Programming and Scripting

shell script data & time validation

How to validate a date and optionly a time in shell scripting when i get the date and time as pararmeters that sent out with the call of the file? (in my case sh union.sh `first parameter ,second parameter...` (4 Replies)
Discussion started by: tal
4 Replies

7. Shell Programming and Scripting

validation of data using filter (awk or other that works...) in csv files

Hello People I have the following file.csv: date,string,float,number,boolean 20080303,abc,1.5,123,Y 20080304,abc,1.2,345,N 20080229,nvh,1.4,098,Y 20080319,ugy,1.9,586,N 20080315,gyh,2.4,345,Y 20080316,erf,3.1,932,N I need to filter the date field where I have a data bigger than I... (1 Reply)
Discussion started by: Rafael.Buria
1 Replies

8. UNIX for Dummies Questions & Answers

Data Validation

Hello, I am trying to use data validation with a program. I have everything else working fine. I just can't figure out what I am doing wrong with the data validation in one of my files. Here is the code: # data validation loop while : do # get input from keyboard ... (2 Replies)
Discussion started by: ericelysia
2 Replies

9. UNIX for Advanced & Expert Users

awk data validation

Hi , This is a general doubt.... Is there any way to do data validation inside an awk script.. Let me make it more clear.... I have a string variable inside awk script .. Is there any way to check whether it is number or string etc... Thanks in advance. Shihab (1 Reply)
Discussion started by: shihabvk
1 Replies
Login or Register to Ask a Question