Bash to verify and validate file header and data type
The below bash is a file validation check executed that will verify the correct header count of 10 and the correct data type in each field of the tab-delimited file. The key has the data type of each field in it. My real data has 58 headers in it but only the header and next row need to be checked. The below files are examples that have all possible data types in them. That is the data type of each line after the header is the same as the line above it. All lines will have some sort of data in it, either a numeric, alpha charter or a . (dot) for a null value. If the file is validate a message is written to the output indicated this, else the missing header or bad data type is written to output.
I'm not sure if the below is the best way to do this, but hopefully it is close. Each line is commented as to what I think is happening. Thank you .
There are 3 example files represent each of the only possibilities.
key
file1
file2
file3
Last edited by cmccabe; 04-17-2017 at 09:19 AM..
Reason: added details added red color to file2, corrected syntax errors detected by shell check
Hi all,
Can anyone tell me a little about the datatype FILE, which represents stream. What does its structure look like, and in which header file is it defined and so on...
Ex :
FILE *fp ;
fp = fopen("filename", "w") ; (6 Replies)
I am trying to validate the third column in a pipe delimited file.
The column must be 10 char long and all digits 0-9.
I am writing out two new files from the existing file, if it would be quicker, I could leave the bad rows in the file and ignore them in the next process.
What I have is... (12 Replies)
I am seeking help on this UNIX function, please help. Thanks in advance.
I have a large file, named as 'MyFile'. It was tab-delmited, I am told that each record in column 1 is unique. How would I verify this with UNIX function or command? (1 Reply)
Hello Experts,
I have a script that that transfers a file (via sftp) and it works fine but we ran into a snag where the target server asked for the ssh key and the script didn't know what to do. I want to add some logic to this script that at least sends an email that it didn't complete as... (4 Replies)
Hi,
I need a script that validates a file header/detail/trailer. File layout is:
Header - Rec_Type|File_name|File_Date
Detail - Rec_Type|field1|field2|field3...
Trailder - Rec_Type|File_name|File_Date|Record_count
Sample Data:
HDR|customer_data.dat|20120709... (7 Replies)
Hi All,
File contains header row.. we need to exclude the header row...no need to validate the first row in the file.
Data in the file should take valid data(two columns)..we need to exclude the more than two columns in the file except the first line.
email|firstname
a|123|100
b|345... (4 Replies)
Hi, Great minds, I have some files, in fact header files, of CTD profiler, I tried a lot C programming, could not get output as I was expected, because my programming skills are very poor, finally, joined unix forum with the hope that, I may get what I want, from you people,
Here I have attached... (17 Replies)
Hi,
I am looking for a ready shell script that can help in loading and validating a high volume (around 4 GB) .Dat file . The data in the file has to be validated at each of its column, like the data constraint on each of the data type on each of its 60 columns and also a few other constraints... (2 Replies)
Hi All;
I am struggling to write a script that validates file header.
Header file would be like below with TAB separated
TRX # TYPE REF # Source Piece Code Destination Piece Code
every time I need to check the txt file if the header was same as above fields if validation success... (6 Replies)