File validation


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting File validation
# 1  
Old 12-19-2012
Wrench File validation

Hi there,

As a part of file validation, I needed to check for delimiter count in the file.
My aim is to find, how many records have failed to have predefined numbers of delimiters in the file.
My code looks like below

Code:
i=`awk -F '|'  'NF != 2 {print NR, $0} ' ${pinb_fldr}/${pfile}DAT | wc -l | awk '{print $1}'`

echo "number of records with failed delimiter count is $i"

But however, I need to parameterise the delimter count value (in this case 2).
So I am trying to do like this, but it fails..

Code:
dil=2
i=`awk -F '|'  'NF != `echo $dil` {print NR, $0} ' ${pinb_fldr}/${pfile}DAT | wc -l | awk '{print $1}'`
echo "number of records with failed delimiter count is $i"

Appreciate any suggestion in this regard. Thanks .....
# 2  
Old 12-19-2012
Best idea is to pass a variable into awk:

Code:
dil=2
i=`awk -F'|' -v fields=$dil 'NF != fields {print NR, $0} ' ${pinb_fldr}/${pfile}DAT | wc -l | awk '{print $1}'`
echo "number of records with failed delimiter count is $i"


or course awk can count the lines without using wc and another awk

Code:
dil=2
i=`awk -F'|' -v fields=$dil 'NF != fields {miss++} END{print miss}' ${pinb_fldr}/${pfile}DAT`

# 3  
Old 12-19-2012
Thanks Chubler, it made my task a lot easier,

Still have another tricky question.
I have two files for delimter validation, each with differeent delimiter count..

I am tring to call the awk function once in the for loop...

I can pass files names differently in two iterations using "pfile" variable, but how it get its correponding delimter count into the loop.
Any idea or suggestion is highly appreciated.......

Code:
for pfile in ${PSD} ${PSB}

do
i=`awk -F'|' -v fields=$dil 'NF != fields { miss++ } END { print miss }' ${pinb_fldr}/${pfile}DAT`

if [ $i == 0 ]
then 
log_msg "File ${pfile}DAT has correct number of delimiter count.. Proceeding!"
else
ExitIfError 2 "There is/are $i record(s) with incorrect delimiter count in file ${pfile}DAT"
fi
done


Thanks
# 4  
Old 12-19-2012
An array might be the way to go here (expecially if you intend more than 2 files at a later date):

Code:
#!/bin/bash
FILES=( $PSD $PSB )
FIELDS=( 2 7 )
 
for((f=0;f<${#FILES[@]};f++))
do
    i=`awk -F'|' -v fields=${FIELDS[f]} 'NF != fields { miss++ } END { print miss }' ${pinb_fldr}/${FILES[f]}DAT`
 
    if [ $i == 0 ]
    then 
        log_msg "File ${FILES[f]}DAT has correct number of delimiter count.. Proceeding!"
    else
         ExitIfError 2 "There is/are $i record(s) with incorrect delimiter count in file ${FILES[f]}DAT"
    fi
done

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Validation before moving the File

Hi, Before moving the files from source directory to target directory I need to check if the files have these strings 'Error', 'Not Valid' ,' YTF-' or if the file is a zero byte file, if the file contains these strings or if it is a zero byte file i should log a entry in the log file and fail... (5 Replies)
Discussion started by: gaur.deepti
5 Replies

2. Shell Programming and Scripting

Unicode file validation

I don't want HTML_CONTENT,RICH_CONTENT,TEXT_CONTENT columns data in the file and reset of data we need to extract. Find the attached file. Need to extract date in between DI_UX_ROW_END tag. Can help me using unix command using AWK. Thanks, (2 Replies)
Discussion started by: bmk
2 Replies

3. Shell Programming and Scripting

XML file validation

Hi, i am new to unix script. i have two xml files. one is orderinfo.xml and another one is xsd file(EmailMessage.xml). we need to compare the both file for poper nodes exists are not. for example: <xsd:element name="EmailMessage"> tag in the EmailMessage.xml file(xsd sheet),this tag... (2 Replies)
Discussion started by: bmk
2 Replies

4. Shell Programming and Scripting

File validation

Hello, File contains 1,3 and 5 are required columns. it's working use this command. awk -F\| '$1 && $3 && $5' test1.txt > test2.txt How can use this unix programming.while using runnign this scirpt,it's raising the error. awk: ^ syntax error #!/usr/bin/ksh `awk -F\| '$1 &&... (3 Replies)
Discussion started by: bammidi
3 Replies

5. Shell Programming and Scripting

Input validation of file

Hi, I'm trying to do input validation for a file. If the user fails to enter a file name, the script should prompt for the file name or exit. I'm using this script to integrate with a bigger script which does ftp of the files which are "read". #!/bin/ksh echo "enter a file name" if then... (16 Replies)
Discussion started by: sam_bd
16 Replies

6. Shell Programming and Scripting

File Name Validation

Hi All, I am trying to validate the file name against the naming convention mentioned in configuration file. In the configuration file, the file name convention is mentioned as: Myfile_SQ<NN>_<NN>_YYYYMMDD_HHMMSS.xml The actual file received is Myfile_SQ10_30_20110423_073002.xml How do... (1 Reply)
Discussion started by: angshuman
1 Replies

7. UNIX for Dummies Questions & Answers

file name with timestamp validation

Hi , I am trying to write a shell script which will validate all .csv file names in a directory , my file name format is as below. CDR_SCAN_Report_YYYYMMDDHHMI.csv 1. I need to validate the name should start with CDR_SCAN_Report. 2. And the time stamp part is a valid time stamp,if it is... (1 Reply)
Discussion started by: shruthidwh
1 Replies

8. Shell Programming and Scripting

File name and format validation

Hi Gurus, I used unix long time back. I need help for writing a unix script which can be automated and execute every day on specific time. 1.) This is the actual functional requirement. Informatica should reject incoming files that have invalid filenames or file formats 2.) My File... (6 Replies)
Discussion started by: vsmeruga
6 Replies

9. UNIX for Dummies Questions & Answers

File Validation

Hi All, I am new to UNIX scripting. I need to validate a file. I need to check whether the file has a header , detail records and footer. If all the file is good I need to create a status file with the status 'Y' else 'N'. I have pasted the an example of the file below: ... (5 Replies)
Discussion started by: kumar66
5 Replies

10. Shell Programming and Scripting

validation :file <filename>

My user is ftp a CSV file from window. There is possiblity to doing FTP through different mode. This file is then picked by another program, but I want to make sure the FTP file would be always ascii format, bu releasing a commaon File <file name> Can any one provide input , abut how to put a... (2 Replies)
Discussion started by: u263066
2 Replies
Login or Register to Ask a Question