Shell script for validating fields in a file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Shell script for validating fields in a file
# 1  
Old 01-28-2010
Shell script for validating fields in a file

Hi,

I have not used Unix in a very long time and I am very rusty. I would appreciate any help I can get from the more experienced and experts in Shell script.

I am reading one file at a time from a folder. The file is a flat file with no delimeters or carriage return. Col1 through col6 is the header section and should be read once. Col7 - 14 is the body and can have 1 or multiple records. I read the file check to make sure none of these fields are blank. If there is a blank field, I would like to write a message to a log file that can be viewed later. I haven't gotten to the log file yet. Just trying to read the file with an inner loop just doesn't seem to work. Please help me.

Code:
 
#!/bin/ksh
 
#This script was created on 01/27/2010 to check outbound
# files for missing required fields
###########################################################
 
 
for file in /folder/file.1$.5$
 
do
 
i=1
 
exec< file
 
while read line
do
 
col1=`echo $line | cut -c1-4`
col2= `echo $line | cut -c5-5`
col3= `echo $line | cut -c6-7`
col4= `echo $line | cut -c8-16`
col5= `echo $line | cut -c17-23`
col6= `echo $line | cut -c51-56`
 
if [ -z "$col1" ]
then
echo "Line No. $i -- No String in position 1-4 "
else
echo "Line No. $i -- String in position 1-4 : $col1"
fi
 
if [ -z "$col2" ]
then
echo "Line No. $i -- No String in position 5 "
else
echo "Line No. $i -- String in position 5 : col2"
fi
 
if [ -z "$col3" ]
then
echo "Line No. $i -- No String in position 6-7 "
else
echo "Line No. $i -- String in position 6-7 : $col3"
fi
 
if [ -z "$col4" ]
then
echo "Line No. $i -- No String in position 8-16 "
else
echo "Line No. $i -- String in position 8-16 : $col4"
fi
 
if [ -z "$col5" ]
then
echo "Line No. $i -- No String in position 17-23 "
else
echo "Line No. $i -- String in position 17-23: $col5"
fi
 
if [ -z "$col6" ]
then
echo "Line No. $i -- No String in position 51-56 "
else
echo "Line No. $i -- String in position 51-56 : $col6"
fi
 
i=`expr $i + 1`
 
exec< file
 
 While read line
    do
 
        col7=`echo $line | cut -c66-72`
        col8= `echo $line | cut -c73-74`
        col9= `echo $line | cut -c75-83`
        col10= `echo $line | cut -c84-85`
        col11= `echo $line | cut -c86-88`
        col12= `echo $line | cut -c89-96`
        col13= `echo $line | cut -c97-100`
        col14= `echo $line | cut -c108-115`
 
 
        if [ -z "$col7"] or [ -z "$col8"] or [ -z "$col9"] or [ -z "$col10"]
          or[ -z "$col11" ] or [ -z "$col12" ] or [ -z "$col13"] or [ -z "$col14"]
        then
        echo "Line No. $i -- Missing String "
        else
        echo "Line No. $i -- String is valid"
        fi
 
        i=`expr $i + 1`
        done
done
 
done


Thank you all for your help.

Last edited by jim mcnamara; 01-28-2010 at 11:31 AM.. Reason: corrected code tags
# 2  
Old 01-28-2010
Remove space between = and `
Code:
col2= `echo $line | cut -c5-5`

If you dont have data for a column, then do you have blank spaces?

Can you show us input?
# 3  
Old 01-28-2010
Thanks for responding. I appreciate it.

Yes if there are no data, it will show up as blanks

1111B0216262626111111 999999 22222227823876489050456201001222010 20101031


The date above will represent one record. So a second record will be

1111B0216262626111111 245367 33333338860870499050456201001182010 20100930

looking at the data again, I changed the code to . Col1 through col14 are in every record in the file.

Code:
 
for file in /folder/file.1$.5$
 
do
 
i=1
 
exec< file
 
while read line
do
 
col1=`echo $line | cut -c1-4`
col2= `echo $line | cut -c5-5`
col3= `echo $line | cut -c6-7`
col4= `echo $line | cut -c8-16`
col5= `echo $line | cut -c17-23`
col6= `echo $line | cut -c51-56`
 col7=`echo $line | cut -c66-72`
 col8= `echo $line | cut -c73-74`
 col9= `echo $line | cut -c75-83`
 col10= `echo $line | cut -c84-85`
 col11= `echo $line | cut -c86-88`
 col12= `echo $line | cut -c89-96`
 col13= `echo $line | cut -c97-100`
 col14= `echo $line | cut -c108-115`
 
if [ -z "$col1" ]
then
echo "Line No. $i -- No String in position 1-4 "
else
echo "Line No. $i -- String in position 1-4 : $col1"
fi
 
if [ -z "$col2" ]
then
echo "Line No. $i -- No String in position 5 "
else
echo "Line No. $i -- String in position 5 : col2"
fi
 
if [ -z "$col3" ]
then
echo "Line No. $i -- No String in position 6-7 "
else
echo "Line No. $i -- String in position 6-7 : $col3"
fi
 
if [ -z "$col4" ]
then
echo "Line No. $i -- No String in position 8-16 "
else
echo "Line No. $i -- String in position 8-16 : $col4"
fi
 
if [ -z "$col5" ]
then
echo "Line No. $i -- No String in position 17-23 "
else
echo "Line No. $i -- String in position 17-23: $col5"
fi
 
if [ -z "$col6" ]
then
echo "Line No. $i -- No String in position 51-56 "
else
echo "Line No. $i -- String in position 51-56 : $col6"
fi

 if [ -z "$col7"] or [ -z "$col8"] or [ -z "$col9"] or [ -z "$col10"]
 or[ -z "$col11" ] or [ -z "$col12" ] or [ -z "$col13"] or [ -z "$col14"]
 then
 echo "Line No. $i -- Missing String "
 else
 echo "Line No. $i -- String is valid"
 fi

 
i=`expr $i + 1`


done
 
done

Also instead of using all these if statements, can I use a case? if yes how do I do it?

Thanks in advance for your help.

Last edited by Scott; 01-28-2010 at 01:50 PM..
# 4  
Old 01-28-2010
If you run that code over dozens of big files it will take forever. Every one of those backtick lines creates a separate process.

awk (I used nawk, same thing ) was meant for stuff like this.
create a file: pos.txt that has the offsets
Code:
1     4
5     5
6     7
8    16
17   23
51   56     
66   72             
73   74    
75   83    
84   85  
86   88  
89   96  
97  100 
108 115

Code:
for infile in /folder/file.1$.5$   
do

  nawk -v infile=$infile  ' {
      if(FILENAME=="pos.txt" )
      { 
           pos[FNR]=$1; len[FNR]=($2-$1)+1; vals++; next 
      }
      if(FILENAME==infile )
      {         
           for(i=1; i<=vals; i++)
           {
             testval=substr($0,pos[i], len[i])
             gsub(/ /, "", testval)
             if(length(testval)==0) {print infile, "line:", FNR, " blank field:", i}
           }  
           next         
      } 
  } '  pos.txt  $infile

done > report.txt

# 5  
Old 01-29-2010
Thanks Jim.

Now I am having problems opening the file

/folder/file.1$.5$

I have multiple files in this folder with the following names

file.10.5232004
file.12.2112003
.........

What do I do?

Thanks for your help again.

---------- Post updated at 03:40 PM ---------- Previous update was at 02:10 PM ----------

Thanks I got the code to work.

The problem I am having now is that it is repeating the same thing over and over again in the report.txt file.

If I wanted to print out only the file name, number of records and the length of each record for each file, how do I do this? with the code below

Code:
 
{print infile, "line:", FNR, " blank field:", i}

Thank you

---------- Post updated 01-29-10 at 12:47 AM ---------- Previous update was 01-28-10 at 03:40 PM ----------

Thank you!! I was able to find a solution in the forumn

---------- Post updated at 03:26 PM ---------- Previous update was at 12:47 AM ----------

I am having a hard time extracting the file name from the above code. In stead of printing /folder/file.1$.5$, I would like it to print the file name file.1$.5$.

I have tried using basename but it looks like NAWK or AWK does not recognise basename. Each time I type it in, it prints out the word basename

Code:
{print basename infile, "line:", FNR, " blank field:", i}

Thank you for your help!!!

---------- Post updated at 03:27 PM ---------- Previous update was at 03:26 PM ----------

I am having a hard time extracting the file name from the above code. In stead of printing /folder/file.1$.5$, I would like it to print the file name file.1$.5$.

I have tried using basename but it looks like NAWK or AWK does not recognise basename. Each time I type it in, it prints out the word basename



Code:
{print basename infile, "line:", FNR, " blank field:", i}

Thank you for your help!!!
# 6  
Old 01-30-2010
hello everybody

i have almost the same problem with my files (i am on AIX 5.3), i am gonna try this solution
The main difference that in my case i don't have header but also i can't use "pos.txt"
because i have offest like that
Code:
 
1     2
3     5
6     7
8    10

11   12
13   15
16   17     
18   20             
....
121   122    
123   125    
126   127  
128   130  
....

so is there another way to fit for my purpose

thx jim for your great work and thx asemota for asking so i could fond a solution
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell script to pull certain fields

I/m a beginner so be easy. I have text files that live on an AIX server. The files come in and I've been charged with writing a shell script to email out that pulls the first date, and the last date of the file. I need to load these 2 dates into 2 separate variables. I can figure out the variables,... (13 Replies)
Discussion started by: mattadams1983
13 Replies

2. Shell Programming and Scripting

Shell Script to Group by Based on Multiple Fields in a file

Hi, I want to know if there is any simple approach to SUM a field based on group by of different fields for e.g. file1.txt contains below data 20160622|XXX1||50.00||50.00|MONEY|Plan1| 20160622|XXX1||100.00||100.00|MONEY|Plan1| 20160623|XXX1||25.00||25.00|MONEY|Plan1|... (3 Replies)
Discussion started by: cnu_theprince
3 Replies

3. Shell Programming and Scripting

Validating the file

Hello All, I have the following file. The first column is Type. Always the file will have one H and one T type in between all D type reocrds. Need todo some validations. H|ABCD D|TAB N0003107809CD2013-11-14|RYAN|FRY|7 DR|RICHMOND HILL|GA|32431|X|C95|000009999|000000001|TAB||C0001 D|TAB... (3 Replies)
Discussion started by: karumudi7
3 Replies

4. Shell Programming and Scripting

How to get fields and get output with awk or shell script.?

I have a flat file A.txt with field seperate by a pipe 2012/11/13 20:06:11 | 284:hawk pid=014268 opened Locations 12, 13, 14, 15 for /home/hawk_t112/t112/macteam/qt/NET12/full_ddr3_2X_FV_4BD_1.qt/dbFiles/t112.proto|2012/11/14 15:19:26 | still running |norway|norway 2012/11/14 12:53:51 | ... (6 Replies)
Discussion started by: sabercats
6 Replies

5. Shell Programming and Scripting

Aligning output with null fields in shell script

Hello Gurus ! I have what probably amounts to a few simply changes to fix; however for the life of me I cannot seem to get it ti work. I need to align the output of my script (I am writing to a logfile)... here's the lines in my code: if then echo "NODE: $node" >> $logfile... (6 Replies)
Discussion started by: gvolpini
6 Replies

6. Shell Programming and Scripting

validating(pingable or not) remote ip address in shell script

i need to verify whether the ip adress given as input to the shell script is pingable or not... that is whether the ip is alive and responding.. ping $ip_adress the above wont work in script because the execution is continuous... so the shell script keeps will dwell in this pinging process...... (8 Replies)
Discussion started by: vivek d r
8 Replies

7. UNIX for Dummies Questions & Answers

Validating a file

Pardon my ignorance but I am lost on how to do this I have a file called "Sample.txt", it is pipe delimited, it should have 13 fields. But some of the records do not, I would like to set up a shell script where I can pass in a parameter "Sample.txt" and it would split the file into records... (2 Replies)
Discussion started by: dgeehot
2 Replies

8. Shell Programming and Scripting

Validating month value in shell script

Hi, the below script is to get & check the correct values for minutes (0-59) that is inputed by user : printf "$FBOLD\nPlease enter the minutes (0-59): $FREG" read MIN case "$MIN" in |) break 2;; *) echo "" echo "Invalid minutes, please... (4 Replies)
Discussion started by: milink
4 Replies

9. UNIX for Dummies Questions & Answers

Validating input based on fixed number of fields

Yes, i did... let me state my problem in more detail Inputs: I have one input CSV file And, i have stored no. of comma each line should in a variable. e.g. $ cat cmt.csv this, is a ,comma ,count test1 ,,this, is a ,comma ,count test2 this, is a ,comma ,count test3... (6 Replies)
Discussion started by: Dipali
6 Replies

10. Shell Programming and Scripting

Validating variables in shells script

All Can you help me to validate a variable only for string and digit. That is variable should either fully alphabets or digits. Please send me result to my mail id also: REMOVED Thanx in advance Regards Deepak Xavier (1 Reply)
Discussion started by: DeepakXavier
1 Replies
Login or Register to Ask a Question