Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

File format check

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers File format check
# 1  
Old 10-16-2008
File format check
How to check if file is in a given format?

For instance: if file records are delimeted with "|" ( pipes) and have exactly 26 fields?

File is pretty big (~3 mil reccords), so not sure if I have to check all records or just head/tail records or smth.

Any ideas are much much more than welcome!
# 2  
Old 10-16-2008
If there is a possibility that some records are bad, you need to read the whole thing. If you are just identifying the format, you can read a few lines.
whole file scan
Code:
awk -F'|'  'BEGIN{ max=0} 
             { if (NF > max ) {max =NF} 
             END{ print "max fileds found=", max}' inputfile

partial:
Code:
head inputfile | awk -F'|'  'BEGIN{ max=0} 
             { if (NF > max ) {max =NF} 
             END{ print "max fileds found=", max}'

# 3  
Old 10-16-2008
Thanks you,

That is definately the right way to do it!!!
# 4  
Old 10-17-2008
What is wrong?
Quote:
> awk -F '|' '(NF!=27){print "Invalid line:", $0}' ../data/CUSTOMER.txt
Invalid line: 2540000069483|2540000069479|||x||x||||M||99991231||N||N|||||||||N |2008-02-02 04:02:52|2008-09-15 05:09:15
Invalid line: 2870000077702|2970000076098|N0010139||x||x||||M||19540610||N||N|||||||||N |2008-06-10 06:06:49|2008-09-15 06:09:20
Invalid line: 2840000235502|2800000229498|N0026317||x|W|x||||M||19490220||N||N|||||||||N |2008-07-27 10:07:30|2008-09-15 06:09:58
.................................................................................................... ..........................

>tmp=27
> awk -F '|' '(NF!=$tmp){print "Invalid line:", $0}' ../data/CUSTOMER.txt
awk: 0602-562 Field $() is not correct.
The input line number is 1. The file is ../data/CUSTOMER.txt.
The source line number is 1.
# 5  
Old 10-19-2008
The single quotes are protecting $tmp from being variable expanded by the shell. Unquote around $tmp:

Code:
awk -F '|' '(NF!='$tmp'){print "Invalid line:", $0}' ../data/CUSTOMER.txt

MrC
# 6  
Old 10-21-2008
Thanks MRC!

Another quick one:
What is the right awk syntax and how to write/redirect records into two files(good and bad):
if (NF!='$temp')
then write to the bad_file.txt
else write to the good_file.txt
fi
# 7  
Old 10-21-2008
awk -F '|' '{if (NF=='$tmp') { print "Good line:", $0 >> "good_file.txt" } else { print "Invalid line:", $0 >> "bad_file.txt" }}'
MrC
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Check if file is EBCDIC or ASCII format

So, i have this requirement where i need to check the file format, whether it's EBCDIC or ASCII, and based on format retrieve the information from that file: my file is: file1.txt-->this ebcdic file file2.txt-->ascii file i tried below code: file=file1.txt type="`file $file`" i get... (7 Replies)
Discussion started by: gnnsprapa
7 Replies

2. Shell Programming and Scripting

Telephone Format and Check

How can I script to check telephone number entered by user in a uniz script: print -n "Enter telephone number to check ? " ; read telno How do I script to : 1) Check if entered without dashes or with dashes 2) If without dashes how can I check: if telno is only 10... (2 Replies)
Discussion started by: mrn6430
2 Replies

3. Shell Programming and Scripting

How to check user entered correct file format or not?

Hi Experts, path=/db/files/ format=$1 User can enter any file format.compare the user file format with actual file format existed in the directory /db/files. User enter all characters as "A" apart from date format. example1: user will be entering the file format AAA_AA_YYYYMMDD.AAA Actual... (6 Replies)
Discussion started by: nalu
6 Replies

4. Shell Programming and Scripting

Script to check dos format in a file

Hi All, I am trying to check if the file is in dos format using simple grep command but the problem is lines inside the file with have special characters in between and in some lines end of the line will have the '^M' character. I tried the below command in simple line(without special... (7 Replies)
Discussion started by: Optimus81
7 Replies

5. Shell Programming and Scripting

finding date numeral from file and check the validity of date format

hi there I have file names in different format as below triss_20111117_fxcb.csv triss_fxcb_20111117.csv xpnl_hypo_reu_miplvdone_11172011.csv xpnl_hypo_reu_miplvdone_11-17-2011.csv xpnl_hypo_reu_miplvdone_20111117.csv xpnl_hypo_reu_miplvdone_20111117xfb.csv... (10 Replies)
Discussion started by: manas_ranjan
10 Replies

6. Shell Programming and Scripting

Need to check date format yyyymm

My source file having one date column. The formate of the date column is yyyymm. I need to validate whether all the rows are in same format in the given file. If it is not I have captured that records in a separate file. I am very new to Unix. I don't how to achieve this. Plz help me to achieve... (2 Replies)
Discussion started by: suresh01_apk
2 Replies

7. Shell Programming and Scripting

How to check file name format using shell script?

Hi, I am writting a script, which accepts input file as parameter. Input file name is aa_bb_cc_dd_ee.<ext> I need to check that input file name should be of 5 fileds. Please help me out. :confused: (7 Replies)
Discussion started by: Poonamol
7 Replies

8. Shell Programming and Scripting

How to check for file name of specific format using find?

I have to find the specific formatted file is present in the received list in the directory, for which I have written: file_list=`ls -lrt /tmp/vinay/act/files |grep "$cdate"| awk '{print $9}'` while read fileStr do find $file_list $fileStr > /dev/null status=`echo $?` if ; then ... (3 Replies)
Discussion started by: IND123
3 Replies

9. Shell Programming and Scripting

Check the format of Date

Hi, I have a date field in my input file. I just want to check if its in the format "DD-MM-YYYY". Is there any command which can achieve this? Thanks and Regards, Abhishek (2 Replies)
Discussion started by: AAA
2 Replies

10. Shell Programming and Scripting

Check whether a given file is in ASCII format and data is tab-delimited

Hi All, Please help me out with a script which checks whether a given file say abc.txt is in ASCII format and data is tab-delimited. If the condition doesn't satisfy then it should generate error code "100" for file not in ASCII format and "105" if it is not in tab-delimited format. If the... (9 Replies)
Discussion started by: Mandab
9 Replies
Login or Register to Ask a Question