How to screen each variable record for right # of fields


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to screen each variable record for right # of fields
# 1  
Old 04-26-2005
How to screen each variable record for right # of fields

Hi,
In a script I have to check that input text files with a variable number of tab delimited fields have at least n fields and no more than m fields. Records are delimited by <CR> and <LF>.

I have figured out code that will strip out all the alpha-numeric characters, convert the tabs to hyphens and then with each record consisting of only hyphens, I could pipe the output to awk to make sure that for each line the number of hyphens was correct as follows:

cat ${FILENAME} | tr '[:print:]' '[\0*]' | tr '[\t]' '[-]' | awk '/-{3,6}/'

The first part, cat ${FILENAME} | tr '[:print:]' '[\0*]' | tr '[\t]' '[-]' , seems to work, producing the following output, which is correct:
----------
--
----
-----
----------
----------
----------
Once I pipe this into awk though, I get no output at all, so not sure what I'm doing wrong. Please help - thanks!! -j

Last edited by Perderabo; 04-26-2005 at 11:51 PM.. Reason: Disable smilies for readability
# 2  
Old 04-26-2005
Are these file DOS (Windows) files? dos2ux or dos2unix (check your manpage)
will take care of that problem:
Code:
dos2unix $filename |
awk -F"\t" {
                   if(NF>6 || NF<3)
                   {
                          print "bad line ", $0
                    }
              } '

# 3  
Old 04-27-2005
Thanks much for the code suggestion - this looks totally straightforward though I'm having trouble getting it to work. I also tried several variations of it, omitting curly brackets and the 'if' - none work - not sure what I'm doing wrong.
Here's the latest version. It seems to partially work as follows (omitted the 'if' and outer curly brackets - not sure if that's the right thing to do). When is the 'if' needed and when is it not ???
awk -F"\t" ' (NF >= 6 || NF <= 3)
{
print NF $1 $2 $3
} ' < ${FILENAME}

The problem is that the code outputs every record in the file (plus some extra output lines) with the filter criteria determining what is printed: the output is right for lines/records that meet the filter criteria. For records in which NF does not meet the criteria, this prints out the whole record, not just NF $1 $2 $3.

Here's the original test file data and the output from code above:
101 111111 N
245 22 N STRING1
333 373323 N STRING1 STRINGSTRING2 A3 C3
189 64399 Y STRING1 STRINGSTRING2 A5 B0 C9 E4
555 7 N STRING1 STRINGSTRING2

Output:
101 111111 N
6101111111N
524522N
333 373323 N STRING1 STRINGSTRING2 A3 C3
10333373323N
189 64399 Y STRING1 STRINGSTRING2 A5 B0 C9 E4
918964399Y
55557N

Thanks for any and all help! -j
# 4  
Old 04-27-2005
Got it to work using basically your original code:

awk -F"\t" ' { if (NF >= 6 || NF <= 3)
{
print NF "--" $1 $2 $3
}
} ' ${FILENAME}

THANKS!

Last edited by jvander; 04-27-2005 at 12:00 PM.. Reason: now working
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk - compare 1st 15 fields of record with 20 fields

I'm trying to compare 2 files for differences in a selct number of fields. When differnces are found it will write the whole record of the second file including appending '|C' out to a delta file. Each record will have 20 fields, but only want to do comparison of 1st 15 fields. The 1st field of... (7 Replies)
Discussion started by: sljnk
7 Replies

2. Shell Programming and Scripting

Delete last 2 fields from every record in a file

Sample file record : "20130617003","2013-06-18T07:00:03","OUTWARD","01001011","TEST PLC","","HFX834346364364","20130617","10","DUM87534758","","1.28","826","020201","65879278","","","","","","010101","56789","DUMMY... (3 Replies)
Discussion started by: bigbuk
3 Replies

3. Shell Programming and Scripting

Newline between unequal record fields

Assume the following 5 records (field separator is a space): 0903 0903 0910 0910 0910 0910 0910 0910 0917 0917 0917 0917 0924 1001 1001 1001 1001 1008 1008 1008 1008 1015 1015 1015 1015 1022 1029 1029 1029 1029 1105 1105 1105 1105 1112 1112 1112 1112 1119 1126 1126 1126 1126 1203 1203 1203 1203... (8 Replies)
Discussion started by: tree
8 Replies

4. Shell Programming and Scripting

Compare fields and keep record with bigger ID?

How do you write a shell script to compare records with the same fields then keep the biggeer id number fields (field separate by a pipe) 1150| San Jose|8|15|7|2013-02-19 00:00:00.000|2013-02-20 00:00:00.000 1263|San Jose|8|15|7|2013-02-19 00:00:00.000|2013-02-20 00:00:00.000... (4 Replies)
Discussion started by: sabercats
4 Replies

5. Shell Programming and Scripting

Print all the fields of record using awk

Hi, i want to generate print statement using awk. i have 20+ and 30+ fields in each line Now its priting only first eight fields print statement as output not all. my record is as shown below filename ... (2 Replies)
Discussion started by: raghavendra.nsn
2 Replies

6. Shell Programming and Scripting

Making changes in the fields of a record

:confused: Hi Friends, In the record below i have to make changes in the fields by putting the values stored in the temporary variables, x, y, z, p, q, r: 2) In the TBT record store the values in the various fields as: a) X in a field position 51 to 56 b) Y... (5 Replies)
Discussion started by: kanu_pathak
5 Replies

7. UNIX for Dummies Questions & Answers

change order of fields in header record

Hello, after 9 months of archiving 1000 files, now, i need to change the order of fields in the header record. some very large, space padded files. HEADERCAS05212008D0210DOMEST01(spacepadded to record length 210) must now be 05212008HEADERCASD0210DOMEST01(spacepadded to record length 210) ... (1 Reply)
Discussion started by: JohnMario
1 Replies

8. Shell Programming and Scripting

How count number of fields in a record

Dear All , I have the query cat temp.txt |28-07-1997|IF_LEG_DCCT|TOV JV sdfsdfdsfdsfdsCLOSED* KIEV|381015280 I need to count the number of fields in this pipe-seperated file. I beleive this is possible via AWK command. The in above file, output of the count should be 5.... Can some-one... (5 Replies)
Discussion started by: sureshg_sampat
5 Replies

9. Shell Programming and Scripting

Manipulating fields record wise

Hi all, I have an input file with no delimiter. Let us say the file is abc.txt having values for fields namely, EmpNumEnameDesigSalDept. Ofcourse the file has got several records. Every field has got a fixed start and end position. I need to assign the fields to corresponding varibles say... (1 Reply)
Discussion started by: rinku11
1 Replies

10. Shell Programming and Scripting

awk: record has too many fields

Hi, I'm trying this command - but get this error. Do you guys have any workaround for this? cat tf|sed 's/{//g'|sed 's/,//g'|awk '{for (i=1;i<=NF;i++) {if ($i == "OPTIME") {k = i + 2; print $i,$k}}}' awk: record `2005 Jul 28 17:35:29...' has too many fields record number 15 This is how... (3 Replies)
Discussion started by: chaandana
3 Replies
Login or Register to Ask a Question