Can ksh read records with blank fields


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Can ksh read records with blank fields
# 1  
Old 02-26-2015
Blade Can ksh read records with blank fields

I have a tab delimited file with some fields potentially containing no data. In ksh 'read' though treats multiple tabs as a single delimiter. Is there any way to change that behavior so I could have blank data too? I.e. When encountering 2 tabs it would take it as a null field? Or do I have to use awk?
Code:
# where <TAB> would be a real tab:
while IFS="<TAB>" read a b c d; do echo $c; done < file.txt

vs.
Code:
awk -F"\t" '{print $3}' file.txt

The shell version will output the wrong field if the 1st or 2nd record is blank.
# 2  
Old 02-26-2015
Yes try setting IFS to two consecutive hard tabs:
Code:
while IFS="<TAB><TAB>" read ...

and it is still best to use quotes around the variable expansion:
Code:
echo "$c"


--
Note: in ksh93 you could also use:
Code:
while IFS=$'\t\t' read


Last edited by Scrutinizer; 02-26-2015 at 12:09 PM..
# 3  
Old 02-26-2015
Scrutinizer,
I think the OP is saying the input file has lines with 4 fields using <tab> as a separator and some fields are empty. Unfortunately, when IFS is set to <tab> (or <space>) groups of <tab>s (or <space>s and <tab>s) are treated as a single field separator. I don't believe that this behavior for IFS set to a single <tab> character conforms to the standards, but since this seems to be existing practice in both bash and ksh, this may be considered a bug in the standard.

benalt,
If I understand what you're trying to do, the following should work with ksh (or any shell that performs the parameter expansions required by the POSIX standards):
Code:
#!/bin/ksh
while IFS='' read -r line
do	printf 'input:"%s"\n' "$line"
	d=${line##*	}
	line=${line%	*}
	c=${line##*	}
	line=${line%	*}
	b=${line##*	}
	line=${line%	*}
	a=${line##*	}
	printf 'a=%s,b=%s,c=%s,d=%s\n' "$a" "$b" "$c" "$d"
done < file.txt

(where the whitespace in % * and in * } is a single literal <tab> character in all of the above parameter expansions.

If file.txt contains:
Code:
a	b	all fields present	d
a		field 2 empty	d
	b	field 1 empty	d
		fields 1 & 2 empty	d
		fields 1, 2, & 4 empty	
a	b	c	d \
e	f	g	h

the output produced is:
Code:
input:"a	b	all fields present	d"
a=a,b=b,c=all fields present,d=d
input:"a		field 2 empty	d"
a=a,b=,c=field 2 empty,d=d
input:"	b	field 1 empty	d"
a=,b=b,c=field 1 empty,d=d
input:"		fields 1 & 2 empty	d"
a=,b=,c=fields 1 & 2 empty,d=d
input:"		fields 1, 2, & 4 empty	"
a=,b=,c=fields 1, 2, & 4 empty,d=
input:"a	b	c	d \"
a=a,b=b,c=c,d=d \
input:"e	f	g	h"
a=e,b=f,c=g,d=h

This User Gave Thanks to Don Cragun For This Post:
# 4  
Old 02-26-2015
Hi Don, you are right, I always thought this was part of the POSIX standard, but on rereading it seems to be specific to ksh93 and zsh. It does not work in ksh88.

In man ksh93:
Quote:
In addition, if the same isspace character appears consecutively inside IFS, this character is treated as if it were not in the isspace class, so that if IFS consists of two tab characters, then two adjacent tab characters delimit a null field.
Code:
$ ksh93 -c 'echo "  field3 field4  filed6" | { IFS="  " read a b c d e f g; echo "$d" ;}'
field4
$ zsh -c 'echo "  field3 field4  filed6" | { IFS="  " read a b c d e f g; echo "$d" ;}'
field4

So the answer to the original question is, yes with ksh93, no with ksh88.

Last edited by Scrutinizer; 02-26-2015 at 10:19 PM..
This User Gave Thanks to Scrutinizer For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Print . in blank fields to prevent fields from shifting

The below code works great, kindly provided by @Don Cragun, the lines in bold print the current output. Since some of the fields printed can be blank some of the fields are shifted. I can not seem too add . to the blank fields like in the desired output. Basically, if there is nothing in the field... (10 Replies)
Discussion started by: cmccabe
10 Replies

2. Shell Programming and Scripting

Script to find blank records in a file except for few columns

I have a file with the following format: X|High|2|GIC|DM||XHM|||6 Months X|Moderate|2|GIC|DM||XHM|||6 Months X|High|2|GCM|DM||XSF|||6 Months X|Med|2|GCM|DM||XSF|||6 Here there are ten columns but I need to print rows having blank records in any of the rows (except for 6th,8th and 9th... (10 Replies)
Discussion started by: chatwithsaurav
10 Replies

3. Shell Programming and Scripting

To check Blank Lines, Blank Records and Junk Characters in a File

Hi All Need Help I have a file with the below format (ABC.TXT) : ®¿¿ABCDHEJJSJJ|XCBJSKK01|M|7348974982790 HDFLJDKJSKJ|KJALKSD02|M|7378439274898 KJHSAJKHHJJ|LJDSAJKK03|F|9898982039999 (cont......) I need to write a script where it will check for : blank lines (between rows,before... (6 Replies)
Discussion started by: chatwithsaurav
6 Replies

4. Shell Programming and Scripting

Select records and fields

Hi All I would like to modify a file like this: >antax gioq21 tris notes abcdefghij klmnopqrs >betax gion32 ter notes2 tuvzabcdef ahgskslsooin this: >tris abcdefghij klmnopqrs >ter tuvzabcdef ahgskslsoo So, I would like to remove the first two fields(and output field 3) in record... (4 Replies)
Discussion started by: giuliangiuseppe
4 Replies

5. Shell Programming and Scripting

Count blank fields in every line

Hello All, I am trying a one liner for finding the number of null columns in every line of my flat file. The format of my flat file is like this a|b|c|d||||e|f|g| a|b|c|d||||e|f|g| I want to count the number of fields delimited by "|" which are blank. In above case the count should be... (6 Replies)
Discussion started by: nnani
6 Replies

6. Shell Programming and Scripting

remove blank spaces from fields

Hi Friends, I have large volume of data file as shown below. Beganing or end of each filed, there are some blank spaces. How do I remove those spaces? AAA AAA1 | BBB BB1 BB2 |CC CCCC DDDD DD | EEEEEEE EEEEEEEE | FFF FFFFFF FFFF GG GGGGGG |HH HH ... (3 Replies)
Discussion started by: ppat7046
3 Replies

7. Shell Programming and Scripting

KSH Programming to read and mail fields

I have a file with the following values: File name à a.log (bulk file with 100+ lines with the similar format) aaaa|bbbb|cccc|dddd|eeee|ffff|gggg|hhhh|iiii| aaaa|bbbb|cccc|dddd|eeee|ffff|gggg|hhhh|iiii| aaaa|bbbb|cccc|dddd|eeee|ffff|gggg|hhhh|iiii|... (3 Replies)
Discussion started by: shivacbz
3 Replies

8. UNIX for Dummies Questions & Answers

how to grep for blank records (space,tab,/n)?

I want to exclude (-v) blank records from a file before analysing it. I know I can use '^]$' for spaces and tabs but how do you look for lines that have nothing (/n or line feed) ? (2 Replies)
Discussion started by: Browser_ice
2 Replies

9. Shell Programming and Scripting

how to include field separator if there are blank fields?

Hi, I have the following data in the format as shown (note: there are more than 1 blank spaces between each field and the spaces are not uniform, meaning there can be one blank space between field1 and field2 and 3 spaces between field3 and field4, in this example, # are the spaces in between... (19 Replies)
Discussion started by: ReV
19 Replies

10. UNIX for Dummies Questions & Answers

awk delete blank records

Hello I have a file like this... Name |Sex|Security-Number abx |F |33787728 cdr |M |823483993 derf |F | i would like to use awk to delete all records from the file that has a blank in the the 3 rd feild . the output should be like ... (1 Reply)
Discussion started by: xiamin
1 Replies
Login or Register to Ask a Question