Shell Programming and Scripting

I have delimited-text files ( > 4GB ) and is just one line.


OS: HP-UX 11.23

Awk / cut / sed all have line_max limitations. & unable to read one line in (Buffered-mode).


Sample file:
xxxx|adsfadf|Afdsa|adsf|afds|Asdfas|ads|Afds|Asdf| .....till forever,

I want to put a carriage return every 17 fields.

xxx|asdfad|asdfadsf|adfsadsf
afdsadsf|adfsadsf|Afds|adsf|Asdf


...this has to be in -unbuffered mode..
as i tried to fold -w 500 inputfile > outputfile

then while loop (line-by-line) .. read multiples of 17 fields & append remaing to next line..


see trials below:


#!/usr/bin/sh -x

inputfile=$1
foldfile=${inputfile}.fold
outputfile=${inputfile}.new

i=0
multiplier=17 ### This is 17 Fields to be split



##gawk -F"|" 'OFS="," { for (i=1;i<=NF;i += 17) { print $i, $(i+1 ), $(i+2) , $(i+3) , $(i+4) , $(i+5) , $(i+6) , $(i+7) , $(i+8) , $(i+9) , $(i+10) , $(i+11) , $(i+12) , $(i+13) , $(i+14) , $(i+15) , $(i+16) } fflush(""); } ' $inputfile > $outputfile


fold -w 500 $inputfile > $foldfile

truncated_fields=''

while read line
do

line=`echo $truncated_fields$line`
fields_per_line=`echo $line | awk -F"|" '{print NF}'`



## ex: 69 / 17 = 4 loops + 1 remaining field

fields_multiples_multiplier=`expr ${fields_per_line} / ${multiplier}`
start_index_of_trunc_fields=`expr ${fields_multiples_multiplier} \* ${multiplier} + 1`

if [ $start_index_of_trunc_fields -gt $fields_per_line ]; then
$start_index_of_trunc_fields=$fields_per_line
fi



while [ $i -ne $fields_multiples_multiplier ]

do

start=`expr ${multiplier} \* ${i} + 1 `
end=`expr ${start} + 16 `

echo $line | cut -d"|" -f${start}-${end} >> $outputfile

i=`expr ${i} + 1`


done


truncated_fields=`echo $line | cut -d"|" -f${start_index_of_trunc_fields}-${fields_per_line}`


## echo $fields_multiples_multiplier $start_index_of_trunc_fields $fields_per_line
## echo $truncated_fields
## sleep 10

done < $foldfile

have you tried converting '|' to newLines '\n' and then reconstructing a record/line from the 17 consecutive lines?

Code:

tr '|' '\n' < myFile | nawk 'ORS=(FNR%17)?OFS:RS' OFS='|'

Thank you very much for your help,

But what if the delimiter is --> (1)

exone huge line that does not work with sed awk gawk ...etc

text_a(1)text_b(1)text_c(1)text_d(1)text_e(1)text_c(1)text_d(1)texte(1)


required output is :

text_a(1)text_b(1)text_c(1)text_d(1)
text_e(1)text_c(1)text_d(1)texte(1)

...etc


tr does not work with regexp

tr '\(1\)' '\n' < file

any ideas..

Thanks again for your help

assuming your 'text' fields don't contain '(':

Code:

# with '|' as a separator
tr '(' '\n' < myFile | nawk '{sub("^[0-9][0-9]*[)]", "")}ORS=(FNR%17)?OFS:RS' OFS='|'

OR
assuming your 'text' fields don't contain ')':

Code:

# preserving the original separators
tr ')' '\n' < myFile | nawk 'ORS=(FNR%17)?OFS:")"RS' OFS=')' | sed '$s/)$//'

text_a , text_b & text_c can contain anything .. they contain
( alone
) alone
and also
( ) at the same time.

one question, what's the best approach to tackle such problem..
perl
C under unix.
or what.. ?
because my basic understanding .. that if i fold the (one-line infinite file) to fixed width ..
then loop line-by-line.. it is the same as reading chunk by chunk of (1048 byte or something ).

does the unix re-open the file every line.. (having file handler.. open/shut thus the time is quite extensive ? )

thanks for clarification .

Thanks Issue Resolved..