Shell Programming and Scripting

I'm reading in some >41,000 line files and doing some manipulations of columns based on the values of other columns. Arrays make a ton of sense for this application.

Not to slow it down too much, I want to spit out the lines efficiently and not have another loop indexing the entries for each line.

For each row

Code:

$echo "${array[@]}

gives space delimiters (and some of the values contain space so piping it through tr won't work)

Code:

IFS=','; echo "${array[@]}

does not change anything, the output is still space delimited (I was not expecting this to work).

Is there time and processor power efficient way to do this without a loop indexing through all the columns?

Mike

PS. topic related to the code

Code:

IFS=','; While read -r -a array
Do a bunch of stuff
echo "${array[@]} >> out_file
done < in_file

is much much slower than

Code:

(IFS=','; While read -r -a array
Do a bunch of stuff
echo "${array[@]}
done ) < in_file > out_file

I suspect this is because the file is only opened and closed once, not 41,000 times. What are the risks of the subshell over-running it's available memory? I'm looking at ~10 mb files for just Q1 2013 data and I still need to port over data from 2010 on. This will have to be deployed to a Cygwin enviroment.

Can you post some example input and what you want the output to be like.

Code:

$ array=(1 2 3 4 5 6 7 8 9 10)
$ echo ${array[@]}
1 2 3 4 5 6 7 8 9 10 but want 1,2,3,4,5,6,7,8,9,10

How about something like this:

Code:

$ array=(1 2 3 4 5 6 7 8 9 10)

$ printf "%s,"  "${array[@]}"
1,2,3,4,5,6,7,8,9,10,

I can use pattern matching to remove the last ',' but is there a way to do that within printf?

Mike

How about this:

Code:

$ array=(1 2 3 4 5 6 7 8 9 10)

$ printf -v var "%s," "${array[@]}"

$ echo "${var%?}"
1,2,3,4,5,6,7,8,9,10

Use "${array[*]}" instead.

Regards,
Alister