Shell Programming and Scripting

I have a text file that contains 4 million lines, each line contains 2 fields(colon as field separator). as shown:

Code:

123:444,555,666,777,888,345
233:5444,555,666,777,888,345
623:454,585,664,773,888,345
......

Here I have to split the second field(can be up to 40,000 fields) by comma into an array for analysis, but I find the "split" function is too slow.

I tried to find an alternative to replacing the split function. I think I found one but there's still something that can not be achieved without your help. now I have this code:

Code:

awk -F: -vcmd="awk ' BEGIN { RS=\",\"} { print $1 }'" '{ print $2 | cmd; close(cmd)} ' data.txt

This code has "split"ed the second field fast enough, but I don't know how to store the splitted data in an array in the *first* AWK program as shown above.

How to solve this problem? or you have other alternatives to replacing the split function?

Thanks.

Maybe, as the file is so big, it will be better to achieve the problem through a C program....

Not sure if this is what you want but it's weird to call awk with awk like that.
You can split the 2nd field like this:

Code:

awk -F: '{
  split($2,a,",")
  # Do your stuff with a[1], a[2] ...
}' file

I used the split function, but it was too slow, and that was why I raised this thread.

What are you trying to achieve? What is the desired output from your input file?

Have you tried something with the split function above?

What is your desired final output? You are not using awk to your advantage.

Calling the external command will be to slow (4,000,000 x 40,000 = 160,000,000,000 calls).
Please answer Franklin52 previous question.