Shell Programming and Scripting

Before I ask my actual question, is it going to be a problem that I want to run this process on a 15 Gig file that is ~140 million rows?

What I'm trying to do:
I have a file that looks like

Code:

Color,Type,Count,Day
Yellow,Full
5
Tuesday
Green,Half
6
Wednesday
Purple,Half
8
Tuesday
...

and my desired output is

Code:

Color,Type,Count,Day
Yellow,Full,5,Tuesday
Green,Half,6,Wednesday
Purple,Half,8,Tuesday
...

Leaving the title row alone, lines 2,3, and 4 should be combined into 1 line seperated by commas, same with lines 5,6, and 7, all the way to the last sets of lines in the file 139,999,997, 139,999,998 and 139,999,999.

Nothing I've tried with sed has allowed me to combine multiple lines into one line, since sed reads everything line by line. Some things I've read make this sound possible, but I have no idea how.

Thanks,
GoldFish

sed reads a line at a time, so the actual number of lines is not a problem as such. If you do complex combining of lines, you might use up a lot of memory, but if your lines aren't particularly long and you don't need to process over long ranges, you should be fine even with a large input file.

You basically want to merge lines with commas until there are three commas, correct?

Code:

sed -e :0 -e '/,.*,.*,/b' -e N -e 's/\n/,/' -e b0 file

There are dialect differences between different sed implementations, but this works for me on Ubuntu Linux. In particular, the \n might be something you need to fiddle to get right.

Code:

 sed '1n;N;N;s/\n/,/g' file
Color,Type,Count,Day
Yellow,Full,5,Tuesday
Green,Half,6,Wednesday
Purple,Half,8,Tuesday

Edit: Decoded a bit
1n; (skip first line)
N;N; (Read line (implied), Read another line, Read another line)
s/\n/,/g (substitute \n with , through to the end of the file)

Thanks both for your solutions. Era, your solution takes a little long though... for my data it'd take a little less than 11 years based on a sample run I did (In my actual data set it has 22 commas in the first line of each set, so had to adjust yours according).

Thanks for the breaking it down too, arvonius!

I've learned a lot from both of these, thanks.