Shell Programming and Scripting

Hi, I'm trying to break a large csv file into smaller files and use unique values for the file names. The shell script i'm using is tcsh and i'm after a gawk one-liner to get the desired outcome. To keep things simple I have the following example with the desired output.

fruitlist.csv
apples 20 40
pears 22 45
grapes 24 50
plums 26 55
peaches 28 60
apples 30 65
pears 32 70
grapes 34 75
plums 36 80
peaches 38 85
apples 40 90
pears 42 95
grapes 44 100
plums 46 105
peaches 48 110

uniquefruit.csv
apples
pears
grapes
plums
peaches

What I want is for gawk to print the rows containing apples in the fruitlist.csv file and redirect to a file called apples.csv. Similarly for pears I want all rows in fruitlist.csv containing pears to be redirected to pears.csv etc etc. Note: Each time I generate a fruitlist.csv file the unique values will be different. For this reason I have generated a uniquefruit.csv file and tried using it within a loop as follows:

Code:

 
#!/bin/tcsh -f
set ugs = `cat uniquefruit.csv`
foreach ug ($ugs)
  gawk -v fruit=$ug '{if(fruit ~ $1); print $0 >> fruit".csv"}' fruitlist.csv
end

I also tried the code:

Code:

 
gawk -F"," '{while ((getline fruit < uniquefruit.csv) > 0); {if(fruit ~ $1) print $0 >> fruit".csv"}' fruitlist.csv

In both cases I got the same result. The desired file names were generated ok. However, all the rows within fruitlist.csv were appended 5 times to each of the breakdown files (apples.csv, pears.csv etc etc.). Something appears to be going wrong when I assign the strings in uniquefruit.csv to the variable fruit.
Any help would be greatly appreciated,
Regards, theflamingmoe.

how about this?

Code:

awk '{print >$1".csv"}' fruitlist.csv

Thanks for the reply yinyuemi. I will try it at work tomorrow!!!
Regards,
Theflamingmoe

---------- Post updated 22-03-11 at 10:25 AM ---------- Previous update was 21-03-11 at 07:34 PM ----------

It still doesn't work. I ended up with 16 csv files. I only want 5 .csv files ie. apples.csv, pears.csv, grapes.csv, plums.csv and peaches.csv. Any rows that contain apples ie (apples, 20, 40), (apples, 30, 65), and (apples, 40, 90) I want redirected to the apples.csv file. The same for the other fruit unique to field 1 ($1). Do I need a BEGIN pattern before gawk is processed?

Cheers,
Theflamingmoe

---------- Post updated at 10:52 AM ---------- Previous update was at 10:25 AM ----------

For anyone interested I got it to work with the following code:

Code:

 
set ugs = `cat uniquefruit.csv`
foreach ug ($ugs)
gawk -v fruit=$ug '$1 ~ fruit {print $0 >> fruit".csv"}' fruitlist.csv 
end

Cheers,
Theflamingmoe

Hi Theflamingmoe,

the following is my testing results, are these what you want?

Best,

Y

Code:

awk '{print >$1".csv"}' fruitlist.csv

ls *.csv
apples.csv  fruitlist.csv  grapes.csv  peaches.csv  pears.csv  plums.csv

cat apples.csv
apples 20 40
apples 30 65
apples 40 90

cat grapes.csv
grapes 24 50
grapes 34 75
grapes 44 100

Hi, Theflamingmoe.

Your solution will pass over the file once for each unique key you have. For a short file that's likely not a problem. For a large file, it will be time-consuming.

The idea of yinyuemi worked for me. I used a slightly different filename scheme because I like suffixes to be created with the key:

Code:

gawk '{print  >> "body."$1}' input-filename

and it produced:

Code:

body.apples  body.grapes  body.peaches	body.pears  body.plums

with a sample -- from the grapes file being:

Code:

grapes 24 50
grapes 34 75
grapes 44 100

I used:

Code:

gawk GNU Awk 3.1.5

Best wishes ... cheers, drl

I think because it is a .CSV file you need modifications as below:-

Code:

awk -F"," '{print >> $1".csv"}' OFS="," fruitlist.csv

Thanks for the replies guys. When I first tried Yinyuemi's code using gawk rather than awk, I got 16 different csv files. Is this because I needed to set both the OFS variable and the field separator to a comma as suggested by Ahmad.diab?? Thanks again for the comments guys......it's been a great help )