Split file by data group


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Split file by data group
# 8  
Old 11-17-2009
They only have to read the forums to see how rubbish my sed is... they won't suspect a thing Smilie
# 9  
Old 11-21-2009
Okay, finally got back to finishing this off, I hacked it up pretty ugly but it works as intended so here it goes...

Example Input:

Code:
$ cat ./scripts/tests/beers.dat
1. LIGHT LAGER
1A. Lite American Lager
Aroma: Little to no malt aroma, although it can be grainy, sweet or corn-like if present.  Hop aroma may range from none to a light, spicy or floral hop presence.  Low levels of yeast character (green apples, DMS, or fruitiness) are optional but acceptable.  No diacetyl.
Appearance: Very pale straw to pale yellow color.  White, frothy head seldom persists.  Very clear.
Flavor: Crisp and dry flavor with some low levels of grainy or corn-like sweetness.  Hop flavor ranges from none to low levels.  Hop bitterness at low level.  Balance may vary from slightly malty to slightly bitter, but is relatively close to even.  High levels of carbonation may provide a slight acidity or dry "sting."  No diacetyl.  No fruitiness.
Mouthfeel: Very light body from use of a high percentage of adjuncts such as rice or corn.  Very highly carbonated with slight carbonic bite on the tongue.  May seem watery.
Overall Impression: Very refreshing and thirst quenching.
Comments: A lower gravity and lower calorie beer than standard international lagers.  Strong flavors are a fault. Designed to appeal to the broadest range of the general public as possible.
Ingredients: Two- or six-row barley with high percentage (up to 40%) of rice or corn as adjuncts.
Vital Statistics:    OG:    1.028 - 1.040    IBUs:    8 - 12    FG:    0.998 - 1.008    SRM:    2 - 3    ABV:    2.8 - 4.2%
Commercial Examples: Bitburger Light, Sam Adams Light, Heineken Premium Light, Miller Lite, Bud Light, Coors Light, Baltika #1 Light, Old Milwaukee Light, Amstel Light

1B. Standard American Lager
Aroma: Little to no malt aroma, although it can be grainy, sweet or corn-like if present.  Hop aroma may range from none to a light, spicy or floral hop presence.  Low levels of yeast character (green apples, DMS, or fruitiness) are optional but acceptable.  No diacetyl.
Appearance: Very pale straw to medium yellow color.  White, frothy head seldom persists.  Very clear.
Flavor: Crisp and dry flavor with some low levels of grainy or corn-like sweetness.  Hop flavor ranges from none to low levels.  Hop bitterness at low to medium-low level.  Balance may vary from slightly malty to slightly bitter, but is relatively close to even.  High levels of carbonation may provide a slight acidity or dry "sting."  No diacetyl.  No fruitiness.
Mouthfeel: Light body from use of a high percentage of adjuncts such as rice or corn.  Very highly carbonated with slight carbonic bite on the tongue.
Overall Impression: Very refreshing and thirst quenching.  
Comments: Strong flavors are a fault.  An international style including the standard mass-market lager from most countries.
Ingredients: Two- or six-row barley with high percentage (up to 40%) of rice or corn as adjuncts.
Vital Statistics:    OG:    1.040 - 1.050    IBUs:    8 - 15    FG:    1.004 - 1.010    SRM:    2 - 4    ABV:    4.2 - 5.3%
Commercial Examples: Pabst Blue Ribbon, Miller High Life, Budweiser, Baltika #3 Classic, Kirin Lager, Grain Belt Premium Lager, Molson Golden, Labatt Blue, Coors Original, Foster's Lager

14. INDIA PALE ALE (IPA)
14A. English IPA
Aroma: A moderate to moderately high hop aroma of floral, earthy or fruity nature is typical, although the intensity of hop character is usually lower than American versions.  A slightly grassy dry-hop aroma is acceptable, but not required.  A moderate caramel-like or toasty malt presence is common.  Low to moderate fruitiness, either from esters or hops, can be present.  Some versions may have a sulfury note, although this character is not mandatory.
Appearance: Color ranges from golden amber to light copper, but most are pale to medium amber with an orange-ish tint.  Should be clear, although unfiltered dry-hopped versions may be a bit hazy.  Good head stand with off-white color should persist.
Flavor: Hop flavor is medium to high, with a moderate to assertive hop bitterness.  The hop flavor should be similar to the aroma (floral, earthy, fruity, and/or slightly grassy).  Malt flavor should be medium-low to medium-high, but should be noticeable, pleasant, and support the hop aspect.  The malt should show an English character and be somewhat bready, biscuit-like, toasty, toffee-like and/or caramelly.  Despite the substantial hop character typical of these beers, sufficient malt flavor, body and complexity to support the hops will provide the best balance. Very low levels of diacetyl are acceptable, and fruitiness from the fermentation or hops adds to the overall complexity.  Finish is medium to dry, and bitterness may linger into the aftertaste but should not be harsh.  If high sulfate water is used, a distinctively minerally, dry finish, some sulfur flavor, and a lingering bitterness are usually present.  Some clean alcohol flavor can be noted in stronger versions.  Oak is inappropriate in this style.
Mouthfeel: Smooth, medium-light to medium-bodied mouthfeel without hop-derived astringency, although moderate to medium-high carbonation can combine to render an overall dry sensation in the presence of malt sweetness.  Some smooth alcohol warming can and should be sensed in stronger (but not all) versions.  
Overall Impression: A hoppy, moderately strong pale ale that features characteristics consistent with the use of English malt, hops and yeast.  Has less hop character and a more pronounced malt flavor than American versions.
History: Brewed to survive the voyage from England to India.  The temperature extremes and rolling of the seas resulted in a highly attenuated beer upon arrival.  English pale ales were derived from India Pale Ales.
Comments: A pale ale brewed to an increased gravity and hop rate.  Modern versions of English IPAs generally pale in comparison (pun intended) to their ancestors.  The term "IPA" is loosely applied in commercial English beers today, and has been (incorrectly) used in beers below 4% ABV.  Generally will have more finish hops and less fruitiness and/or caramel than English pale ales and bitters.  Fresher versions will obviously have a more significant finishing hop character.
Ingredients: Pale ale malt (well-modified and suitable for single-temperature infusion mashing); English hops; English yeast that can give a fruity or sulfury/minerally profile. Refined sugar may be used in some versions.  High sulfate and low carbonate water is essential to achieving a pleasant hop bitterness in authentic Burton versions, although not all examples will exhibit the strong sulfate character.
Vital Statistics:    OG:    1.050 - 1.075    IBUs:    40 - 60    FG:    1.010 - 1.018    SRM:    8 - 14    ABV:    5 - 7.5%
Commercial Examples: Meantime India Pale Ale, Freeminer Trafalgar IPA, Fuller's IPA, Ridgeway Bad Elf, Summit India Pale Ale, Samuel Smith's India Ale, Hampshire Pride of Romsey IPA, Burton Bridge Empire IPA,Middle Ages ImPailed Ale, Goose Island IPA, Brooklyn East India Pale Ale

14B. American IPA
Aroma: A prominent to intense hop aroma with a citrusy, floral, perfume-like, resinous, piney, and/or fruity character derived from American hops.  Many versions are dry hopped and can have an additional grassy aroma, although this is not required.  Some clean malty sweetness may be found in the background, but should be at a lower level than in English examples.  Fruitiness, either from esters or hops, may also be detected in some versions, although a neutral fermentation character is also acceptable.  Some alcohol may be noted.
Appearance: Color ranges from medium gold to medium reddish copper; some versions can have an orange-ish tint.  Should be clear, although unfiltered dry-hopped versions may be a bit hazy.  Good head stand with white to off-white color should persist.
Flavor: Hop flavor is medium to high, and should reflect an American hop character with citrusy, floral, resinous, piney or fruity aspects.  Medium-high to very high hop bitterness, although the malt backbone will support the strong hop character and provide the best balance.  Malt flavor should be low to medium, and is generally clean and malty sweet although some caramel or toasty flavors are acceptable at low levels. No diacetyl.  Low fruitiness is acceptable but not required.  The bitterness may linger into the aftertaste but should not be harsh.  Medium-dry to dry finish.  Some clean alcohol flavor can be noted in stronger versions.  Oak is inappropriate in this style.  May be slightly sulfury, but most examples do not exhibit this character.
Mouthfeel: Smooth, medium-light to medium-bodied mouthfeel without hop-derived astringency, although moderate to medium-high carbonation can combine to render an overall dry sensation in the presence of malt sweetness.  Some smooth alcohol warming can and should be sensed in stronger (but not all) versions.  Body is generally less than in English counterparts.
Overall Impression: A decidedly hoppy and bitter, moderately strong American pale ale.
History: An American version of the historical English style, brewed using American ingredients and attitude.
Ingredients: Pale ale malt (well-modified and suitable for single-temperature infusion mashing); American hops; American yeast that can give a clean or slightly fruity profile. Generally all-malt, but mashed at lower temperatures for high attenuation.  Water character varies from soft to moderately sulfate.  Versions with a noticeable Rye character ("RyePA") should be entered in the Specialty category.
Vital Statistics:    OG:    1.056 - 1.075    IBUs:    40 - 70    FG:    1.010 - 1.018    SRM:    6 - 15    ABV:    5.5 - 7.5%
Commercial Examples: Bell's Two-Hearted Ale, AleSmith IPA, Russian River Blind Pig IPA, Stone IPA, Three Floyds Alpha King, Great Divide Titan IPA, Bear Republic Racer 5 IPA, Victory Hop Devil, Sierra Nevada Celebration Ale, Anderson Valley Hop Ottin',  Dogfish Head 60 Minute IPA, Founder's Centennial IPA, Anchor Liberty Ale, Harpoon IPA, Avery IPA

Example Output:

Code:
$ ls ./scripts/tests/beers
INDIA PALE ALE (IPA).dat  
LIGHT LAGER.dat  

$ cat ./scripts/tests/beers/INDIA\ PALE\ ALE\ \(IPA\).dat
Category|Style|Aroma|Appearance|Flavor|Mouthfeel|Overall Impression|Comments|Ingredients|OG|IBUs|FG|SRM|ABV|Commercial Examples
INDIA PALE ALE (IPA)|English IPA|A moderate to moderately high hop aroma of floral, earthy or fruity nature is typical, although the intensity of hop character is usually lower than American versions.  A slightly grassy dry-hop aroma is acceptable, but not required.  A moderate caramel-like or toasty malt presence is common.  Low to moderate fruitiness, either from esters or hops, can be present.  Some versions may have a sulfury note, although this character is not mandatory.|Color ranges from golden amber to light copper, but most are pale to medium amber with an orange-ish tint.  Should be clear, although unfiltered dry-hopped versions may be a bit hazy.  Good head stand with off-white color should persist.|Hop flavor is medium to high, with a moderate to assertive hop bitterness.  The hop flavor should be similar to the aroma (floral, earthy, fruity, and/or slightly grassy).  Malt flavor should be medium-low to medium-high, but should be noticeable, pleasant, and support the hop aspect.  The malt should show an English character and be somewhat bready, biscuit-like, toasty, toffee-like and/or caramelly.  Despite the substantial hop character typical of these beers, sufficient malt flavor, body and complexity to support the hops will provide the best balance. Very low levels of diacetyl are acceptable, and fruitiness from the fermentation or hops adds to the overall complexity.  Finish is medium to dry, and bitterness may linger into the aftertaste but should not be harsh.  If high sulfate water is used, a distinctively minerally, dry finish, some sulfur flavor, and a lingering bitterness are usually present.  Some clean alcohol flavor can be noted in stronger versions.  Oak is inappropriate in this style.|Smooth, medium-light to medium-bodied mouthfeel without hop-derived astringency, although moderate to medium-high carbonation can combine to render an overall dry sensation in the presence of malt sweetness.  Some smooth alcohol warming can and should be sensed in stronger (but not all) versions.  |A hoppy, moderately strong pale ale that features characteristics consistent with the use of English malt, hops and yeast.  Has less hop character and a more pronounced malt flavor than American versions.|Brewed to survive the voyage from England to India.  The temperature extremes and rolling of the seas resulted in a highly attenuated beer upon arrival.  English pale ales were derived from India Pale Ales.|A pale ale brewed to an increased gravity and hop rate.  Modern versions of English IPAs generally pale in comparison (pun intended) to their ancestors.  The term "IPA" is loosely applied in commercial English beers today, and has been (incorrectly) used in beers below 4% ABV.  Generally will have more finish hops and less fruitiness and/or caramel than English pale ales and bitters.  Fresher versions will obviously have a more significant finishing hop character.|Pale ale malt (well-modified and suitable for single-temperature infusion mashing); English hops; English yeast that can give a fruity or sulfury/minerally profile. Refined sugar may be used in some versions.  High sulfate and low carbonate water is essential to achieving a pleasant hop bitterness in authentic Burton versions, although not all examples will exhibit the strong sulfate character.| 1.050 - 1.075|40 - 60|1.010 - 1.018|8 - 14|5 - 7.5%|Meantime India Pale Ale, Freeminer Trafalgar IPA, Fuller's IPA, Ridgeway Bad Elf, Summit India Pale Ale, Samuel Smith's India Ale, Hampshire Pride of Romsey IPA, Burton Bridge Empire IPA,Middle Ages ImPailed Ale, Goose Island IPA, Brooklyn East India Pale Ale
INDIA PALE ALE (IPA)|American IPA|A prominent to intense hop aroma with a citrusy, floral, perfume-like, resinous, piney, and/or fruity character derived from American hops.  Many versions are dry hopped and can have an additional grassy aroma, although this is not required.  Some clean malty sweetness may be found in the background, but should be at a lower level than in English examples.  Fruitiness, either from esters or hops, may also be detected in some versions, although a neutral fermentation character is also acceptable.  Some alcohol may be noted.|Color ranges from medium gold to medium reddish copper; some versions can have an orange-ish tint.  Should be clear, although unfiltered dry-hopped versions may be a bit hazy.  Good head stand with white to off-white color should persist.|Hop flavor is medium to high, and should reflect an American hop character with citrusy, floral, resinous, piney or fruity aspects.  Medium-high to very high hop bitterness, although the malt backbone will support the strong hop character and provide the best balance.  Malt flavor should be low to medium, and is generally clean and malty sweet although some caramel or toasty flavors are acceptable at low levels. No diacetyl.  Low fruitiness is acceptable but not required.  The bitterness may linger into the aftertaste but should not be harsh.  Medium-dry to dry finish.  Some clean alcohol flavor can be noted in stronger versions.  Oak is inappropriate in this style.  May be slightly sulfury, but most examples do not exhibit this character.|Smooth, medium-light to medium-bodied mouthfeel without hop-derived astringency, although moderate to medium-high carbonation can combine to render an overall dry sensation in the presence of malt sweetness.  Some smooth alcohol warming can and should be sensed in stronger (but not all) versions.  Body is generally less than in English counterparts.|A decidedly hoppy and bitter, moderately strong American pale ale.|An American version of the historical English style, brewed using American ingredients and attitude.|Pale ale malt (well-modified and suitable for single-temperature infusion mashing); American hops; American yeast that can give a clean or slightly fruity profile. Generally all-malt, but mashed at lower temperatures for high attenuation.  Water character varies from soft to moderately sulfate.  Versions with a noticeable Rye character ("RyePA") should be entered in the Specialty category.| 1.056 - 1.075|40 - 70|1.010 - 1.018|6 - 15|5.5 - 7.5%|Bell's Two-Hearted Ale, AleSmith IPA, Russian River Blind Pig IPA, Stone IPA, Three Floyds Alpha King, Great Divide Titan IPA, Bear Republic Racer 5 IPA, Victory Hop Devil, Sierra Nevada Celebration Ale, Anderson Valley Hop Ottin',  Dogfish Head 60 Minute IPA, Founder's Centennial IPA, Anchor Liberty Ale, Harpoon IPA, Avery IPA

$ cat ./scripts/tests/beers/LIGHT\ LAGER.dat
Category|Style|Aroma|Appearance|Flavor|Mouthfeel|Overall Impression|Comments|Ingredients|OG|IBUs|FG|SRM|ABV|Commercial Examples
LIGHT LAGER|Lite American Lager|Little to no malt aroma, although it can be grainy, sweet or corn-like if present.  Hop aroma may range from none to a light, spicy or floral hop presence.  Low levels of yeast character (green apples, DMS, or fruitiness) are optional but acceptable.  No diacetyl.|Very pale straw to pale yellow color.  White, frothy head seldom persists.  Very clear.|Crisp and dry flavor with some low levels of grainy or corn-like sweetness.  Hop flavor ranges from none to low levels.  Hop bitterness at low level.  Balance may vary from slightly malty to slightly bitter, but is relatively close to even.  High levels of carbonation may provide a slight acidity or dry "sting."  No diacetyl.  No fruitiness.|Very light body from use of a high percentage of adjuncts such as rice or corn.  Very highly carbonated with slight carbonic bite on the tongue.  May seem watery.|Very refreshing and thirst quenching.|A lower gravity and lower calorie beer than standard international lagers.  Strong flavors are a fault. Designed to appeal to the broadest range of the general public as possible.|Two- or six-row barley with high percentage (up to 40%) of rice or corn as adjuncts.| 1.028 - 1.040|8 - 12|0.998 - 1.008|2 - 3|2.8 - 4.2%|Bitburger Light, Sam Adams Light, Heineken Premium Light, Miller Lite, Bud Light, Coors Light, Baltika #1 Light, Old Milwaukee Light, Amstel Light
LIGHT LAGER|Standard American Lager|Little to no malt aroma, although it can be grainy, sweet or corn-like if present.  Hop aroma may range from none to a light, spicy or floral hop presence.  Low levels of yeast character (green apples, DMS, or fruitiness) are optional but acceptable.  No diacetyl.|Very pale straw to medium yellow color.  White, frothy head seldom persists.  Very clear.|Crisp and dry flavor with some low levels of grainy or corn-like sweetness.  Hop flavor ranges from none to low levels.  Hop bitterness at low to medium-low level.  Balance may vary from slightly malty to slightly bitter, but is relatively close to even.  High levels of carbonation may provide a slight acidity or dry "sting."  No diacetyl.  No fruitiness.|Light body from use of a high percentage of adjuncts such as rice or corn.  Very highly carbonated with slight carbonic bite on the tongue.|Very refreshing and thirst quenching.  |Strong flavors are a fault.  An international style including the standard mass-market lager from most countries.|Two- or six-row barley with high percentage (up to 40%) of rice or corn as adjuncts.| 1.040 - 1.050|8 - 15|1.004 - 1.010|2 - 4|4.2 - 5.3%|Pabst Blue Ribbon, Miller High Life, Budweiser, Baltika #3 Classic, Kirin Lager, Grain Belt Premium Lager, Molson Golden, Labatt Blue, Coors Original, Foster's Lager

Awk program file:

Code:
$ cat ./scripts/split_group2.awk
function f1()                            { if(dat!="") {print dat > ( path cat ext ) } }

BEGIN                                    {
                                           path="/tools/scripts/tests/beers/"
                                           dlm="|"
                                           ext=".dat"
                                           head="Category|Style|Aroma|Appearance|Flavor|Mouthfeel|Overall Impression|Ingredients|OG|IBUs|FG|SRM|ABV|Commercial Examples"
                                         }

/^[0-9]+\./                              {
                                           f1()
                                           cat=substr($0,(match($0," ")+1))
                                           print head > ( path cat ext )
                                           dat = ""
                                         }

/^[0-9]+[A-Z]+\./                        {
                                           f1()
                                           dat = cat dlm substr($0,(match($0," ")+1))
                                         }

/^Vital/                                 {
                                           gsub(/[a-zA-Z]/,"")
                                           gsub("::","")
                                           gsub(":","|")
                                           dat = dat dlm $0
                                         }

/^[A-Z]/ && (!/^History/ || !/^Comments) { dat = dat dlm substr($0,(match($0,":")+2)) }

END                                      { f1() }

The actual line submitted:

Code:
$ sed 's/\t//g' ./scripts/tests/beers.dat | awk -f ./scripts/split_group2.awk

I am using sed to cleanup the tabs from the Vital Statistics: lines of the input file, I suppose I could've just done so in awk, but, whatever...
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Split binary file every occurrence of a group of characters

Hello I am new to scripts, codes, bash, terminal, etc. I apologize this my be very scattered because I frankly don't have any idea where to begin and I have had trouble sleeping lately. I have several 2GB files I wish to split. This Code 00 00 01 BA ** ** ** ** ** ** ** ** C3 F8 00 00 01 BB 00... (17 Replies)
Discussion started by: PatrickE
17 Replies

2. Solaris

Create file for group of data:

Hi folks, I have the following data.Any help is greatly appreciated. order File_name 7222245 7222245.pdf 7222245 7222245a.pdf 7222245 7222245b.pdf 7222245 7222245c.pdf 7222245 7222245d.pdf 7222250 ... (1 Reply)
Discussion started by: kumar444
1 Replies

3. Shell Programming and Scripting

Split a file into several files using a data

Hi All, I have file(File1) with data like below: 102100|LName|Gender|Company|Branch|Bday|Salary|Age 102100|bbbb|male|cccc|dddd|19900814|15000|20| 102101|asdg|male|gggg|ksgu|19911216||| 102102|bdbm|male|kkkk|acke|19931018||23| 102102|kfjg|male|kkkc|gkgg|19921213|14000|24|... (2 Replies)
Discussion started by: sarav.shan
2 Replies

4. Shell Programming and Scripting

Split File data using awk

HI Guys, I need to split the file in to number of files . file contains FILEHEADER and EOF . I have to split n number of times . I have to form the file with each splitted message between FILEHEADER and EOF using awk beign and end . how to implement please suggest. (2 Replies)
Discussion started by: manish8484
2 Replies

5. Shell Programming and Scripting

How to split a data file into separate files with the file names depending upon a column's value?

Hi, I have a data file xyz.dat similar to the one given below, 2345|98|809||x|969|0 2345|98|809||y|0|537 2345|97|809||x|544|0 2345|97|809||y|0|651 9685|98|809||x|321|0 9685|98|809||y|0|357 9685|98|709||x|687|0 9685|98|709||y|0|234 2315|98|809||x|564|0 2315|98|809||y|0|537... (2 Replies)
Discussion started by: nithins007
2 Replies

6. Shell Programming and Scripting

split input data file and put into same output file

Hi All, I have two input file and need to generate a CSV file. The existing report just "GREP" the records with the Header and Tailer records with the count of records. Now i need to split the data into 25 records each in the same CSV file. id_file (Input file ) 227050994 232510151... (4 Replies)
Discussion started by: rasmith
4 Replies

7. Shell Programming and Scripting

Split, Search and Reformat by Data Group

Hi, I am writing just to share my appreciation for help I have received from this site in the past. In a previous post Split File by Data Group I received a lot of help with a troublesome awk script to reformat some complicated data blocks. What I learned really came in hand recently when I... (1 Reply)
Discussion started by: mkastin
1 Replies

8. Shell Programming and Scripting

split file based on group count

Hi, can some one please help me to split the file based on groups. like in the below scenario x indicates the begining of the group and the file should be split each with 2 groups below there are 10 groups it should create 5 files. could you please help? (4 Replies)
Discussion started by: hitmansilentass
4 Replies

9. Shell Programming and Scripting

Can I split a 10GB file into 1 GB sizes using my repeating data pattern

I'm not a unix guy so excuses my ignorance... I'm the database ETL guy. I'm trying to be proactive and devise a plan B for a ETL process where I expect a file 10X larger than what I process daily for a recast job. The ETL may handle it but I just don't know. This file may need to be split... (3 Replies)
Discussion started by: john091
3 Replies

10. HP-UX

Need to split a large data file using a Unix script

Greetings all: I am still new to Unix environment and I need help with the following requirement. I have a large sequential file sorted on a field (say store#) that is being split into several smaller files, one for each store. That means if there are 500 stores, there will be 500 files. This... (1 Reply)
Discussion started by: SAIK
1 Replies
Login or Register to Ask a Question