Shell Programming and Scripting

Just noticed this:

Code:

PARTNUM6546-048

If this is supposed to be one record, you'll need to change FS slightly.

Code:

# won't accept -
# FS="[^a-zA-Z0-9_]"
# should accept -
FS="[^a-zA-Z0-9_\\-]"

I don't have anything useful to add over what Corona gave you, but, I am very interested to hear what the results are on your time savings.

Something tells me it will be significant. Guesstimate is that it looks to be roughly 150mb of raw data (3M * 50 char per line). I've ran similar on terabytes worth of data and it only took me an hour or so.

You brought up a great point Corona, there are "." and " " and "-" in user entered fields that I don't want awk to use as a delimiter ... I haven't yet been able to figure out the syntax for including these characters in the FS statement.

I'm experimenting so that I can understand HOW awk works, so that I can use the same type of logic for some of the other parsing tasks I have in my original script

I'm still working on the IF statements (to grab just the first occurence)

But I really like this solution ... I agree this will save hours of processing time

---------- Post updated at 06:24 PM ---------- Previous update was at 06:15 PM ----------

It works great BTW!

Just add them, if they don't work, put a \\ in front of them. Easy enough to test them by echo "a-b-c-d-e-f| awk 'BEGIN { FS="..." } { print $1,$2,$3; }' If you followed the logic for my example, you're already over the main hurdle, understanding that awk has its own built-in "while" loop to read in data record by record based on RS(usually newline) then split into tokens based on FS(usually whitespace). You can solve quite a few tasks by just by carefully adjusting its input and output settings.

But you can also cavalierly ignore them whenever you please and just deal with whole lines via $0, print arbitrary anything with printf, read an extra line into $0 with getline, etc.

Other mindblowers:
...and lots more. My own understanding of awk is far from complete.

Thanks again Corona ... I'm still having trouble with excluding - as a delimiter in

Code:

FS="[^a-zA-Z0-9\\_\\.\\/\\-\\ ]"
 
I've tried \\-, \-, -, '-'

---------- Post updated at 01:56 PM ---------- Previous update was at 01:52 PM ----------

Thanks again for your help ... the difference in processing time reduced greatly ... from almost 6 hours to 20 minutes ... for approx over 3 million lines in about 35K lines

---------- Post updated at 02:21 PM ---------- Previous update was at 01:56 PM ----------

I have a slightly different issue with delimiters. I've been isolating and replacing the delimiters with grep, cut and sed but I know there is a much better way to do this with awk or even perl.

In my file the token seperator can be any non-word character, however the line terminator will also be a non-word character (the rule dictates that these two characters must be different). To add to the issue, user entered fields can include non-word characters, as long as those characters aren't the ones used for the token/line seperators.

Code:

Example record
ISA¼3213¼part-number¼address~GS¼56756¼control{number~ST¼09898~
 
Into 
ISA|3213|part-number|address
GS|56756|control{number
ST|09898

I need to replace the token seperator ¼ with a more standard delimiter like |
I need to replace line terminator ~ with line feed (0D0A)

However ¼ and ~ change from record to record and there are thousands of them in a file. I've csplit the files such that each file has the same token/field seperator and line terminator.

Let me know if I can explain this more clearly!

To exclude - as a delimiter, don't put it in there. (things like 0-9 don't count.)

Is there or is there not a newline already where ~ is?

How did you csplit them?

there is no newline where ~ is ... I csplit by the record header "ISA"

... most are on a newline
... but if the file has less than three lines, I know that the record has this issue
... "ISA" always has a non-word character before and after it
... the line terminator is always in position 107 (in my file)

in the shell version of my script, I used an if and sed to put ISAs on its own line and then csplit the file by ISA

I'm not sure how to cleverly add a newline where ~