Shell Programming and Scripting

I'm working on a different stage of a project that someone helped me address elsewhere in these threads.
The .docs I'm cycling through look roughly like this:

Code:

1 of 26 DOCUMENTS


Copyright 2010 The Age Company Limited
All Rights Reserved 
The Age (Melbourne, Australia)

November 27, 2010 Saturday  
First Edition

SECTION: NEWS; In Brief Overseas; Pg. 16

LENGTH: 114 words

There are about four .doc files I'm cycling through and I need to get them to look like this:

Code:

1 of 26 Documents, Copyright 2010 etc, All Rights Reserved, The Age (Melbourne, Australia), November 27, 2010
2 of 26 Documents, Copyright 2010 etc, All Rights Reserved, The Age (Melbourne, Australia), November 27, 2010
3 of 26 Documents, Copyright 2010 etc, All Rights Reserved, The Age (Melbourne, Australia), November 27, 2010

I'm working with the following script

Code:

#!/usr/bin/sh
for i in /Users/simon/canadian/*
do
  textutil -convert txt $i 
done

for i in /Users/simon/canadian/*
do
  sed -i "" '/^$/d' $i  
done

rm /Users/simon/canadian/*.doc

for i in /Users/simon/canadian/*
do
  grep -A5 'DOCUMENTS$' $i > $i.tmp
  mv $i.tmp $i
done

for i in /Users/simon/canadian/*
do
  gawk '$1=$1' RS="DOCUMENTS$" FS="\n" OFS=, $i >$i.tmp
  mv $i.tmp $i
done

It's doing what I want, EXCEPT, the output is all being put on one line:
1 of 26 DOCUMENTS,Copyright 2010 The Age Company Limited,All Rights Reserved ,--,2 of 26 DOCUMENTS,Copyright 2010 Nationwide News Pty Limited,All Rights Reserved ,--,3 of 26 DOCUMENTS,Copyright 2010 John Fairfax Publications Pty Ltd,All Rights Reserved ,--

I know that the double-dash is inserted by AWK when it encounters a a new record separator, so it is encountering the "DOCUMENTS$" record separator properly, but it's not outputting the text on separate lines.
Can someone provide any insight?
Simon

Could you display the expected result?
Thank you.

Hi spindoctor,

Based on sample an option would be:

Code:

for each in *.doc
do
awk '
/^[0-9].*DOCUMENTS/{a=$0}
/^Copyright/||/^All/||/^The Age/{a=a","$0}
/[A-Z].* [0-9]+,/{$NF="";a=a","$0;print a}' $each
done

Hope it helps.

Regards

@Shell_Life:
The expected result is this (lines only separated at 2 of 26, 3 of 26, etc.:
1 of 26 Documents, Copyright 2010 etc, All Rights Reserved, The Age (Melbourne, Australia), November 27, 2010
2 of 26 Documents, Copyright 2010 etc, All Rights Reserved, The Age (Melbourne, Australia), November 27, 2010
3 of 26 Documents, Copyright 2010 etc, All Rights Reserved, The Age (Melbourne, Australia), November 27, 2010

@cgkmal, I don't think that will work. There are many different newspaper names.
If you look in the script that I'm using, I first grep the 5 lines from each line that matches "[0-9] Documents$", so that grabs all the text from the Document Number down to the date. But it produces it a file where all the information is in rows...

1 of 26 Documents
Copyright
All Rights reserved
NOvember 27, 2010.

So I'm trying to transpose. I don't think your solution will work because I've got other newspaper titles.

How about an awk script for all that...

Code:

awk '{
   if ($0~"DOCUMENTS") {
      f=0
      x=""
   }
   if ($0 ~ "Edition$") {
      f=1
      if (x) printf("%s\n", x)
      x=""
   }
   if ($0 && !f) {
      if (x) x=sprintf("%s,%s", x, $0)
      else   x=sprintf("%s", $0);
   }
}' 1.doc 2.doc 3.doc ...

Code:

grep -A5 'DOCUMENTS$' tst | awk '/Copyright/{NF=3;$3="etc"}$0!~/^$/' | paste -s -d"," -

---------- Post updated at 10:11 PM ---------- Previous update was at 10:09 PM ----------

Instead of tst, put your filename (*.doc , $i, or whatever )

---------- Post updated at 10:16 PM ---------- Previous update was at 10:11 PM ----------

You may want to choose another delimiter (colon for example) to avoid further parsing problem because the sequence <coma><space> also match in (Melbourne, Australia) whereas it is not supposed to be a delimiter but may be interpreted as is later... or may make further parsing tedious.

@ctsgnb
That might work, but again, several of the files have different text contained in the first part of each document.
How do I write the output from the paste command to a file?

---------- Post updated at 05:11 PM ---------- Previous update was at 04:50 PM ----------

@shamrock, I tried your script and the output is marvellous, but it is not matching all the documents in each file I'm working on.
The problem is that each file has information entered in different ways. In one file, the word edition comes before the date, so your script stops printing before the crucial information, the date.

The only thing that unifies all the files is that each section is separated by the phrase
[0-9] of [0-9] DOCUMENTS, followed by some information, followed by a date. The only way to match that text is to grep a few lines after the Documents phrase, transpose the rows, and then play around with the columns I've got (probably in Excel) to get all the dates lined up. I was really very close with my original script, but it would just output everything on one line, rather than on one line for each record (defined as the space between the DOCUMENTS string.

---------- Post updated at 10:40 PM ---------- Previous update was at 05:11 PM ----------

The problem is really with the newline. I can get all the information I need out of every file, but awk is just outputting everything onto one line
And where it encounters the new record string that I define at the beginning of the script, it inserts a double dash (see right before 2 of 28) in the output, but it won't print them on two lines.
If I can just break up that output into one record per line, which I thought awk was supposed to do, I'm off to the races...