Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Extracting information from text fields.

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Extracting information from text fields.
# 1  
Old 06-03-2007
Extracting information from text fields.
Dear friends,
I'm a novice Unix user and I'm trying to learn the ropes. I have a big task I have to accomplish and I'm convinced Unix can get the job done, I just haven't figured out how. I recently posted on the topic of cutting text between unique text patterns and somebody helped me a great deal. It worked great.
There are other tasks, however, that I want to accomplish.
I'm doing a content analysis of newspaper articles that I've exported in .txt format from a ProQuest database. The .rtf files look like this when I cat them in Unix.

Quote:
\
Adoptive parents often face tactless questions: Curiosity lies behind rude\
inquiries:[Final Edition]\
Karen Miles. Edmonton Journal. Edmonton, Alta.:Mar 15, 2000. p. F7 \
\
Author(s): Karen Miles\
\
Document types: News\
\
Section: Living\
\
Publication title: Edmonton Journal. Edmonton, Alta.: Mar 15, 2000. pg. F.7\
\
Source type: Newspaper\
\
ProQuest document 220675241\
ID:\
\
Text Word Count 314\
\
Document URL: {\field{\*\fldinst{HYPERLINK "http://proquest.umi.com"}}{\fldrslt http://proquest.umi.com}}/\
pqdweb?did=220675241&Fmt=3&clientId=14119&RQT=309&VName=PQD\
\
Abstract (Document Summary)\
\
"Is she adopted?" the school registrar whispered to Kathryn Creedy, of\
Alexandria, Va., when Creedy signed Alexis, her then five- year-old Romanian\
daughter, up for school. Since Alexis could clearly hear the question, Creedy\
curtly replied: "Yes, my daughter is adopted -- and she knows it!"\
\
"The media loves to sensationalize stories." Then point out that those stories\
are rare, and that there are many successful people who were adopted -\
- including playwright Edward Albee, Olympic skater Scott Hamilton, actor\
Melissa Gilbert and Apple Computer co- founder Steven Jobs.\
\
\
\
Full Text (314 words)\
\
Copyright Southam Publications Inc. Mar 15, 2000\
\
"Is she adopted?" the school registrar whispered to Kathryn Creedy, of\
Alexandria, Va., when Creedy signed Alexis, her then five- year-old Romanian\
daughter, up for school. Since Alexis could clearly hear the question, Creedy\
curtly replied: "Yes, my daughter is adopted -- and she knows it!"\
\
But Creedy resented the probe, especially when she later learned that the\
registrar hadn't needed the information. "The woman violated our privacy just\
to satisfy her own curiosity," she says.\
The information contained in the 'fields' such as author, date, document type, and text word count would be immensely valuable to me. I would like to be able to extract the information (preferably only the text after the field titles (author, date, text word count) although this is not necessary. I imagine find and replace functions in Excel could be used to delete that easily. I would like to be able to get this information to some kind of excel database, probably via a .csv file.
Ultimately, I will have hundreds of these news stories to extract the information from.

Does anybody have any suggestions?

Simon
# 2  
Old 06-03-2007
just a snippet of it:
Code:
awk 'BEGIN{FS=":"}
    { gsub(/\\/,"")}
    /Author/{print "Author: "$2 }
    /Document types/{print "Document: "$2 }
    /Text Word Count/{sub(/Text Word Count/,"");print "Word count: " $0}
' "file"

output:
Code:
# ./test1.sh
Author:  Karen Miles
Document:  News
Word count:  314

# 3  
Old 06-05-2007
That's really helpful. For some reason, however, the gsub command doesn't delete the backslashes in the files. When I play with it, the command doesn't return an error; it seems to complete, but the files still contain the slashes.

Also, the field variables don't write exactly right. The /Author/{print "Author: "$2 }string only prints the first, not both names of the author. I had to modify it to read "$2, $3" to get first and last names. Any clue why that might be?

Thanks again though, this is what I was looking for!
Simon
# 4  
Old 06-05-2007
Quote:
Originally Posted by spindoctor
That's really helpful. For some reason, however, the gsub command doesn't delete the backslashes in the files.
gsub() modifies what you passed to it "in place". eg gsub("","",$1);print $1


Quote:
Also, the field variables don't write exactly right. The /Author/{print "Author: "$2 }string only prints the first, not both names of the author. I had to modify it to read "$2, $3" to get first and last names. Any clue why that might be?
it should be no problem if the field separator is specified as ":" . show your code if you want.
# 5  
Old 06-06-2007
Hi, On the first point, this is the code I was playing with from the command line, based on your suggestion.
awk 'BEGIN { gsub(/\\/,"")}' 03152000.rtf

Did you put the gsub command in there just to get rid of the backslashes? Or is it related to the field extraction process?

Regarding the second point:
Here is the text, copied straight from the .txt file I'm trying to extract information from.

Author(s): Ashley Geddes, Provincial Affairs Writer

Here is the code (entered in the command line) I got from you and it's output.

awk 'Begin{FS=":"} ; /Author/{print $2 }' 03152000.rtf > Author.txt

Ashley

Here is the modified way I wrote the code and its result

awk 'Begin{FS=":"} ; /Author/{print $2,$3 }' 03152000.rtf > Author.txt

Ashley Geddes,

the same problem occurred with the field "Document Types" I had to change around the fields as you wrote them to get the result.

Obviously I'm not that concerned, because it seems to be working. However I am very curious because I'd like to know how the thing works. Thanks for your help!
# 6  
Old 06-06-2007
use BEGIN not Begin. and yes ,the gsub is there to remove slash
# 7  
Old 06-07-2007
For some reason, I really can't get the slash to be removed, but it's no problem.

how would I modify this script to make it perform this command to 100 different files in the same directory i.e. how do I scroll through 100 different files, extracting the 'author' field from each and posting it to one single 'author.txt' file?

Thanks for all your help!
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extracting information from XML to excel

Hi, I am trying to extract information from a XML file and write it to a excel sheet. I am not sure where to start from. Here is the content from my input XML file. <com.cloudbees.hudson.plugins.folder.properties.FolderProxyGroupContainer plugin="nectar-rbac@4.5"> <groups> ... (4 Replies)
Discussion started by: Sajjadmehdi
4 Replies

2. Shell Programming and Scripting

[Solved] Extracting information from DDL's

Dear Experts, I need your help here. I have lot of teradata DDL's as follows, i want to extract field names , field attributes and NOT NULL information from DDL.Could you please help here. Sample DDL: CREATE MULTISET TABLE APS_CALL_IN_PICKUP_CANCELED ,NO FALLBACK , NO BEFORE... (2 Replies)
Discussion started by: srikanth38
2 Replies

3. Shell Programming and Scripting

Extracting information using awk

I want to write a script that extracts a value from a line of text. I know it can be done using awk but I've never used awk before so I don't know how to do it. The text is: Mem: 100M Active, 2150K Cache, 500M Buf, 10G Free I want to extract the free memory value to use as a variable. In... (5 Replies)
Discussion started by: millsy5
5 Replies

4. Shell Programming and Scripting

Problems extracting some information

Hi there! Well, I'm writing a script to obtain certain information about files. Specifically, I want to get the information about those files which last access were in the last 24 hours, so I'm doing something like this: find <directory_name> -atime -1 -printf '%f %a\n' I would also... (4 Replies)
Discussion started by: Skirmish
4 Replies

5. Shell Programming and Scripting

Extracting relevant information from syslogs.

I need to analyse some syslogs and I want to print out all the lines containing SSH connections to the inside interface of the firewall and ignore lines where the originating port is 22. So basically I want to print all matches after "to inside:" that contains /22 and ignore lines where /22 occur... (2 Replies)
Discussion started by: lewk
2 Replies

6. Shell Programming and Scripting

extracting information from multiple files

Hello there, I am trying to extract (string) information ( a list words) from 4 files and then put the results into 1 file. Currently I am doing this using grep -f list.txt file1 . and repeat the process for the other 3 files. The reasons i am doing that (a) I do know how to code (b) each file... (4 Replies)
Discussion started by: houkto
4 Replies

7. Shell Programming and Scripting

Extracting information from Config files /text processing

Hello All, This is my first post on this forums, which I consider one of the best of its kind. The reason for my post is that I want to export some information form Nagios configuration files to a DB. I know that there are other tools available to do this, like NDO, monarch, etc... But I want to... (3 Replies)
Discussion started by: oconmx
3 Replies

8. Shell Programming and Scripting

Problems with extracting information

Hi all, <select name="comp" id="comp" style="width:130px;"> <?php $sqlcomp = mysql_query("SELECT * FROM comp"); while ($redcomp = mysql_fetch_array($sqlcomp)) { extract($redcomp); echo "<option value=\"$comp_id\">comp_name</option>"; } ?> ... (0 Replies)
Discussion started by: c0mrade
0 Replies

9. Shell Programming and Scripting

extracting fields

Hi, i have a line with several fields (indefinite number of - count varies) separated by colon. Now, i need to pick each field (except the first one) and have it assigned to variable within a loop. In other words, in the first iteration of the loop, the variable must be assigned with 2nd... (2 Replies)
Discussion started by: prvnrk
2 Replies

10. Shell Programming and Scripting

Extracting information from a template

I have a template that I usually use to generate stats on an hourly basis for a number of cell sites altogether. I would like to be able to write a script that would go to the template and extract the information for any single site at any time during the day. For example, let's say that my... (4 Replies)
Discussion started by: Ernst
4 Replies
Login or Register to Ask a Question