Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Creating Frequency of words from a file by accessing a corpus
Top Forums Shell Programming and Scripting Creating Frequency of words from a file by accessing a corpus Post 302836273 by durden_tyler on Wednesday 24th of July 2013 12:03:17 AM
Old 07-24-2013
Quote:
Originally Posted by gimley
...I need to identify the frequency of each of these strings from a large corpus (which I cannot attach unfortunately because of size limitations) and identify the frequency of each string.
...
Code:
$ 
$ # "wordlist.txt" is a list of words that we have to check
$ cat wordlist.txt
be
at
for
if
being
attract
$ 
$ # "poe_the_gold_bug.txt" is a text file against which we have to
$ # check the words. This file contains the story "The Gold Bug" by
$ # Edgar Allen Poe from the Project Gutenberg website.
$ wc poe_the_gold_bug.txt
 1460 13462 76460 poe_the_gold_bug.txt
$ 
$ # A Perl program to check the frequency of words from "wordlist.txt"
$ # in the file "poe_the_gold_bug.txt"
$ cat -n word_occurrences.pl
     1    #!/usr/bin/perl -w
     2    use strict;
     3    my $wordfile = $ARGV[0];
     4    my $testfile = $ARGV[1];
     5    my %occurrences;
     6    open(WF, "<", $wordfile) or die "Can't open $wordfile: $!";
     7    while (<WF>) {
     8      chomp;
     9      $occurrences{$_} = 0
    10    }
    11    close(WF) or die "Can't close $wordfile: $!";
    12    open(TF, "<", $testfile) or die "Can't open $testfile: $!";
    13    while (<TF>) {
    14      chomp;
    15      while (/(\w+)/g) {
    16        $occurrences{$1}++ if defined $occurrences{$1};
    17      }
    18    }
    19    close(TF) or die "Can't close $testfile: $!";
    20    while (my ($k, $v) = each %occurrences) {
    21      printf("%-10s occurs %5d times\n", $k, $v);
    22    }
$ 
$ # Execution of the Perl program
$ perl word_occurrences.pl wordlist.txt poe_the_gold_bug.txt
attract    occurs     0 times
for        occurs   109 times
be         occurs    72 times
at         occurs    96 times
being      occurs    13 times
if         occurs    24 times
$ 
$

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Creating String from words in a file

Hi i have a file called search.txt Which contains text like Car Bus Cat Dog Now i have to create a string from the file which should look like Car,Bus,Cat,Dog ( appending , is essential part) String must be stored in some variable so i can pass it as argument to some other... (5 Replies)
Discussion started by: deepakthaman
5 Replies

2. Shell Programming and Scripting

Splitting Concatenated Words in Input File with Words from a Master File

Hello, I have a complex problem. I have a file in which words have been joined together: Theboy ranslowly I want to be able to correctly split the words using a lookup file in which all the words occur: the boy ran slowly slow put child ly The lookup file which is meant for look up... (21 Replies)
Discussion started by: gimley
21 Replies

3. Shell Programming and Scripting

count frequency of words in a file

I need to write a shell script "cmn" that, given an integer k, print the k most common words in descending order of frequency. Example Usage: user@ubuntu:/$ cmn 4 < example.txt :b: (3 Replies)
Discussion started by: mohit_iitk
3 Replies

4. Shell Programming and Scripting

Splitting concatenated words in input file with words from the same file

Dear all, I am working with names and I have a large file of names in which some words are written together (upto 4 or 5) and their corresponding single forms are also present in the word-list. An example would make this clear annamarie mariechristine johnsmith johnjoseph smith john smith... (8 Replies)
Discussion started by: gimley
8 Replies

5. Shell Programming and Scripting

Assigning the same frequency to more than one words in a file

I have a file of names with the following structure NAME FREQUENCY NAME NAME FREQUENCY NAME NAME NAME FREQUENCY i.e. more than one name is assigned the same frequency. An example will make this clear SANDHYA DAS 6901 ARATI DAS 6201 KALPANA DAS 4714 GITA DAS 4550 BISWANATH DAS 3949... (4 Replies)
Discussion started by: gimley
4 Replies

6. Shell Programming and Scripting

How count the number of two words associated with the two words occurring in the file?

Hi , I need to count the number of errors associated with the two words occurring in the file. It's about counting the occurrences of the word "error" for where is the word "index.js". As such the command should look like. Please kindly help. I was trying: grep "error" log.txt | wc -l (1 Reply)
Discussion started by: jmarx
1 Replies

7. UNIX for Dummies Questions & Answers

Replace the words in the file to the words that user type?

Hello, I would like to change my setting in a file to the setting that user input. For example, by default it is ONBOOT=ON When user key in "YES", it would be ONBOOT=YES -------------- This code only adds in the entire user input, but didn't replace it. How do i go about... (5 Replies)
Discussion started by: malfolozy
5 Replies

8. Shell Programming and Scripting

Frequency of Words in a File, sed script from 1980

tr -cs A-Za-z\' '\n' | tr A-Z a-z | sort | uniq -c | sort -k1,1nr -k2 | sed ${1:-25} < book7.txt This is not my script, it can be found way back from 1980 but once it worked fine to give me the most used words in a text file. Now the shell is complaining about an error in sed sed: -e... (5 Replies)
Discussion started by: 1in10
5 Replies

9. HP-UX

Problems creating and accessing with user

Hi, I have created the user 'mastersa' in several servers. I need to change the user ID to '0'. However, after doing this, I am not able to login (Access denied). Even after I change the password, I still get this error. Why is this? Also, when I attempt to delete the user account, I get... (5 Replies)
Discussion started by: anaigini45
5 Replies

10. Shell Programming and Scripting

Replace particular words in file based on if finds another words in that line

Hi All, I need one help to replace particular words in file based on if finds another words in that file . i.e. my self is peter@king. i am staying at north sydney. we all are peter@king. How to replace peter to sham if it finds @king in any line of that file. Please help me... (8 Replies)
Discussion started by: Rajib Podder
8 Replies

LEARN ABOUT CENTOS

prezip-bin

PREZIP-BIN(1)						 Aspell Abbreviated User's Manual					     PREZIP-BIN(1)

NAME

       prezip-bin - prefix zip delta word list compressor/decompressor

SYNOPSIS

       prezip-bin [ -V | -d | -z ]

DESCRIPTION

       prezip-bin compresses/decompresses sorted word lists from standard input to standard output.

       Prezip-bin  is  similar	to  word-list-compress(1)  but	it  allows  a  larger  character set of {0x00...0x09, 0x0B, 0x0C, 0x0E...0xFF} and
       multi-words larger than 255 characters in length.  It can also decompress word-list-compress(1) compatible files.

COMMANDS

       Prezip-bin accepts only one of these commands.

       -V     Display prezip-bin version number to standard output.

       -d     Read a compressed word list from standard input and decompress it to standard output.  This can  be  a  word-list-compress(1)  or  a
	      prezip-bin compressed file.

       -z     Read a binary word list from standard input and compress it to standard output.

EXAMPLES

       prezip-bin -d <wordlist.cwl >wordlist.txt
	      Decompress file wordlist.cwl to text file wordlist.txt

       prezip-bin -z <wordlist.txt >wordlist.pz 2>errors.txt
	      Compress wordlist.txt to binary file wordlist.pz and send any error messages to a text file named errors.txt

       LC_COLLATE=C sort -u <wordlist.txt | prezip-bin -z >wordlist.pz
	      Sort a word list, then pipe it to prezip-bin to create a compressed binary wordlist.pz file.

       prezip-bin -d <words.pz | aspell create master ./words.rws
	      Decompress a wordlist, then pipe it to aspell(1) to create a spelling list.  Please check the aspell(1) info manual for proper usage
	      and options.

TIPS

       Prezip-bin is best used with sorted word list type files.  It is not a general purpose compression program since resulting files may  actu-
       ally increase in size.

       Unlike  word-list-compress(1)  if your word list has leading or trailing blank spaces for formatting purposes, you should remove them first
       before you compress your list using prezip-bin -z , otherwise those spaces will be included in the compressed binary output.

DIAGNOSTICS

       Prezip-bin normally exits with a return code of 0. If it encounters an error, a message is sent to  standard  error  output  (stderr),  and
       prezip-bin exits with a non-zero return value.  Error messages are listed below:

       (display help/usage message)
	      Unknown command given on the command line so prezip-bin displays a usage message to standard error output.

       unknown format
	      The input file appears not to be an expected format, or may possibly be a more advanced format.  The output file will be empty.

       corrupt input
	      This  is	only for the decompression command -d. The input file appeared to be of a correct format, but something appears wrong now.
	      There may be some valid data in output, but due to input corruption, the rest of the file can not be completed.

       unexpected EOF
	      The input file appeared okay but ended sooner than expected, therefore the output file is not complete.

SEE ALSO

       aspell(1), run-with-aspell(1), word-list-compress(1)

       Aspell is fully documented in its Texinfo manual.  See the `aspell' entry in info for more complete documentation.

REPORTING BUGS

       For help, see the Aspell homepage at <http://aspell.net>.  Send bug reports/comments to the Aspell user list at the above address.

AUTHOR

       This info page was written by Jose Da Silva <digital@joescat.com>.

prezip-bin-0.1.2						    2005-09-30							     PREZIP-BIN(1)
All times are GMT -4. The time now is 04:11 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy