Sponsored Content
Top Forums Shell Programming and Scripting Creating a syllable concordance Post 302522254 by Skrynesaver on Saturday 14th of May 2011 02:20:29 AM
Old 05-14-2011
Hi,

I'm not sure of your required format however the Perl example below should provide the necessary structures.

It loads the entire memory into a hash, this is expensive with a large corpus, however it need only be done once and each search request afterwards is very fast indeed.

If you are running this on a DOS machine, you needn't set the input record separator ($/) as it defaults to the appropriate value for the current environment.
Code:
#! /usr/bin/perl

use strict;
use warnings;

my %corpus;  # The hash we will use to store a map of the corpus in
$/ = "\r\n"; # These files are DOS files, so set the end of line accordingly
open (my $corpus, '<', 'Output')||die 'Could not open Output file, $!';
while(<$corpus>){
   chomp;
   if ($_){ # Avoid emptylines
      my($syllable, $example)=split(/=/, $_ );#extract the values
      $corpus{$syllable}=$example;            #and store in the global hash
   }

}
close $corpus;
open (my $syllable ,'<', 'Syllables' )||die 'Could not open Syllables file, $!';
while (<$syllable>){
   chomp;
   print $corpus{$_} ? "$_ has the example $corpus{$_} in the provided corpus\n": "I do not have an example for $_\n";
}


Last edited by Skrynesaver; 05-14-2011 at 03:26 AM.. Reason: added check for existence of entry in corpus file
This User Gave Thanks to Skrynesaver For This Post:
 

5 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

CREATING A SYLLABLE CONCORDANCE WITH POSITIONAL VARIANTS

Hello, Some time back I had posted a request for a syllable concordance in which if a syllable was provided in a file, the program would extract a word from a file entitled "Corpus" matching that syllable. The program was The following script was provided which did the job and for which I am... (3 Replies)
Discussion started by: gimley
3 Replies

2. Shell Programming and Scripting

Syllable splitter in Perl

Hello, I am a relative newbie and want to split Names in English into syllables. Does anyone know of a perl script which does that. Since my main area is linguistics, I would be happy to add rules to it and post the perl script back for other users. I tried the CPan perl modules but they don't... (6 Replies)
Discussion started by: gimley
6 Replies

3. Shell Programming and Scripting

Writing a clustering concordance for a Perso-Arabic script

I am working on a database of a language using Arabic Script. One of the major issues is that the shape of the characters changes according to their initial, medial or final positioning. Another major issue is that of the clustering of vowels within the word: the clustering changes totally the... (9 Replies)
Discussion started by: gimley
9 Replies

4. Shell Programming and Scripting

Modifying an awk script for syllable splitting

I have found this syllable splitter in awk. The code is given below. Basically the script cuts words and names into syllables. However it fails when the word contains 2 consonants which constitute a single syllable. An example is given below ashford raphael The output is as under: ... (4 Replies)
Discussion started by: gimley
4 Replies

5. Shell Programming and Scripting

Find Syllable count mismatch

Hello, I have written a syllable splitter for Pseudo English and Indic. I have a large database with the following structure Syllables in Pseudo English delimited by |=Syllables in Devanagari delimited by | The tool produces syllables in both scripts. An example is given below: ... (2 Replies)
Discussion started by: gimley
2 Replies
GRIND(1)						      WordNettm User Commands							  GRIND(1)

NAME
grind - process WordNet lexicographer files SYNOPSIS
grind [ -v ] [ -s ] [ -Llogfile ] [ -a ] [ -d ] [ -i ] [ -o ] [ -n ] filename [ filename... ] DESCRIPTION
grind() processes WordNet lexicographer files, producing database files suitable for use with the WordNet search and interface code and other applications. The syntactic and structural integrity of the input files is verified. Warnings and errors are reported via stderr and a run-time log is produced on stdout. A database is generated only if there are no errors. Input Files Input files correspond to the syntactic categories implemented in WordNet - noun, verb, adjective and adverb. Each input lexicographer file consists of a list of synonym sets (synsets) for one part of speech. Although the basic synset syntax is the same for all of the parts of speech, some parts of the syntax only apply to a particular part of speech. See wninput(5WN) for a description of the input file format. Each filename specified is of the form: pathname/pos.suffix where pathname is optional and pos is either noun, verb, adj or adv. suffix may be used to separate groups of synsets into different files, for example noun.animal and noun.plant. One or more input files, in any combination of syntactic categories, may be specified. See lexnames(5WN) for a list of the lexicographer files used to build the complete WordNet database. Output Files grind() produces the following output files: +------------+----------------------------------------+ | Filename | Description | +------------+----------------------------------------+ |index.pos | Index file for each syntactic category | |data.pos | Data file for each syntactic category | |index.sense | Sense index | +------------+----------------------------------------+ See wndb(5WN) for a description of the database file formats. Each time grind() is run, any existing database files are overwritten with the database files generated from the specified input files. If no input files from a syntactic category are specified, the corresponding database files are not overwritten. Sense Numbers Senses are generally ordered from most to least frequently used, with the most common sense numbered 1. Frequency of use is determined by the number of times a sense is tagged in the various semantic concordance texts. Senses that are not semantically tagged follow the ordered senses in an arbitrary order. Note that this ordering is only an estimate based on usage in a small corpus. The tagsense_cnt field for each entry in the index.pos files indicates how many of the senses in the list have been tagged. The cntlist file provided with the database lists the number of times each sense is tagged in the semantic concordances. grind() uses the data from cntlist to order the senses of each word. When the index.pos files are generated, the synset_offsets are output in sense number order, with sense 1 first in the list. Senses with the same number of semantic tags are assigned unique but consecutive sense numbers. The WordNet OVERVIEW search displays all senses of the specified word, in all syntactic categories, and indicates which of the senses are represented in the semantically tagged texts. OPTIONS
-v Verify integrity of input without generating database. -s Suppress generation of warning messages. Usually grind is run with this option until all syntactic and structural errors are corrected since the warning messages may make it difficult to spot error messages. -Llogfile Write all messages to logfile instead of stderr. -a Generate statistical report on input files processed. -d Generate distribution of senses by string length report on input files processed. -i Generate sense index file. -o Order senses using cntlist. -n Generate nominalization (derivational morphology) links in database. filename Input file of the form described in Input Files. FILES
pos.* lexicographer files to use to build database cntlist file of combined semantic concordance cntlist files. Used to assign sense numbers in WordNet database SEE ALSO
cntlist(5WN), lexnames(5WN), senseidx(5WN), wndb(5WN), wninput(5WN), uniqbeg(7WN), wngloss(7WN). DIAGNOSTICS
Exit status is normally 0. Exit status is -1 if non-specific error occurs. If syntactic or structural errors exist, exit status is number of errors detected. usage: grind [-v] [-s] [-Llogfile] [-a ] [-d] [-i] [-o] [-n] filename [filename...] Invalid options were specified on the command line. No input files processed. None of the filenames specified were of the appropriate form. n syntactic errors found. Syntax errors were found while parsing the input files. n structural errors found. Pointer errors were found that could not be automatically corrected. BUGS
Please report bugs to wordnet@princeton.edu. WordNet 3.0 Dec 2006 GRIND(1)
All times are GMT -4. The time now is 01:16 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy