Script to count word occurrences, but exclude some? Post: 302657019

Sponsored Content

Top Forums Shell Programming and Scripting Script to count word occurrences, but exclude some? Post 302657019 by agama on Friday 15th of June 2012 10:16:38 PM

06-15-2012

Registered User

Given your requirements for black listing and finding capitalised leading letters, I'd probably have approached it this way:

Code:

awk '
    NR == FNR { $1; blist[$1]; next; }      # read black list

    {
        for( i=1; i <= NF; i++ )
        {
            ignore = ignore_nxt;
            ignore_nxt = 0;
            ignore_nxt = ( match(  $i, "[?.!]" ) && RSTART == length( $i ) );

            gsub( "[:,%?<>&@!=+.()]", "", $(i) );       # trash punctuation not considered part of a word
            if( length( $(i) ) > 3 )
            {
                count[$(i)]++;
                fc = substr( $(i), 1, 1 );
                if( !ignore && fc >= "A" && fc <= "Z" )
                    cap++;
            }
        }
    }
    END {
        printf( "words starting with a capital: %d\n", cap ) >"/dev/fd/2";  # out to stderr so it doesnt sort
        for( x in count )
        {
            if( !( x in blist ) )
                print x, count[x];
        }
    }
' blacklist.file text-file | sort -k 2nr,2

The captialisation is tricky. You can count all words with capitalised letters, or ignore those that immediatly follow a full stop (.), question (?) or explaination (!). The code above does the latter -- effectively counting proper names that appear in the middle of the sentence. You can comment out the statements that check for and set the ignore variables, and it will count all words that start with a capitalised letter and are larger than 3 characters in length.

Might not be exactly what you want, but it should give you an idea of one method.

This User Gave Thanks to agama For This Post:

agama

View Public Profile for agama

Find all posts by agama

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

count occurrences and substitute with counter

Hi Unix-Experts, I have a textfile with several occurrences of some string XXX. I'd like to count all the occurrences and number them in reverse order. E.g. input: XXX bla XXX foo XXX output: 3 bla 2 foo 1 I tried to achieve this with sed, but failed. Any suggestions? Thanks in...

2. Shell Programming and Scripting

Count the number of occurrences of the word

I am a newbie in UNIX shell script and seeking help on this UNIX function. Please give me a hand. Thanks. I have a large file. Named as 'MyFile'. It was tab-delmited. I am told to write a shell function that counts the number of occurrences of the ord �mysring� in the file 'MyFile'.

3. Shell Programming and Scripting

Count occurrences in awk

Hello, I have an output from GDB with many entries that looks like this 0x00007ffff7dece94 39 in dl-fini.c 0x00007ffff7dece97 39 in dl-fini.c 0x00007ffff7ab356c 50 in exit.c 0x00007ffff7aed9db in _IO_cleanup () at genops.c:1022 115 in dl-fini.c 0x00007ffff7decf7b in _dl_sort_fini (l=0x0,...

4. Shell Programming and Scripting

How to count occurrences in a specific column

Hi, I need help to count the number of occurrences in $3 of file1.txt. I only know how to count by checking one by one and the code is like this: awk '$3 ~ /aku hanya poyo/ {++c} END {print c}' FS="\t" file1.txt But this is not wise to do as i have hundreds of different occurrences in that...

5. Shell Programming and Scripting

Word Count In A Script

I am in need of a basic format to 1. list all files in a directory 2. list the # of lines in each file 3. list the # of words in each file If someone could give me a basic format i would appreicate it ***ALSO i can not use the FIND command***

6. Shell Programming and Scripting

Word Occurrences script using awk

I'm putting together a script that will the count the occurrences of words in text documents. It works fine so far, but I'd like to make a couple tweaks/additions: 1) I'm having a hard time displaying the array index number, tried freq which just spit 0's back at me 2) Is there any way to...

7. Shell Programming and Scripting

Count occurrences in first column

input amex-11 10 abc amex-11 20 bcn amed-12 1 abc I tried something like this. awk '{h++}; END { for(k in h) print k, h }' rm1 output amex-11 1 10 abc amex-11 1 20 bcn amed-12 2 1 abc Note: The second column represents the occurrences. amex-11 is first one and amed-12 is the...

8. UNIX for Beginners Questions & Answers

UNIX script to check word count of each word in file

I am trying to figure out to find word count of each word from my file sample file hi how are you hi are you ok sample out put hi 1 how 1 are 1 you 1 hi 1 are 1 you 1 ok 1 wc -l filename is not helping , i think we will have to split the lines and count and then print and also...

9. UNIX for Beginners Questions & Answers

awk or sed script to count number of occurrences and creating an average

Hi Friends , I am having one problem as stated file . Having an input CSV file as shown in the code U_TOP_LOGIC/U_HPB2/U_HBRIDGE2/i_core/i_paddr_reg_2_/Q,1,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,1,1,0,0,0,0...

LEARN ABOUT DEBIAN

poterminology

poterminology(1)					      Translate Toolkit 1.9.0						  poterminology(1)

NAME

       poterminology - reads a set of .po or .pot files to produce a pootle-terminology.pot

SYNOPSIS

       poterminology [--version] [-h|--help] [--manpage] [--progress PROGRESS] [--errorlevel ERRORLEVEL] [-i|--input] INPUT [-x|--exclude EXCLUDE]
       [-o|--output] OUTPUT [-u|--update UPDATEFILE] [-S|--stopword-list STOPFILE] [-F|--fold-titlecase]  [-C|--preserve-case]	[-I|--ignore-case]
       [--accelerator  ACCELERATORS] [-t|--term-words LENGTH] [--nonstop-needed MIN] [--inputs-needed MIN] [--fullmsg-needed MIN] [--substr-needed
       MIN] [--locs-needed MIN] [--sort ORDER] [--source-language LANG] [-v|--invert]

DESCRIPTION

       See: http://translate.sourceforge.net/wiki/toolkit/poterminology for examples and usage instructions

OPTIONS

       --version
	      show program's version number and exit

       -h/--help
	      show this help message and exit

       --manpage
	      output a manpage based on the help

       --progress
	      show progress as: dots, none, bar, names, verbose

       --errorlevel
	      show errorlevel as: none, message, exception, traceback

       -i/--input
	      read from INPUT in pot, po formats

       -x/--exclude
	      exclude names matching EXCLUDE from input paths

       -o/--output
	      write to OUTPUT in po, pot formats

       -u/--update
	      update terminology in UPDATEFILE

       -S/--stopword-list
	      read stopword (term exclusion) list from STOPFILE (default /usr/share/pyshared/translate/share/stoplist-en)

       -F/--fold-titlecase
	      fold "Title Case" to lowercase (default)

       -C/--preserve-case
	      preserve all uppercase/lowercase

       -I/--ignore-case
	      make all terms lowercase

       --accelerator
	      ignores the given accelerator characters when matching

       -t/--term-words
	      generate terms of up to LENGTH words (default 3)

       --nonstop-needed
	      omit terms with less than MIN nonstop words (default 1)

       --inputs-needed
	      omit terms appearing in less than MIN input files (default 2, or 1 if only one input file)

       --fullmsg-needed
	      omit full message terms appearing in less than MIN different messages (default 1)

       --substr-needed
	      omit substring-only terms appearing in less than MIN different messages (default 2)

       --locs-needed
	      omit terms appearing in less than MIN different original source files (default 2)

       --sort output sort order(s): frequency, dictionary, length (default is all orders in the above priority)

       --source-language
	      the source language code (default 'en')

       -v/--invert
	      invert the source and target languages for terminology

							      Translate Toolkit 1.9.0						  poterminology(1)